Observation & data
Walk a whole site, bounded and resumable
A breadth-first crawl from a start URL, scoped by depth, page cap, path globs and regex patterns, throttled per domain, and resumable from where a truncated run stopped.
submit_crawl
What it is
The job: Read a defined slice of a site, once, without either missing it or over-reading it.
The crawl is bounded in every direction on purpose, because an unbounded crawl is an unbounded bill. maxPages caps how many pages are read and also caps the credit reservation; maxDepth caps how far from the start URL links are followed; includePaths and excludePaths take globs, allowPatterns and denyPatterns take regular expressions (deny wins), and followSelector restricts link discovery to one region of the page — the difference between crawling a catalogue and crawling the site chrome.
It is polite by default. robots.txt Disallow, Crawl-delay and Request-rate are honoured, rel="nofollow" links are skipped, and autothrottle learns a per-domain delay from response latency, doubling it or honouring Retry-After when the site answers 429 or 403 and relaxing as it recovers. A page that is blocked is re-queued after the backoff up to maxBlockedRetries before being recorded as blocked.
Truncation is recoverable rather than wasteful: a crawl that hits its cap returns nextUrls and seenUrls, and passing those back as startUrls and seenUrls continues exactly where it stopped instead of re-crawling — and re-paying for — pages you already have.
Pass fields or a schema and each page is also structurally extracted, so one job can produce a dataset rather than a pile of text.
The call
Call it exactly like this.
Copied from the tool's registration and the route handler — not paraphrased.
// MCP tool call — server "twin-browser"
{
"tool": "submit_crawl",
"arguments": {
"startUrl": "https://example.com/docs",
"maxPages": 200,
"maxDepth": 3,
"includePaths": ["/docs/**"],
"concurrency": 3
}
}curl -X POST https://twin-browser.com/api/v1/crawl \
-H "Authorization: Bearer $TWIN_API_KEY" \
-H "content-type: application/json" \
-d '{"startUrl":"https://example.com/docs","maxPages":200,"maxDepth":3,"includePaths":["/docs/**"],"concurrency":3}'| Parameter | Type | What it does |
|---|---|---|
| startUrl* | string | The URL to start crawling from — the authorization signal. |
| maxPages | number | Max pages to read. Default 50, max 1000. Also caps the credit reserve. |
| maxDepth | number | Max link depth from the start URL. Default 3. |
| includePaths | string[] | Glob allowlist of path patterns, e.g. ["/docs/**"]. |
| excludePaths | string[] | Glob denylist of path patterns. |
| sameDomainOnly | boolean | Stay on the start host. Default true. |
| respectRobots | boolean | Honour robots.txt Disallow, Crawl-delay and Request-rate. Default true. |
| concurrency | number | Parallel page fetches, 1–5. Default 3. |
| autothrottle | boolean | Learn a per-domain delay from response latency and back off on 429/403. Default true. |
| maxBlockedRetries | number | Times a blocked page is re-queued after backoff before being recorded as blocked. Default 1, max 3. |
| fields | array | object | Optional per-page structured extract spec. |
| schema | object | Optional per-page JSON Schema — sent as extractSchema on the REST body. |
| prompt | string | Extraction hint when fields or schema is set. |
| account | string | Account/credential label, for a logged-in crawl. |
| persistSession | boolean | Persist and resume browser session state for this account. |
| proxy | string | Your own egress proxy URL. |
| proxyRotate | boolean | Force per-request proxy rotation. |
| callbackUrl | string | Absolute http(s) URL to POST the crawl result to on completion. |
| callbackSecret | string | Secret used to HMAC-sign the callback body. |
Returns
HTTP 202
{ "jobId": "…" }
// then GET /api/v1/jobs/{id} → result:
{ "pages": [ { "url": "…", "content": "…", "links": [ … ], "data": { … } } ],
"nextUrls": [ … ], // pass back as startUrls to resume a truncated crawl
"seenUrls": [ … ] }What it costs
3 credits per page
Billed per page actually read. The reservation is maxPages × 3 credits taken up front and settled down to the pages that came back, so a crawl that finishes early costs less than it reserved — and one that would exceed your balance is trimmed rather than failed.
See the full rate cardWhich one
When a different tool is the right call.
The honest answer is often the neighbouring tool. These are the trades.
map_siteMap first. It is flat-priced and tells you what exists; a crawl bills per page and should be pointed at a slice you already understand.
map_siteetlA crawl gives you pages. etl gives you chunked, embedded, queryable content. If the destination is a retrieval index, ingest with etl rather than crawling into your own pipeline.
etlextractextract is one page with one shape. A crawl applies the same shape across many — same extraction machinery, different scope.
extractQuestions
submit_crawl, answered.
- What happens if I set maxPages higher than my balance allows?
- The crawl is trimmed to what you can afford rather than rejected: the reservation is sized to the affordable page count and the job runs to that limit. You then get nextUrls and seenUrls back, so topping up and resuming picks up exactly where it stopped.
- Can I crawl a site I have to be logged into?
- Yes — pass an `account` label and persistSession, exactly as you would on a run. The crawl uses that identity’s stored session and its sticky egress IP, so the site sees one returning user rather than a swarm of anonymous ones.
Keep going
The rest of the tool set.
map_siteFind out what is on a site before deciding how much of it to read.
get_jobFind out whether the thing you started has finished, and get what it produced.
extractGet data off a page in the shape your code already expects.
etlTurn web content into something you can ask questions of, without building a pipeline.