CAPABILITY · CRAWL

Map a site in one call, then crawl only what you meant

Discover a site's URLs from its sitemap, robots and a shallow link scan for a flat fee — then run a bounded, resumable, robots-respecting crawl that can extract per page.

Start free API reference

POST /api/v1/map · POST /api/v1/crawl

Capability

Inside site mapping & crawl

Mapping and crawling are two different jobs and they are priced as two different jobs. Map is a fast, flat-fee discovery pass — sitemap.xml, robots.txt and a shallow link scan — that returns URLs and nothing else. Crawl is an asynchronous breadth-first read that costs per page, so the point of map is to decide what crawl should be allowed to touch.

Discovery before spend

Map returns the URL inventory with per-source counts and a truncated flag, for a flat fee and no page reads. Filter it, then hand crawl a scope you have actually looked at.

Scoping that is not just a depth number

A crawl takes glob allow/deny path lists, allow/deny regular expressions tested against the full URL, a `followSelector` that confines link discovery to one CSS region, and same-host and nofollow switches.

Polite by default, and adaptive

robots.txt Disallow, Crawl-delay and Request-rate are honoured unless you turn that off. Autothrottle learns a per-domain delay from response latency and doubles it — or honours Retry-After — when the site answers 429 or 403.

Resumable, so a truncation is not a re-spend

A truncated crawl returns `nextUrls` and `seenUrls`. Pass them back as `startUrls` and `seenUrls` to continue exactly where it stopped instead of re-reading — and re-paying for — pages you already have.

How it works

The mechanism, in execution order

4 stages, in the order the runtime performs them — not a summary of them.

1Map the sitePOST /api/v1/map with any URL on the site. Sitemaps, robots and a shallow scan produce the URL list, with source counts.
2Choose the scopeDecide the includePaths / excludePaths globs, the allow/deny patterns and the page ceiling from what map actually returned.
3Submit the crawlPOST /api/v1/crawl returns 202 with a jobId. The reserve is capped by `maxPages`, so the worst case is bounded before it starts.
4CollectPoll GET /api/v1/jobs/{id}, stream it, or set callbackUrl and callbackSecret for an HMAC-signed completion webhook.

In practice

A call you can paste and run

Map is the cheap look; crawl is the priced read. Scoping the second with the first is the whole technique.

crawl.shbash

# 1. Discover — flat 2 credits, no page reads
curl -X POST https://twin-browser.com/api/v1/map \
  -H "Authorization: Bearer $TWIN_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://docs.example.com","limit":2000,"search":"/guides/"}'

# 2. Crawl only that subtree — async, 3 credits per page
curl -X POST https://twin-browser.com/api/v1/crawl \
  -H "Authorization: Bearer $TWIN_API_KEY" \
  -H "content-type: application/json" \
  -d '{"startUrl":"https://docs.example.com/guides/",
      "includePaths":["/guides/**"],
      "maxPages":200,"maxDepth":3,
      "fields":["title","summary"],
      "callbackUrl":"https://your.app/hooks/crawl",
      "callbackSecret":"whsec_…"}'
# → 202 { "jobId": "…" }   then GET /api/v1/jobs/{id}

api.twin-browser.com

Site mapping & crawl

Map the sitedone
Choose the scoperunning
Submit the crawlqueued
Collectqueued

What it costs, how to switch it on

Priced from the same rate card the API serves

Every credit figure on this page is read from the rate card behind GET /api/v1/pricing — it is not typed into the copy, so it cannot drift from what you are billed.

2 credits to map · 3 credits per page crawled

Map is a flat 2-credit call with no model in it, whatever the site's size. Crawl bills 3 credits per page READ and reserves against maxPages, so the cap you set is the ceiling on what a crawl can cost. Per-page structured extraction (fields / extractSchema) adds metered model cost on top of the page price.

Full rate card

Turning it on

1Start with mapPOST /api/v1/map with a URL on the site and a `search` substring filter. It costs a flat fee and tells you the shape of what you are about to crawl.
2Set a real ceiling`maxPages` caps both the crawl and the credit reserve. Set it deliberately — the default is 50.
3Scope by path, not by depthincludePaths / excludePaths globs and allow/deny regexes are more precise than maxDepth, and cheaper than discovering the whole site first.
4Wire the completion webhookcallbackUrl plus callbackSecret gets you an HMAC-signed POST on completion instead of a polling loop.

At a glance

The contract, in the fewest rows that say it

Property	Twin Browser
Map	POST /api/v1/map — sync, flat fee
Map ceiling	`limit` default 1000, max 5000
Crawl	POST /api/v1/crawl — async, 202 { jobId }
Crawl ceiling	`maxPages` default 50, max 1000
Concurrency	1–5 parallel fetches, default 3
Robots	Honoured by default (`respectRobots`)

Limits

Where it stops, and what it deliberately does not do

A capabilities page with no limits section is a brochure. These are the ceilings, the defaults that will surprise you, and the things this capability is not.

Limits and defaults

Map returns URLs, never page content. If you need what is on the pages, that is a crawl.
A crawl is asynchronous by construction — there is no synchronous whole-site read.
Concurrency is capped at 5, and autothrottle will lower the effective rate on a site that pushes back.
A page that answers 401/403/407/429/444/503 is retried `maxBlockedRetries` times (default 1, max 3) and then recorded as blocked rather than retried forever.

What it does not do

It does not ignore robots.txt by default. `respectRobots` defaults to true, and Crawl-delay and Request-rate are honoured.
It does not follow links off the start host unless you set `sameDomainOnly: false`.
It does not render a JavaScript-only site into an inventory in map — the shallow scan is a link scan, and `scan: false` reduces it to sitemap-only.
It does not de-duplicate against a previous crawl on its own. Resume is explicit: hand back `seenUrls`.

FAQ

Questions about site mapping & crawl

Why map before crawling?

Because map is flat and crawl is per page. 2 credits buys the URL inventory; scoping a crawl from it rather than guessing is usually the difference between reading 200 pages and reading 2,000.

Can a crawl extract structured data as it goes?

Yes. Pass `fields` or `extractSchema` (with an optional `prompt`) and each page is read into that shape as it is crawled. That adds metered model cost on top of the per-page price.

What happens when a crawl is truncated?

The response carries `nextUrls` and `seenUrls`. Submit a new crawl with those as `startUrls` and `seenUrls` and it continues exactly where the last one stopped — no re-reading, no re-paying.

How does it behave on a site that starts pushing back?

Autothrottle learns a per-domain delay from response latency, and doubles it (or honours Retry-After) when the site answers 429 or 403, relaxing again as it recovers. A page that stays blocked is recorded as blocked rather than retried indefinitely.

Keep reading

The rest of the platform

The capabilities this one works with, the tools that expose it, the guides that use it end to end, and the vocabulary behind it.

map_site submit_crawl get_job

Guides

Crawl a website without crawling the whole website Extract structured data from a page without paying for a model

Vocabulary

Crawler Site map (discovery)robots.txt Autothrottle

Put your agent to work. Keep the decision.

Start free. Hand your agent a goal on a site you authorize, set the guardrails, and let the first successful run compile the skill every run after it replays.

Start free Read the docs