Mapping and crawling are priced differently on purpose: a map is a flat 2 credits whatever the site's size, and a crawl is 3 credits per page READ. That asymmetry is the whole technique — use the cheap call to decide what the expensive one is allowed to touch.
Step one: map the site
POST /api/v1/map takes any URL on the site and returns the URL inventory from sitemap.xml, robots.txt and a shallow link scan, with per-source counts and a truncation flag. It reads no page content, so it is flat-priced. A `search` substring filter narrows the result before it comes back.
curl -X POST https://twin-browser.com/api/v1/map \
-H "Authorization: Bearer $TWIN_API_KEY" \
-H "content-type: application/json" \
-d '{ "url": "https://docs.example.com",
"limit": 2000,
"search": "/guides/" }'
# { "urls": [ … ], "source_counts": { … }, "truncated": false,
# "credits_charged": 2 }Step two: scope by path, not by depth
maxDepth is a blunt instrument — it caps how far you walk but not where. Path globs and regular expressions are precise. `includePaths` and `excludePaths` take globs; `allowPatterns` and `denyPatterns` are regular expressions tested against the full URL with deny winning over allow; and `followSelector` confines link discovery to one CSS region, which is the difference between crawling a catalogue and crawling the site chrome.
- includePaths: ["/guides/**"] — the glob allowlist.
- denyPatterns: ["\\?page=\\d{3,}"] — a regex denylist beats an allow match.
- followSelector: ".product-grid" — only discover links inside that region.
- sameDomainOnly defaults true; respectNofollow defaults true.
Step three: submit the crawl
POST /api/v1/crawl returns 202 with a jobId. `maxPages` caps both the crawl and the credit reserve, so the worst case is bounded before anything starts — set it deliberately, because the default is 50. Per-page structured extraction is available inline with `fields` or `extractSchema`, which adds metered model cost on top of the per-page price.
curl -X POST https://twin-browser.com/api/v1/crawl \
-H "Authorization: Bearer $TWIN_API_KEY" \
-H "content-type: application/json" \
-d '{
"startUrl": "https://docs.example.com/guides/",
"includePaths": ["/guides/**"],
"maxPages": 200,
"maxDepth": 3,
"concurrency": 3,
"fields": ["title", "summary"],
"callbackUrl": "https://your.app/hooks/crawl",
"callbackSecret": "whsec_…"
}'
# → 202 { "jobId": "job_…" }
curl "https://twin-browser.com/api/v1/jobs/job_…" \
-H "Authorization: Bearer $TWIN_API_KEY"Being a good citizen (and staying unblocked)
robots.txt Disallow, Crawl-delay and Request-rate are honoured by default. Autothrottle learns a per-domain delay from response latency and doubles it — or honours Retry-After — when the site answers 429 or 403, easing back as it recovers. Concurrency is capped at 5. These defaults are not just etiquette: a crawl that backs off finishes, and a crawl that hammers gets its egress blocked with nothing refunded for the pages already read.
Resuming a truncated crawl
This is the part that saves real money. When a crawl hits its page ceiling the response carries `nextUrls` and `seenUrls`. Submit a new crawl passing those back as `startUrls` and `seenUrls` and it continues exactly where the last one stopped — without re-reading, or re-paying for, the pages you already have.
curl -X POST https://twin-browser.com/api/v1/crawl \
-H "Authorization: Bearer $TWIN_API_KEY" \
-H "content-type: application/json" \
-d '{
"startUrl": "https://docs.example.com/guides/",
"startUrls": [ /* the "nextUrls" from the previous response */ ],
"seenUrls": [ /* the "seenUrls" from the previous response */ ],
"maxPages": 200
}'Crawling behind a login
A crawl accepts the same account and session controls as a run: pass `account` to use a stored login for the host and `persistSession` so the session is reused across the crawl rather than re-established per page. If the site blocks automated sign-in outright, connect the account once with a handoff link first.