Observation & data
Enumerate a site’s URLs without reading them
sitemap.xml plus robots.txt plus one shallow link scan, deduped into a URL list with per-source counts — the cheap reconnaissance step before an expensive crawl.
map_site
What it is
The job: Find out what is on a site before deciding how much of it to read.
Crawling a site to find out how big it is costs the same as crawling it for real. map_site is the reconnaissance step that avoids that: it reads sitemap.xml and robots.txt, does one shallow link scan, and returns the deduped URL list with a count per discovery source and a flag telling you whether the result was truncated.
No page content comes back, which is exactly why it is cheap. The right pattern is map first, filter with the `search` substring or your own logic, then feed a bounded set of paths to submit_crawl — instead of pointing a crawl at a domain and hoping the page cap lands somewhere sensible.
Set scan: false for the fastest possible answer from sitemaps alone; set includeSubdomains when the site’s real content lives on more than one host.
The call
Call it exactly like this.
Copied from the tool's registration and the route handler — not paraphrased.
// MCP tool call — server "twin-browser"
{
"tool": "map_site",
"arguments": {
"url": "https://example.com",
"limit": 2000,
"search": "/docs/"
}
}curl -X POST https://twin-browser.com/api/v1/map \
-H "Authorization: Bearer $TWIN_API_KEY" \
-H "content-type: application/json" \
-d '{"url":"https://example.com","limit":2000,"search":"/docs/"}'| Parameter | Type | What it does |
|---|---|---|
| url* | string | A URL on the site to map — its origin is the target. |
| limit | number | Max URLs to return. Default 1000, max 5000. |
| includeSubdomains | boolean | Include subdomains of the site. Default false. |
| search | string | Case-insensitive substring filter applied to the URL. |
| scan | boolean | Perform the shallow link scan. Default true; set false for the fastest sitemap-only answer. |
Returns
{
"urls": [ "https://example.com/docs/intro", … ],
"count": 412,
"source_counts": { "sitemap": 380, "scan": 32 },
"truncated": false
}What it costs
2 credits
Flat, whatever the site’s size — no page content is read and no model runs. Mapping a 5,000-URL site costs the same 2 credits as mapping a five-page one, which is what makes it the right thing to do first.
See the full rate cardWhich one
When a different tool is the right call.
The honest answer is often the neighbouring tool. These are the trades.
submit_crawlA crawl reads the pages and bills 3 credits each. Map first: it is flat, it is fast, and it lets you scope the crawl to the paths that actually matter instead of guessing at maxPages.
submit_crawlobserve_pageOne is site-scale and structural, the other page-scale and element-level. Map answers “which pages exist”, observe answers “what is on this one”.
observe_pageQuestions
map_site, answered.
- Does it respect robots.txt?
- It reads robots.txt as a discovery source — that is where a lot of sitemap references live. The rule enforcement that matters is on the crawl: submit_crawl honours robots.txt Disallow, Crawl-delay and Request-rate by default.
Keep going