For pricing, product marketing and growth
Know the day a competitor changes their pricing.
Map the pages that exist, crawl only the ones that matter, read the rows deterministically, and let a monitor push you a signed webhook the moment a watched value moves — instead of buying a feed or assigning an intern.
The problem
What this costs you today.
Somebody on the team keeps a spreadsheet of competitor prices, and it is accurate on the day it was filled in. The vendors who sell this as a feed cover the two competitors you already understand and none of the four you are worried about. Meanwhile the pages themselves are right there, public, and changing — you just have no reliable way to turn them into rows on a schedule.
- “Our competitive pricing sheet is three months stale and nobody trusts it.”
- “A data vendor quoted us more than the decision is worth.”
- “Our scraper broke when they moved to a JavaScript pricing page.”
- “We found out about their new plan tier from a customer.”
- Map the site’s URLsdone
- Crawl only /pricingrunning
- Expand plan rowsqueued
- Return structured rowsqueued
- Watch for the next changequeued
How Twin solves it
The mechanism, not a promise.
Three cheap, bounded primitives instead of one expensive general crawler: discover the URLs, read only the subset you care about, and then watch. None of the three needs a model in the loop unless you ask for one.
- 1Discover the URLs firstPOST /api/v1/map reads sitemap.xml and robots.txt plus a shallow link scan and returns the site’s URLs — no page content, no model, flat priced. `search` filters them by substring, so you can ask only for the pricing pages.
- 2Crawl a bounded subsetPOST /api/v1/crawl is an async, breadth-first crawl bounded by `maxPages`, `maxDepth`, `includePaths` and `excludePaths`, with `followSelector` to discover links only inside one region of the page. It returns 202 with a jobId.
- 3Be a good citizen by default`respectRobots` and `respectNofollow` default to true, and `autothrottle` learns a per-domain delay from response latency and backs off when the site answers 429 or 403.
- 4Read the rows deterministicallyPOST /api/v1/extract with `rowSelector` and `rowFields` turns a plan grid or a product listing into an array of objects with no model call for the reading.
- 5Then stop lookingPOST /api/v1/monitors watches a page and pushes an HMAC-signed webhook when the watched value changes, so the next price change reaches your channel instead of your calendar.
In practice
The actual call, and what it returns.
Discovery is flat-priced and content-free, so you find out what a crawl would cost before you pay for one. Then you crawl the twenty-five pages that matter, not the whole site.
# 1 — what pages exist? No content, no model, flat price.
curl https://twin-browser.com/api/v1/map \
-H "Authorization: Bearer $TWIN_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "url": "https://competitor.example.com", "search": "/pricing" }'
# 2 — read only the subset that matters, bounded and throttled.
curl https://twin-browser.com/api/v1/crawl \
-H "Authorization: Bearer $TWIN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"startUrl": "https://competitor.example.com/pricing",
"includePaths": ["/pricing/**"],
"maxPages": 25,
"maxDepth": 2,
"fields": ["plan_name", "monthly_price", "seats_included"]
}'
# → 202 { "jobId": "…" } poll GET /api/v1/jobs/{id}
# 3 — watch the page that matters most.
curl https://twin-browser.com/api/v1/monitors \
-H "Authorization: Bearer $TWIN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "competitor pricing",
"check_type": "extract",
"url": "https://competitor.example.com/pricing",
"nl_watch": "the monthly price of each plan",
"interval_seconds": 21600,
"callback_url": "https://growth.example.com/hooks/twin"
}'What this call does
- Map returns URLs only — it is the cheap way to size a crawl before committing to one.
- `includePaths` and `excludePaths` take glob patterns; `allowPatterns` and `denyPatterns` take regular expressions, and deny always wins.
- A truncated crawl is resumable: pass the previous response’s `nextUrls` and `seenUrls` back to continue instead of re-crawling pages you already paid for.
- A monitor can watch named `watch_fields` or a natural-language `nl_watch` description of the value you care about.
What it costs
Priced per action, not per seat.
Discovery is flat and content-free; crawling bills per page read, so the bound you set on the crawl is the bound on the bill.
| Action | Credits | What you get |
|---|---|---|
| POST /map — discover URLs | 2 / site | Sitemap, robots and a shallow link scan. No page content, no model. |
| POST /crawl — read pages | 3 / page | `maxPages` caps the credit reserve as well as the crawl. |
| POST /search — ranked web results | 3 / query | Blended web, news, discussions and Hacker News results, deduped. |
| …with `fetchContent` | 1 / page fetched | Adds the cleaned content of the top results. |
| POST /extract — read the rows | metered | Metered higher-of; deterministic `rowFields` reading has no model cost. |
How the unit works
- $1 buys 1,000 credits; the smallest pack is $5.
- A paid action bills the higher of its flat floor and what it actually spent on model, compute and egress — so a cheap run stays cheap.
- “Metered” means the action has no published flat floor on this page: GET /api/v1/pricing serves the live card.
- A crawl reserves credits against `maxPages` and settles on what it actually read, so an early-terminating crawl is not billed for pages it never fetched.
Be sure this fits
What this does not do.
Every one of these will come up in your evaluation. Here they are first, from us.
It honours robots.txt, and that will sometimes block you
`respectRobots` defaults to true and the crawler honours Disallow, Crawl-delay and Request-rate. Some competitors disallow the exact paths you want. We are not going to pretend that is a switch you should casually flip.
It is not a licensed data feed
There is no historical backfill, no entity resolution across sites, and no guarantee a page will still be there tomorrow. You are collecting public pages yourself, with the obligations that carries — including the target’s terms of service.
A price on a page is not always the price
Geo-priced, logged-in and experiment-bucketed pages show different numbers to different visitors. Pinning egress to a region makes a run reproducible; it does not make one rendered page the whole truth of a competitor’s pricing.
Under the hood
The primitives this runs on.
Nothing here is specific to this problem — the same mechanisms carry every solution on the site.
Token-efficient DOM state
The live page is serialized into a numerically-indexed list of its interactive elements under a token budget, so the planner reasons over signal instead of markup.
Read the mechanism — Token-efficient DOM stateEgress, geo & locale
A single `country` field sets the residential exit AND the browser locale and timezone, so a site cannot serve a different language than the exit you asked for — or bring your own proxy and keep both.
Read the mechanism — Egress, geo & localeDeterministic replay
A successful run is minimized into a named, versioned skill — an ordered action path with its variable parts lifted out — and replaying it is a program, not a prompt.
Read the mechanism — Deterministic replaySemantic dispatch cache
A re-phrased request is embedded, vector-matched against the skills you have already compiled, and replayed deterministically — so the second ask and every one after it skips the planner.
Read the mechanism — Semantic dispatch cacheOver MCP, the same work is these tools
map_siteDiscover a site’s URLs fast from sitemap.xml + robots.txt + a shallow link scan. No page content.
submit_crawlCrawl a whole site asynchronously (BFS), bounded by page/depth limits and path globs; returns a job id. Optional per-page structured extraction.
extractRead an authorized page and return structured JSON matching the fields or JSON schema you request.
searchWeb search returning ranked results; set fetchContent to also fetch and clean the top results.
create_monitorWatch a page on a schedule and push an HMAC-signed webhook when the watched value changes (needs WEB_BASE_URL + TWIN_API_KEY).
FAQ
Competitive and market data — common questions.
How is this different from a scraping API?
What stops a crawl from running away with our credits?
Does it respect robots.txt?
Can it watch a page behind our own login?
Related
Other problems this layer solves.
Reporting from behind a login
Analytics and data-platform teams whose sources are vendor dashboards rather than databases.
When the site turns you away
Teams whose authorized automation is being turned away by bot defences on sites they are entitled to use.
Browser infrastructure for AI products
Founders and engineering leads whose AI product has to act on the live web for every customer.
Try it on your hardest screen.
Start free, point a run at the system that is blocking you, and watch it happen live. If it does not work, the run tells you why — and what to do instead.