Glossary · Data & retrieval

robots.txt

The file at a site’s root declaring what automated clients may fetch, how fast, and from where.

What is robots.txt?

robots.txt is a convention where a site publishes Disallow rules, Crawl-delay and Request-rate directives, and often points at its sitemaps. It is advisory by design rather than enforced by the protocol, which makes honouring it a choice a crawler makes.

Why it matters

Twin’s crawler honours Disallow, Crawl-delay and Request-rate by default, and you have to opt out explicitly. That default is the difference between a crawler that behaves like a citizen of the web and one that gets your egress blocked.

Related terms

CrawlerA bounded, breadth-first reader that follows links from a start URL under path, depth and page limits.
Site map (discovery)A flat-fee inventory of a site’s URLs from sitemap.xml, robots.txt and a shallow link scan — URLs only, no page content.
AutothrottleA crawler pacing itself from the site’s own responses — slowing when latency rises or a 429 arrives, relaxing as it recovers.
Cadence governingPacing runs per account so one identity is not re-poked faster than a real user would — and backing off after a push-back.

See it in context

Capability: site mapping & crawl
Acceptable use

Back to the full glossary

Put your agent to work. Keep the decision.

Give your agent a real browser, authorize the accounts it may use, and keep control of every step. Repeated runs compile into skills that replay without a model.

Start free See pricing