Glossary · Data & retrieval

Site map (discovery)

A flat-fee inventory of a site’s URLs from sitemap.xml, robots.txt and a shallow link scan — URLs only, no page content.

What is site map (discovery)?

Mapping discovers what URLs a site exposes without reading the pages behind them. It combines declared sitemaps, robots.txt directives and a shallow scan of links, returns the list with per-source counts and a truncation flag, and costs a flat fee whatever the site’s size.

Why it matters

It is the cheap look that makes the expensive read deliberate. Filtering a map is how you decide what a crawl is allowed to touch — and it answers "does this site even have the section I think it has" for the price of one call.

Related terms

See it in context

Back to the full glossary

Put your agent to work. Keep the decision.

Give your agent a real browser, authorize the accounts it may use, and keep control of every step. Repeated runs compile into skills that replay without a model.