What is crawler?
A crawler starts from one URL and walks the links it finds, subject to a page ceiling, a depth ceiling, path globs, allow and deny patterns, and a same-host rule. Each page it reads is billed, which is why the scope is a set of explicit bounds rather than a suggestion, and why the page cap also caps the credit reserve.
Why it matters
The failure mode of crawling is not technical, it is economic: an unbounded crawl of a large site is a bill. Scoping a crawl from a map of the site first — a flat-fee discovery pass — is the difference between reading the two hundred pages you wanted and the two thousand you did not.