What is autothrottle?
Autothrottle learns a per-domain delay from observed response latency and adjusts it dynamically: it doubles the delay (or honours a Retry-After header) when the site answers 429 or 403, and eases back as responses normalize. It is on by default.
Why it matters
A fixed delay is either too slow for a healthy site or too fast for a struggling one. Adapting to the target is both more polite and more effective — a crawl that backs off finishes; a crawl that hammers gets blocked and is refunded nothing for the pages it already read.