Crawl Budget
Crawl budget is how much crawling effort search engines allocate to your site. Large sites and sites with many duplicates benefit most from management.
Definition
Crawl budget is the amount of crawling resources search engines allocate to a site over time. It’s influenced by site health (errors, speed), update frequency, internal structure, and duplicate content. Small sites rarely need to obsess; large/complex sites do.
Why it matters
- Prevents bots wasting time on duplicates and low-value URLs
- Lower error rates improve crawl efficiency
- Faster discovery and refresh of important pages
How to implement
- Reduce duplicates: canonical, noindex, parameter strategy
- Submit only important canonical URLs in sitemaps
- Improve performance and stability: lower errors and better TTFB
Related
Tutorials
Tools
FAQ
Common questions about this term.