Crawl budget
The number of pages a search engine will crawl on your site in a given period, worth managing on large sites.
In depth
What it really means
Crawl budget is roughly how many of your pages a search engine will fetch in a given window. For small sites it is rarely a concern. For large sites with thousands of URLs, wasting it on junk pages means important pages get crawled and updated less often.
You manage it by not making crawlers waste time: fewer low-value URLs, clean architecture, and fast responses.
Best practices
- Block or remove low-value and duplicate URLs from crawling.
- Keep a clean, shallow site architecture.
- Fix crawl errors and slow responses.
- Use the sitemap to point crawlers at what matters.
- Consolidate thin pages so budget goes to strong ones.
FAQs
What is crawl budget?
The number of pages a search engine will crawl on your site in a given period.
Do I need to worry about crawl budget?
Mostly on large sites with many URLs. Small sites are usually crawled fully without issue.
How do I optimize crawl budget?
Reduce low-value and duplicate URLs, keep architecture clean, fix errors, and speed up responses.
What wastes crawl budget?
Duplicate pages, endless parameter URLs, broken links and slow servers all waste it.
How does crawl budget affect rankings?
Indirectly. If budget is wasted, important pages get crawled and refreshed less often, which can slow updates.
Keep reading
Related on LymLyt
Want this working on your site?
We build the content behind the term, ranked in search and cited by AI.
Book a 30-min call →