Crawl budget
How many pages a search engine will crawl on your site in a given period, determined by your server's capacity and how much demand the engine has for your content.
In depth
What it really means
Crawl budget is two things multiplied: crawl rate limit, which is how hard the engine can push your server without degrading it, and crawl demand, which is how much it wants your pages based on popularity and staleness.
It is worth saying plainly that most sites should ignore this. Under roughly 10,000 URLs, crawl budget is almost never the reason a page is not indexed. It becomes real for large ecommerce, faceted navigation, and sites generating URLs programmatically.
How it works
- The engine estimates how much crawling your server can handle without slowing down.
- It estimates demand from page popularity and how often content changes.
- The lower of the two effectively caps crawling.
- Low-value URLs consume budget that would otherwise reach your important pages.
Pros & cons
Pros
- On large sites, managing it directly improves how quickly new content gets indexed.
- The audit that finds waste usually finds other technical problems too.
- Server load drops as a side benefit.
Cons
- Irrelevant for most sites, and a common distraction from real problems.
- You cannot increase demand directly, only reduce waste.
- Diagnosis needs log file analysis, which many teams cannot access.
Common mistakes
- Worrying about it on a 300-page site, where it is never the issue.
- Letting faceted navigation generate millions of parameter URLs.
- Leaving redirect chains in place, which spend budget on hops rather than pages.
- Blocking crawling with robots.txt to save budget on pages that should be noindex instead.
Best practices
- Check whether you actually have a problem before acting. Search Console crawl stats will tell you.
- Kill redirect chains and fix broken internal links.
- Block genuinely worthless URL patterns, such as internal search results and infinite filter combinations.
- Keep your XML sitemap clean and current so demand is directed at real pages.
FAQs
What is crawl budget?
The number of pages a search engine will crawl on your site in a given period, set by your server’s capacity and the engine’s demand for your content.
Do I need to worry about crawl budget?
Probably not. Below roughly 10,000 URLs it is almost never the reason a page is unindexed. Look at indexation and content quality first.
What wastes crawl budget?
Faceted navigation generating endless parameter URLs, redirect chains, soft 404s, internal search result pages, and duplicate URLs from tracking parameters.
How do I check my crawl budget?
Search Console crawl stats shows requests per day and response times. Server log files give the complete picture of what bots actually fetched.
Keep reading
Related on LymLyt
Beyond LymLyt
Further reading
Want this working on your site?
We build the content behind the term, ranked in search and cited by AI.
Book a 30-min call →