Robots.txt
A file at your site root telling crawlers which paths they may and may not crawl. It controls crawling, not indexing, and those are different things.
In depth
What it really means
Robots.txt is the first thing almost every bot requests. It is also the single most dangerous file on your site, because one stray Disallow line can remove you from search entirely and nothing will alert you.
The most common misunderstanding is worth stating plainly: blocking a page in robots.txt does not remove it from the index. It prevents crawling, which means Google never reads your noindex tag and the URL can keep appearing without a description. To deindex, allow crawling and use noindex.
How it works
- A bot requests /robots.txt before crawling.
- Directives are matched per user-agent, with the most specific group winning.
- Disallowed paths are not fetched, but may still be indexed if linked from elsewhere.
- AI crawlers use their own user-agent tokens, separate from search crawlers.
Pros & cons
Pros
- Universally respected by legitimate crawlers.
- Controls both search and AI crawler access in one place.
- Simple to write and instantly effective.
Cons
- A single wrong line can deindex a site silently.
- It cannot deindex anything, which people persistently expect it to.
- Malicious bots ignore it entirely.
Common mistakes
- Disallow: / left over from staging, the classic catastrophic error.
- Using it to remove pages from the index, which does the opposite of what is intended.
- Blocking CSS and JS, which stops Google rendering the page properly.
- Blocking all AI crawlers without distinguishing training bots from retrieval bots.
Best practices
- Review it after every site migration and every staging deployment.
- Use noindex, not Disallow, when the goal is removal from the index.
- Keep CSS and JavaScript crawlable.
- Set AI crawler rules deliberately, treating training and retrieval as separate decisions.
- Link your XML sitemap from it.
FAQs
What is robots.txt?
The first file almost every bot requests, listing which paths each crawler may fetch. It governs crawling only, which is the distinction that causes most of the trouble with it.
Does robots.txt remove pages from Google?
No. It prevents crawling, which means Google cannot read a noindex tag on that page. A blocked URL can still appear in results without a description.
How do I block AI crawlers?
Add Disallow rules for their specific user-agent tokens. Training bots and retrieval bots use different tokens, so decide each separately.
What is the most dangerous robots.txt mistake?
Shipping Disallow: / from a staging environment to production. It removes the whole site from crawling and nothing warns you.
Keep reading
Related on LymLyt
Beyond LymLyt
Further reading
Want this working on your site?
We build the content behind the term, ranked in search and cited by AI.
Book a 30-min call →