Robots.txt
A file that tells crawlers which parts of your site they may or may not crawl.
In depth
What it really means
Robots.txt is a plain text file at your site root that gives crawlers instructions on which paths they may or may not crawl. It is a crawl-control tool, not a security or indexing guarantee.
A common, costly mistake is accidentally blocking important sections, or assuming robots.txt hides a page. Blocked pages can still be indexed if linked elsewhere.
Best practices
- Keep robots.txt minimal and deliberate.
- Never block CSS, JS or important content by accident.
- Use it to stop crawling of low-value paths, not to hide secrets.
- Reference your sitemap in it.
- Test changes before shipping them.
FAQs
What is robots.txt?
A file at your site root that tells crawlers which parts of your site they may or may not crawl.
Does robots.txt stop indexing?
No. It controls crawling, not indexing. A blocked page can still be indexed if other pages link to it.
How do I actually keep a page out of search?
Use a noindex directive or authentication, not robots.txt, which only controls crawling.
What’s a common robots.txt mistake?
Accidentally blocking important content or resources like CSS and JS, which can harm rendering and rankings.
Should I list my sitemap in robots.txt?
Yes, adding the sitemap URL helps crawlers discover your important pages.
Keep reading
Related on LymLyt
Want this working on your site?
We build the content behind the term, ranked in search and cited by AI.
Book a 30-min call →