AI crawler
A bot that fetches web pages so AI systems can train on them or read them at answer time. GPTBot, ClaudeBot and PerplexityBot are the ones most publishers see in their logs.
In depth
What it really means
AI crawlers do two different jobs, and the difference matters. Training crawlers collect text to build the next version of a model. Retrieval crawlers fetch pages live, in the seconds after someone asks a question, so the assistant can quote something current. Blocking one does not block the other, and most robots.txt files written before 2024 block neither.
If you want to appear in AI answers, the retrieval bots need to reach you. Block them and you are invisible to that assistant no matter how good the page is. This is the single most common reason a well-written page never gets cited, which is why it leads our guide to why sites don’t show up in ChatGPT.
How it works
- The bot requests your robots.txt and checks whether its user-agent token is allowed.
- If allowed, it fetches the HTML. Most AI crawlers do not run JavaScript, so client-rendered content is often invisible to them.
- The text is stripped of navigation and boilerplate, then split into passages. See content chunking.
- Passages go either into a training corpus or into a live retrieval index, depending on the bot.
- At answer time a retrieval bot may fetch the page again to get the current version.
Pros & cons
Pros
- Allowing retrieval bots is the entry ticket to being cited in AI answers.
- Server logs show exactly which AI systems read you and how often, which is free measurement most brands ignore.
- Access costs nothing and needs no ongoing work once robots.txt is right.
Cons
- Your content trains commercial models you get nothing back from.
- Answers can satisfy the user without a click, so traffic falls even as visibility rises.
- Crawler traffic adds server load, which is real for large sites.
Common mistakes
- Treating all AI bots as one decision. Blocking training crawlers is defensible. Blocking retrieval crawlers removes you from answers.
- Blocking by accident. A CDN bot-protection rule or a WAF default can block AI user-agents without anyone choosing it.
- Blocking GPTBot and assuming ChatGPT cannot see you. ChatGPT’s live browsing uses OAI-SearchBot and ChatGPT-User, which are separate tokens.
- Relying on JavaScript to render the content you want cited.
- Never checking the logs, so you have no idea whether any of this is working.
Best practices
- Decide training access and retrieval access separately, and write both into robots.txt deliberately.
- Allow OAI-SearchBot, ChatGPT-User, PerplexityBot and Google-Extended if AI visibility is a goal.
- Serve your important content in server-rendered HTML.
- Grep your access logs monthly for AI user-agents and track the trend.
- Keep pages fast. Retrieval bots work against a timeout and will give up on a slow page.
FAQs
What is an AI crawler?
A bot that fetches web pages for AI systems, either to build training data or to retrieve live sources at answer time. GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot are common examples.
How is an AI crawler different from a search crawler?
A search crawler builds an index that ranks pages as links. An AI crawler collects text that gets summarized into an answer, often with no click. Same fetch, very different outcome.
Should I block AI crawlers?
Block training crawlers if you object to your work training commercial models. Keep retrieval crawlers open if you want to appear in AI answers. They are separate choices controlled by separate tokens.
Is Google-Extended a crawler?
No, and this trips people up. Google-Extended is a standalone product token in robots.txt, not a bot. Googlebot does the fetching. Google-Extended controls whether your content helps improve Gemini Apps and the Vertex AI Gemini APIs. Google states plainly that it does not affect your inclusion or ranking in Google Search, so blocking it is a low-risk choice if you want to opt out of Gemini training.
How do I see which AI bots visit my site?
Search your server access logs for the user-agent strings: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Bytespider, Applebot-Extended. Frequency tells you which assistants treat you as a source.
Keep reading
Related on LymLyt
Beyond LymLyt
Further reading
Want this working on your site?
We build the content behind the term, ranked in search and cited by AI.
Book a 30-min call →