Log file analysis
Reading your server access logs to see exactly which bots requested which URLs, when, and what status code they received.
In depth
What it really means
Log files are the only complete record of crawler behavior. Search Console samples and summarizes. Logs show every single request, which means they answer questions nothing else can: which pages Googlebot has never visited, how much budget went to parameter URLs, whether AI crawlers are hitting you at all.
For AI visibility this is the cheapest measurement available and almost nobody does it. Grepping your logs for OAI-SearchBot and PerplexityBot tells you directly whether assistants treat your site as a source.
How it works
- The server records every request with timestamp, URL, user-agent and status code.
- Logs are filtered to legitimate bot user-agents, verified by reverse DNS.
- Requests are aggregated by URL, bot and status code.
- Patterns reveal crawl waste, missed pages and AI crawler activity.
Pros & cons
Pros
- The only complete and unsampled record of crawler behavior.
- Reveals crawl waste that no other tool shows.
- Directly measures AI crawler activity, which nothing else does.
Cons
- Access is often controlled by engineering or hosting.
- Raw logs need processing before they mean anything.
- Large sites generate volumes that need real tooling.
Common mistakes
- Trusting the user-agent string without verifying by reverse DNS, since it is trivially spoofed.
- Analyzing a single day, which shows nothing about patterns.
- Only looking at search bots and never checking AI crawler activity.
Best practices
- Pull at least 30 days of logs so patterns are visible.
- Verify bot identity by reverse DNS rather than trusting the user-agent.
- Segment by bot, then by status code, then by URL pattern.
- Grep specifically for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot.
- Compare crawled URLs against your sitemap to find pages bots never reach.
FAQs
What is log file analysis?
Reading server access logs to see exactly which bots requested which URLs, when, and what response they got. It is the only unsampled record of crawler behavior.
What can logs tell me that Search Console cannot?
Which pages have never been crawled, exactly how crawl budget is being spent, and whether AI crawlers are visiting at all. Search Console reports none of that directly.
How do I check if AI crawlers visit my site?
Search your access logs for the user-agent strings GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot, then verify by reverse DNS.
Keep reading
Related on LymLyt
Beyond LymLyt
Further reading
Want this working on your site?
We build the content behind the term, ranked in search and cited by AI.
Book a 30-min call →