llms.txt: What It Is and Whether You Need One

llms.txt is a plain markdown file, placed at the root of a website, that gives AI systems a curated index of a site’s most important pages. It’s been recommended across dozens of GEO checklists for two years running. It’s also, according to the largest server-log study done on it, sitting unread on 97% of the sites that have one.

Both of those things are true at once. This is the honest version: what the file actually is, what the adoption and traffic data actually show, and when building one is worth your time.

What llms.txt Actually Is

The format was proposed on September 3, 2024, by Jeremy Howard, co-founder of Answer.AI and fast.ai. The problem it set out to solve was narrow: a language model’s context window is too small to hold an entire website, and converting a normal HTML page (navigation, ads, JavaScript, and all) into clean text is slow and error-prone.

The fix is a single file at /llms.txt: one mandatory H1 heading, an optional one-line summary, and a set of H2-delimited link lists pointing to a site’s most important content, each with a short description. A companion file, llms-full.txt, can hold the actual page content inline for an agent that wants everything in one fetch.

Howard’s own framing was always narrow, too. It’s built for inference-time use by coding agents and dev tools reading documentation, not for winning citations in ChatGPT or Google’s AI Overviews. That distinction got lost somewhere in 2025.

How It Differs From robots.txt and a Sitemap

robots.txt controls permission: which crawlers can access which parts of a site. A sitemap lists URLs so a crawler knows what exists. llms.txt does neither. It’s a curated summary, an opinion about what matters most, handed to any model willing to read it.

None of the three formats overlap in function, and having one doesn’t substitute for the others. If your AI-facing content is client-side rendered, an llms.txt file won’t fix that; our guide to AI crawler rendering covers the actual fix for that separate problem.

What the Adoption Data Actually Shows

Multiple independent studies converge on the same rough number, even though their methods differ.

Rankability’s monthly Tranco top-1,000 tracker put adoption at 8.7% as of June 2026. SE Ranking’s analysis of roughly 300,000 domains found 10.13% overall adoption in May 2026, up from 0.4% in April 2025.

Both studies agree on something else: adoption barely varies by site size. Large, high-traffic domains are not meaningfully more likely to have the file than small ones.

Growth is real. What exactly that growth means is a separate question entirely.

Whether Anyone Is Actually Reading It

This is the number that matters more than the adoption rate. Ahrefs analyzed server logs across 137,210 domains with measurable traffic in May 2026, published June 15, 2026. Of the roughly 38,000 domains with a valid llms.txt file, 97% received zero requests to that file all month. No bots. No humans. Nothing.

Of the small fraction that did get traffic, 96% of the requests came from bots, and most of those weren’t AI systems. SEO audit tools accounted for 21% of requests, more than any single AI retrieval bot.

Named AI tools made up roughly 19.5% of the traffic that did occur, and much of that was coding-adjacent crawlers like Claude-Code rather than the retrieval systems behind AI search answers.

Ahrefs also found something worth flagging directly: no AI bot ever requested an llms.txt file on a domain where one didn’t exist. Nothing is out there probing for it.

What Google Says, Directly

Google addressed this head-on. Its own AI optimization guide, updated June 15, 2026, states plainly that special files aren’t part of how Search works:

“You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.”

The same guide adds that keeping one anyway is harmless, just inert for Google specifically: maintaining an llms.txt “won’t harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them.” A Google Search Advocate separately described the idea as speculative, comparing it to the old meta keywords tag once stuffed by spammers and eventually ignored entirely.

Does It Help With Anything?

SE Ranking ran an XGBoost model to test whether having an llms.txt correlates with AI citation frequency, controlling for site authority, schema density, and content recency. Removing the llms.txt variable from the model improved its prediction accuracy. The file added noise, not signal.

That’s the search-citation side settled. The developer-tooling side is genuinely different. Notable adopters, including Anthropic, Stripe, Cursor, Mintlify, and Cloudflare, are overwhelmingly documentation-heavy, API-first companies. For a coding agent pulling up a library’s API reference inside Cursor or Windsurf, a clean markdown index is a real convenience. That was always the actual use case.

If you’re trying to figure out whether any of this is actually moving your citation numbers, our roundup of the best AEO tools for B2B SaaS covers the current field for tracking it directly, rather than guessing from a checklist.

What Actually Correlates With Citation Instead

The same SE Ranking study measured what does move AI citation rates, and the gap versus llms.txt is not close. FAQPage schema with genuine, specific question-and-answer pairs correlated with a 34% citation lift in Perplexity and 28% in ChatGPT.

ClaimReview markup on stat-dense content showed a 41% lift in AI Mode specifically. Organization SameAs linkages, which disambiguate a brand’s identity across the web, correlated with a 22% lift.

Our guide to Google’s Knowledge Graph covers the entity-clarity work behind that last finding in more depth. None of it requires a new file format. It requires structured data most sites already have the tools to add.

Should You Build One Anyway?

Build it if your site is genuinely documentation-heavy: API references, SDKs, developer guides, the kind of content a coding agent needs to read quickly. Skip it if your site is a standard marketing or content site chasing AI Overview or ChatGPT citations specifically, since the evidence says it won’t move that number.

If you fall into the first group, keep it lean: one clear H1, a one-line summary, and links to your actual documentation, not a restatement of your homepage copy. Version-control it like any other piece of infrastructure, and don’t let a CMS auto-generate one you never review.

Our B2B SaaS SEO strategy guide covers where a file like this fits into a broader technical foundation. It’s a small piece of that picture, not the center of it.

Frequently Asked Questions

Does ChatGPT or Perplexity read llms.txt?

Rarely, based on the available evidence. Ahrefs’ server-log analysis found AI retrieval bots accounted for roughly 1% of all requests to llms.txt files, with the traffic that did occur dominated by SEO audit tools checking whether a file exists, not AI systems reading it for answers.

If your content isn’t showing up in ChatGPT at all, an llms.txt file is rarely the actual cause. Our piece on 5 reasons your site doesn’t show up in ChatGPT covers the failure modes that usually are.

Is llms.txt the same as GEO?

No. It’s one narrow technical tactic that got folded into the broader GEO conversation. Our breakdown of GEO vs AEO vs SEO covers how that terminology actually works and where a tactic like this one fits.

Will having an llms.txt hurt my SEO?

No. Google’s own guidance states explicitly that maintaining one neither helps nor hurts Search visibility, since Google Search ignores the file entirely.

Is llms.txt worth building for a B2B SaaS marketing site?

Generally, no, based on the evidence available in mid-2026. The correlation studies find no citation lift, and the traffic data shows AI retrieval systems mostly aren’t requesting the file. Time is better spent on FAQPage schema, entity clarity, and the technical foundation covered in our AI visibility guide for B2B SaaS.

Should companies with heavy API documentation still build one?

Yes, that’s the one case the evidence actually supports. Coding agents and dev tools are the documented, intended audience, and the format was built for exactly that use case from the start.

If you want help figuring out which of these actually matter for your own site instead of guessing from a checklist, book a free 30-minute call.