llms.txt Is Becoming a Standard: Inside the 849-Site Directory
The /llms.txt convention now spans 849 indexed sites, from Anthropic to Stripe. Here's how token counts vary, what the split reveals, and if you need one.
One directory alone lists 849 websites shipping a file called llms.txt at their root — Anthropic, Cursor, Cloudflare, and Next.js among them. Nobody standardized the format. It spread because AI coding assistants kept choking on marketing-heavy HTML, and enough engineering teams decided a plain-Markdown entry point was worth the effort.
What llms.txt actually is
The convention is simple: drop a file named /llms.txt at a site's root, written in Markdown, so a large language model gets a direct content entry point instead of having to parse HTML built for humans and ad networks. It was proposed by Jeremy Howard of Answer.AI, and it has no official governing body — which is exactly why two separate third-party directories have emerged to track adoption, according to one widely-shared roundup of the two directories.
It helps to place llms.txt next to a more familiar file: robots.txt tells crawlers what they're allowed to fetch; llms.txt tells language models where the useful content already lives. One is a permission list; the other is a curated table of contents. They solve different problems, which is why adding llms.txt doesn't mean removing anything else from a site's root.
Two directories, two very different bars
directory.llmstxt.cloud, built by @ifox and @joyceverheije, runs an actual review team and sets a bar that favors influential companies and products. Its own category counts run to 849 Websites, 447 Products, 358 Developer tools, 187 AI tools, and 167 Finance sites — and it's the directory where Anthropic, Cursor, Cloudflare, and Next.js all show up.
llmstxt.site takes the opposite approach. It's independently run, has a low bar to entry, and is larger by raw volume — more than 1,600 sites — but the roundup notes a high share of those entries are SEO or marketing pages rather than genuine documentation.
What the token counts actually look like
Pulled from named examples in the same roundup, the spread in file size is the most striking part of the dataset:
| Site | Category | llms.txt | llms-full.txt |
|---|---|---|---|
| Anthropic (anthropic.com) | AI safety lab | 8K tokens | 481K tokens |
| Sourcegraph (sourcegraph.com/docs) | Code search & navigation | 1.2M tokens | — |
| Cloudflare (developers.cloudflare.com) | Edge platform docs | 34K tokens | 3.8M tokens |
| Next.js (nextjs.org/docs) | React framework docs | 14K tokens | 676K tokens |
| Svelte (svelte.dev) | UI framework | 281 tokens | 226K tokens |
| Stripe (docs.stripe.com) | Payment API docs | 17K tokens | — |
| Better Auth (better-auth.com) | Open-source auth framework | 174K tokens | — |
| Bitcoin.com | Crypto news, wallet, exchange | 722K tokens | — |
| Chainspect (chainspect.app) | Blockchain analytics | 938K tokens | — |
| Mangopay (mangopay.com) | Embedded payments | 11K tokens | 1.7M tokens |
A map or a warehouse — the pattern splits in two
The roundup argues Sourcegraph's 1.2M-token file means the company poured its complete API docs and usage guides straight into the entry point, rather than linking out to them — useful, in its telling, for code assistants like Cursor or Copilot that call Sourcegraph directly. Cloudflare goes further: alongside a 34K-token llms.txt, it ships a 3.8M-token llms-full.txt covering Workers, Pages, R2, D1, and KV, described in the roundup as letting an AI pick a version to match its own context window.
At the other extreme, Svelte's llms.txt runs just 281 tokens — among the smallest in the whole directory. The roundup's read is that llms.txt doesn't need scale to be useful; its job is a precise pointer to the real docs, not a warehouse of them. That's the source's interpretation of a real size gap, not an independently measured outcome.
Bitcoin.com's 722K-token file sits on the warehouse end too — described in the source's words as a 'content-feeding' strategy, stuffing a large volume of news content directly into the entry point. For Chainspect's 938K-token file, the source only speculates that it contains complete on-chain data analysis methodology and API references — it doesn't apply the same content-feeding label. The write-up also singles out Next.js, arguing App Router's configuration complexity is high enough that accurate Markdown docs directly improve AI-generated code — again, the source's claim, not a benchmark anyone has published. It frames Anthropic, Cursor, and Cloudflare together as a 'closed loop': AI companies serving AI with llms.txt, because they understand best what AI needs to read. Read that as one writer's pattern-matching, not a proven causal chain.
The gap in the middle
The same source makes one more observation worth naming directly: genuine small-to-medium technical blogs are largely absent from the directory. What's there splits into two extremes — deep implementations from first-tier brands like Anthropic, Cloudflare, and Next.js, or thin marketing pages filed just to appear in a listing. Independent developer blogs, technical notebooks, and personal knowledge bases are, per the source, the layer that's missing. If that holds, an llms.txt file is currently a cheap way for a smaller docs site to stand out — precisely because so few of them have bothered yet.
Worth flagging plainly: this token census comes from a single roundup scanning one directory's snapshot, not an independently verified adoption survey. Treat every count above as a moment in time, not a settled benchmark.
For an engineering lead or DevRel team, the real question is rarely whether this is trendy — it's whether AI coding assistants and research agents are already trying to read the docs and failing. If support tickets or PR reviews show ChatGPT- or Claude-generated integration code citing stale or hallucinated parameters, that's a stronger signal to act on than any directory listing count.
Should your docs get one?
The examples above sort into a rough decision framework, not a rule:
- Go minimal, Svelte-style, if your docs are already well-organized and mostly need a pointer — a short llms.txt listing your key pages is enough for an AI agent to know where to fetch from.
- Go full-text, Cloudflare- or Next.js-style, if your product surface is large and correctness is expensive to get wrong — payment APIs, infra SDKs, anything where a hallucinated parameter costs a customer. Ship both the short file and the llms-full.txt so agents can pick based on their own context budget.
- Skip it for now if your public docs aren't stable yet. An llms.txt pointing at thin or fast-changing content adds noise, not signal.
Given the missing-middle finding above, a small team plausibly gains more from being one of the few than a first-tier brand gains from being one of many. The fastest way to test that is not committing to a giant llms-full.txt on day one — start with a short llms.txt covering your five most important doc pages, and watch whether AI coding assistants start citing them more accurately in generated output. That's the only benchmark that actually matters.
More from DangMua