An llms.txt file is a plain markdown file at /llms.txt that lists the pages a site most wants an AI system to read. It was proposed in 2024 to give language models a clean map of a site. TechMachaw checked 60 marketing, martech and creator-tool domains to see how many publish one.
Of the 52 domains that gave a clean answer, 21 publish an llms.txt file and 31 do not. That is 40.4 percent, spread very unevenly. The catch matters more than the number: no major AI company has publicly confirmed reading these files, and Google has said the opposite outright.
Key takeaways
- 60 domains checked on 19 September 2026. Eight blocked us, so 52 resolved.
- 21 of 52 resolved domains publish an llms.txt file: 40.4 percent.
- Martech and SaaS lead at 66.7 percent. Ad platform and Indian marketing sites sit at 12.5 percent.
- The median file lists roughly 50 links, the largest close to 1,000, the smallest 9.
- Google has stated that no AI system uses llms.txt, and no vendor has confirmed reading it.
- Treat it as housekeeping, not a ranking lever.
What is llms.txt and who proposed it?
llms.txt is a markdown file at the root of a domain giving AI systems a curated index of a site. Jeremy Howard of Answer.AI proposed it in September 2024. The format is an H1 with the site name, an optional summary in a blockquote, then H2 sections holding markdown links, plus an Optional section for links an agent can skip.
Howard proposed it in September 2024 to solve a practical problem. Web pages are full of menus, cookie notices and promotional blocks, so a model reading raw HTML spends most of its context window on material unrelated to the question.
In the llms.txt specification, the H1 is the only required element. The Optional section has a defined meaning: links an agent can drop when it needs a shorter context, which shows the file was designed around token budgets, not rankings. A related file, llms-full.txt, is not part of the proposal.
How did we run this study?
We built a list of 60 domains across five categories, requested https://domain/llms.txt for each on 19 September 2026 by automated HTTP fetch, and recorded whether a real markdown file came back. Blocked requests were logged separately from genuine 404s and excluded from the rate.
The sample was saved before any fetching and not changed afterwards: martech and SaaS tools (16 domains), marketing media and blogs (12), ad platform marketing sites (10), creator tools (12) and Indian marketing sites (10). For each we recorded whether a file came back, its approximate size and link count, its section headings, any generator line, and whether /llms-full.txt existed.
Eight domains never gave a clean answer: two disallowed by robots.txt, three returning HTTP 403, two blocked by the target site, one timed out. Those are excluded from the percentages. Four other domains returned a normal HTML page at that address, a soft 404, counted as not published.
How many marketing sites publish an llms.txt file?
21 of 52 resolved domains published a real llms.txt file on 19 September 2026, an adoption rate of 40.4 percent. Divide instead by all 60 domains attempted, including the eight that blocked us, and the rate is 35.0 percent. We report the first, because a blocked request is not evidence of absence.
Formula: Adoption rate = Domains with a file / Domains that gave a clean answer
Worked example: 21 files, 52 clean answers, so 21 divided by 52 is 0.4038 or 40.4 percent. Counting the 8 blocked domains as failures makes the denominator 60 and the rate 35.0 percent. That 5.4 point gap is the size of one measurement decision.
Which categories are ahead and which are behind?
Martech and SaaS tools lead at 66.7 percent and creator tools follow at 60.0 percent. Marketing media sits at 27.3 percent. Ad platform marketing sites and Indian marketing sites both come in at 12.5 percent. Companies that sell software are adopting it. Publishers largely are not.
| Category | Domains tried | Clean answers | Published llms.txt | Adoption |
|---|---|---|---|---|
| Martech and SaaS tools | 16 | 15 | 10 | 66.7% |
| Creator tools | 12 | 10 | 6 | 60.0% |
| Marketing media and blogs | 12 | 11 | 3 | 27.3% |
| Ad platform marketing sites | 10 | 8 | 1 | 12.5% |
| Indian marketing sites | 10 | 8 | 1 | 12.5% |
| All categories | 60 | 52 | 21 | 40.4% |
Software companies already maintain developer documentation and already have someone doing technical site hygiene, so one more file costs almost nothing. Publishers sit differently: many are in disputes about AI systems using their archives. Of the three marketing publications that published a file, two were full article dumps.
The ad platform result is the most striking: across eight ad platform sites with a clean answer, exactly one published a file. These are the companies with the deepest investment in AI.
What is actually inside a good llms.txt file?
The files we found are mostly short curated indexes of roughly 7,000 to 20,000 characters listing around 50 links. The most common sections are resources and learning material, product and feature pages, solutions, company and legal pages, and developer access. Pricing appears in fewer than one in five.
The median file listed roughly 50 links, with a huge spread: the smallest listed 9 links in about 1,200 characters, the largest close to 1,000 across roughly 87,000 characters. Median size was around 12,000 characters, and all of these are approximate.
| Section type | Files with it | Share of the 21 | What it usually holds |
|---|---|---|---|
| Resources, learning and blog | 16 | 76% | Guides, glossaries, case studies, reports, webinars |
| Products and core features | 13 | 62% | Main product pages and feature explainers |
| Solutions and use cases | 8 | 38% | Pages split by industry, team or role |
| Company, legal and trust | 8 | 38% | About, careers, privacy, security, contact |
| Developer and agent access | 8 | 38% | API docs, SDKs, changelog, agent instructions |
| Optional or sitemap pointer | 8 | 38% | Skippable links, or the XML sitemap |
| Comparisons and competitors | 7 | 33% | Versus and alternatives pages |
| Pricing | 4 | 19% | Plan and pricing pages |
The best files gave the model instructions, not just links: one opened with a section on how to read the site as a machine, another with a canonical policy and a list of retired product names. That is the real idea, a short briefing note for a reader that will quote whatever it finds first.
Are these files hand written or plugin generated?
Of the 21 files found, 15 looked hand written and curated, 4 were automated dumps of every page on the site, and 2 carried a generator line naming Yoast SEO. The curated files were far shorter and far more useful than the dumps.
- The curated index (15 of 21). Site name, a short summary, then 30 to 120 links in labelled sections, with notes written for the file rather than copied from meta tags.
- The full archive dump (4 of 21). Every post under one heading with dates and excerpts. Two ran to roughly 185,000 and 190,000 characters, and one listed close to 1,000 pages using the site’s own meta descriptions as notes.
- The plugin output (2 of 21). Both named Yoast SEO, one version 27.5 and one version 25.3, both shaped the same way: a sitemap pointer, site name and summary, then sections for pages, posts and taxonomies. One was tiny, about 1,200 characters and 13 links.
We also checked /llms-full.txt. Of the 21 domains with an llms.txt file we got a clean answer on 17 and found llms-full.txt on exactly one, which fits its absence from the specification.
Do any AI companies say they actually use llms.txt?
No major AI company has publicly confirmed that it reads llms.txt when crawling or answering. Google has said the opposite outright. OpenAI documents robots.txt, user agents and IP ranges as publisher controls and does not list llms.txt among them. Several AI companies publish llms.txt files for their own docs, which is not the same as consuming them.
Google’s John Mueller said in June 2025 that no AI system currently uses llms.txt, a flat public statement not replaced since by a contrary official one. We also checked the Google crawler documentation on 19 September 2026: it does not mention llms.txt, and the file it names for controlling crawlers is robots.txt.
The OpenAI crawler documentation lists four crawlers and three publisher controls: robots.txt directives, user agent strings and published IP ranges. It does link an llms.txt file, but that is OpenAI’s own documentation index, not something it reads on your site.
The specification site notes that OpenAI, Anthropic and Gemini publish llms.txt files for their developer documentation. That gets quoted as proof of support, and it is not: publishing a file for coding assistants says nothing about what your crawler reads elsewhere.
There is one counterweight. The Chrome Lighthouse agentic browsing audit includes an llms.txt check under discoverability, flagging a page when a server error occurs while retrieving the file. It names no system that reads the file and makes no claim about rankings. So a Google-built tool audits for it while a Google spokesperson says no AI system uses it. The format is being kept warm rather than adopted.
Our study cannot settle this. We measured what sites publish, not what crawlers fetch, and the second question needs server logs.
Is llms.txt worth doing today?
Yes for most sites, for small and honest reasons: it costs an hour, it cannot hurt you, it forces you to decide which 40 pages matter, and agent tooling has started checking for it. No, if you expect traffic, citations or rankings from it, because there is no public evidence of that effect.
| Reason to do it | Reason to hold off |
|---|---|
| It takes about an hour and costs nothing to host. | No AI vendor has confirmed reading it, and Google says none does. |
| Chrome Lighthouse now audits for it under agentic browsing. | An audit checking the file loads is not evidence a model uses it. |
| It forces a decision about your 30 to 50 most important pages. | You can make that decision without publishing a file. |
| Two in three software companies in our sample have one. | Publishers mostly do not, and they have reasons. |
Our verdict: do it, timebox it to one afternoon, keep it under 60 links, review it twice a year, then spend your time on what affects whether AI systems quote you, starting with what generative engine optimization means and getting your brand mentioned in AI answers.
Our study of 56 top-ranking marketing pages found that favoured pages have clear structure, direct answers near the top, real sources and current dates. An index file does not fix a page that fails to answer the question.
How do you add an llms.txt file to your site?
Create a plain text file named llms.txt, serve it at your root domain with a text content type, start with an H1 of your site name, add a short summary in a blockquote, then group 30 to 60 links under H2 headings. On WordPress, Yoast SEO can generate one for you.
- List your best 30 to 60 pages: core explainers, tools, pricing, policies. Leave out tag archives.
- Write the header. One H1 with your site name, then a blockquote of two or three sentences saying what the site is and who it is for.
- Group links under H2 headings with plain labels: Products, Guides, Tools, Pricing, Company, Legal.
- Write a real note after each link: a markdown link, a colon, a short description written for this file. Do not paste meta descriptions.
- Add an Optional section for skippable links. It is the one section with a defined meaning in the spec.
- Upload it and check it. It must answer at yourdomain.com/llms.txt as text, not HTML. If a private window shows your site header, it is not working.
On WordPress with Yoast SEO, the plugin can generate and serve llms.txt for you, and both generated files in our sample named the plugin version. The trade-off is control: the plugin decides which pages to include, which is why one listed only 13 links.
Keep the language plain and the terms consistent. If your site uses a term unusually, define it, which is what our digital media planning glossary is for.
What this means for marketers and creators
What this means for marketers and creators
For marketers: treat llms.txt as a one-hour housekeeping task, not a channel. Add it, keep it short, and do not put it in a deck as an AI visibility initiative, because you cannot show it moved anything. The by-product is the exercise: naming your 40 most important pages.
For creators: the payoff is smaller and so is the effort. A 10 to 20 link file naming your best work, your about page and your contact details takes fifteen minutes. Six of the ten creator platforms we could check already publish one, and the smallest useful file listed just 9 links. Spend the rest making pieces quotable: clear headings, a direct answer early, your credentials on the page.
For both: run a page through our free GEO readiness score checker and fix what it flags first.
Common mistakes to avoid
- Serving HTML instead of text. Four domains returned a web page at the llms.txt address. To an agent that is a failure, not a file.
- Dumping every page. A 190,000 character index defeats the purpose, and a file listing pages you deleted is worse than no file.
- Expecting it to do the page’s work. An index cannot make a thin page quotable. See the Google guidelines on AI generated content for what counts as useful content.
What we did not test
We measured publication, not consumption. We did not check server logs, did not test whether any AI system quotes a site more after adding the file, used one sample of 60 domains on one day, and our counts are approximate rather than byte-exact.
- Approximate counts. Sizes and link counts were read off each fetched file, not measured byte by byte, so read the median of about 50 links as a rough centre.
- One day, one sample. This is a snapshot from 19 September 2026, eight blocked domains are failures to measure rather than absences, and a category result of 12.5 percent rests on just 8 domains.
- The sample is not random. Well-known names are more likely to have someone doing technical site hygiene, so the true rate across all marketing sites is probably lower.
- No effect measurement. This data cannot show that publishing the file changes anything, and we checked root domains only, so a file on a documentation subdomain would be missed.
Frequently asked questions
What is an llms.txt file?
An llms.txt file is a plain markdown file served at yourdomain.com/llms.txt. It gives AI systems a curated index of the pages a site most wants them to read: an H1 site name, a short summary, then grouped link lists. Jeremy Howard proposed it in September 2024.
Does Google use llms.txt?
No. Google’s John Mueller stated publicly in June 2025 that no AI system currently uses llms.txt, and Google’s crawler documentation does not mention the file. Chrome Lighthouse includes an llms.txt check in its agentic browsing audits, but it only checks that the file loads.
Does ChatGPT read llms.txt?
OpenAI has not confirmed that it does. Its crawler documentation lists robots.txt directives, user agent strings and published IP ranges as the publisher controls, and llms.txt is not among them. It publishes one for its own developer docs, which is not the same as reading yours.
How many marketing sites have an llms.txt file?
In our check of 60 marketing, martech and creator-tool domains on 19 September 2026, 21 of the 52 that gave a clean answer published one, which is 40.4 percent. Martech and SaaS reached 66.7 percent, while ad platform and Indian marketing sites sat at 12.5 percent.
Can Yoast SEO generate an llms.txt file?
Yes. Yoast SEO can generate and serve an llms.txt file on WordPress, and both plugin-generated files we found named the plugin version. The trade-off is control: the plugin decides which pages to include, and one listed only 13 links.
Is it worth adding an llms.txt file in 2026?
For most sites yes, but for modest reasons. It takes about an hour, it cannot harm you, agent tooling has started checking for it, and writing it forces a useful decision about your key pages. Do not expect traffic or rankings from it.
Next steps
Spend one afternoon on this and then move on. Write the file, keep it under 60 links, check it loads as text, set a review date six months out, then put your effort into the work with evidence behind it. Run a page through our free GEO readiness score checker to see where you stand.
For more studies like this, join the free TechMachaw newsletter using the signup form on this page.
