polski
español
Magdalena Szypulska
The idea seems simple enough: instead of hoping a language model finds the most important content on its own, a site owner can hand it a guide to their site.
But as the standard has grown in popularity, the claims about it have gotten bolder. Some call llms.txt "robots.txt for AI." Others suggest it improves your odds of being cited by ChatGPT or Perplexity.
The problem is that opinions are multiplying faster than evidence.
Server log analyses suggest that most published files are barely ever fetched, and Google hasn't confirmed that llms.txt plays any meaningful role in its systems.
That gap between hype and evidence is exactly what makes llms.txt one of the more interesting topics at the intersection of SEO and AI right now. So instead of jumping straight to "does llms.txt work," it's worth looking at what we actually know.
Where did the standard come from? What does its author actually propose? How are AI model providers responding to it? And does the available data really back up the narrative that's built up around it?
Where Did the Idea for llms.txt Come From?
The standard wasn't created by Google or OpenAI, and its origins have nothing to do with search engines.
It comes from Jeremy Howard, co-founder of Answer.AI and creator of fast.ai, who published the first version of the spec in September 2024.
Howard wasn't trying to invent a new ranking signal or an alternative to robots.txt. His starting point was a much more practical problem: modern websites are designed primarily for humans and browsers, while language models need the simplest possible path to the actual content that matters.
In the spec's description, Howard points out that a single web page often contains a huge amount of material that adds zero value from a model's perspective. Menus, footers, ads, JavaScript, banners, and heavy navigation all inflate the number of tokens needed to process a document, making it harder to reach the actual knowledge inside.
That observation is what led to llms.txt. The original goal wasn't SEO — it was making it easier for language models to find the most important content on a site.
The Problem Jeremy Howard Was Trying to Solve
The project's premise can be summed up in one sentence: since language models read the web differently than people do, maybe they need a different way to navigate it too.
Howard's proposal: a site owner writes a simple text file listing the site's most important resources, each with a short description. Instead of parsing the entire structure of a site, an AI agent gets a ready-made list of where to start.
That distinction matters. The spec doesn't propose replacing a site's existing structure or building a new indexing mechanism. The file is only meant to point to whichever documents the author considers most valuable.
You could say llms.txt proposes an additional layer of communication between a site owner and AI systems. Howard himself describes it as a guide to a site — not a new way of publishing content.
Is llms.txt an Official Web Standard?
Not yet.
This is one of the most commonly overlooked distinctions in discussions about llms.txt. Despite its growing popularity, the project hasn't been adopted by standards bodies like the W3C or IETF. That means it has no official status as a web specification, and implementing it is entirely voluntary.
That has real consequences. No AI model provider is obligated to read the file. No site owner is obligated to publish it. The whole project's success depends entirely on whether both sides find it useful.
That's what separates llms.txt from standards that eventually became a fixed part of internet infrastructure. Right now, llms.txt should be treated as an open proposal from the AI community — not a binding web standard.
What Does the llms.txt Spec Actually Look Like?
Simplicity was one of the project's core goals.
Per the spec, the file should live at /llms.txt in a domain's root directory. Its content is written in Markdown, not XML or JSON — a deliberate choice. Howard argues Markdown is one of the formats language models recognize best, since so much technical documentation is already written in it.
At minimum, the document includes a project name, a short site description, and sections grouping the most important resources with brief descriptions and Markdown-formatted links.
What the spec recommends against is just as telling. Howard doesn't suggest copying an entire sitemap or listing hundreds of URLs. The idea is the opposite: a short, hand-picked set of the most important materials. llms.txt is meant to read like a carefully curated table of contents, not an auto-generated list of every page on a site.
Why Isn't This Just Another sitemap.xml?
At first glance the two look similar — both contain links to pages. That's about where the similarities end.
sitemap.xml helps search engines discover URLs to crawl and index. In practice it often lists thousands of pages and is generated automatically by a CMS or SEO tool.
llms.txt does something completely different. It doesn't try to describe an entire site. It's meant to point to a small number of materials that, in the author's judgment, best represent the knowledge the site contains.
The difference isn't just technical — it's philosophical. A sitemap answers "what's on this site?" llms.txt tries to answer "where should you start?"
Actual Adoption of llms.txt: Industry Enthusiasm Is One Thing, AI Usage Is Another
A rising number of implementations doesn't mean a rising amount of usage
The number of domains publishing llms.txt is growing fast. In the roughly 15 months since the spec was published, the file has shown up on hundreds of thousands of sites, and popular SEO tools and documentation platforms have started offering it as a built-in generator.
That doesn't mean language models are actually using it. Site owners implementing the file and AI systems actually reading it are two separate questions — and that gap is really the whole debate.
BuiltWith shows the scale of adoption
BuiltWith's data confirms rapid growth in the number of sites with an llms.txt file. But that only shows that the technology is present — not whether AI models are using it. SEO has a long history of tactics that got adopted en masse mainly because they were bundled into CMS platforms and plugins. A rising implementation count isn't, by itself, evidence that the standard matters in practice.
Ahrefs checked server logs
Ahrefs ran a much more revealing analysis, based on HTTP logs from more than 137,000 domains. Instead of counting implementations, they checked whether llms.txt files were actually being fetched.
The results were sobering: roughly 97% of the files received zero HTTP requests during the study period. That doesn't prove AI models never use them, but it does show that actual usage of the standard is far from widespread today. What's more, a large share of the traffic that was recorded came from monitoring tools and technology scanners — not from systems generating answers for real users. A file being downloaded doesn't necessarily mean a language model made use of it.
OtterlyAI reaches similar conclusions
OtterlyAI published comparable findings. Looking at more than 60,000 AI bot visits, requests for llms.txt accounted for only about 0.1% of that traffic.
Two independent studies point to the same conclusion: llms.txt implementations are growing fast, but there's currently little evidence of widespread use by AI systems.
What Does Google Say About It?
Google's position is far more cautious than the industry conversation around it. John Mueller has said Google's systems can analyze page structure on their own and don't need an extra file to understand a page's content. Google hasn't communicated any ranking benefit tied to implementing llms.txt.
Google has also pointed out that the file's content is written by the site owner — meaning it's a self-reported signal. That kind of information is easy to manipulate, so search engines tend to weight signals they observe independently more heavily than claims made by page authors themselves.
Right now, there's no basis for treating llms.txt as equivalent to robots.txt or sitemap.xml, or as a ranking factor on Google.
If Models Can Already Read HTML, Why Do They Need Another File?
This is the skeptics' central argument
Discussions about llms.txt often boil down to one assumption: if language models get a "better map" of a site, they should produce better answers.
The problem is that modern models already have very sophisticated tools for parsing documents. Both search systems and language models can distinguish main content from navigation, ads, footers, and interface elements — that's not some new AI-specific skill; classic search engines have used similar techniques for years.
If a model can already identify the right part of a document on its own, it's fair to ask what additional value a hand-curated guide really adds. This is precisely the sticking point that keeps coming up between supporters and skeptics of the standard: the better models get at parsing pages, the harder it becomes to justify an extra descriptive layer on top.
Why the Distinction Between a Training Crawler and a RAG System Matters
Not every AI system visits a site for the same reason
A training crawler works much like a classic search-indexing bot: its job is to gather a huge volume of documents that will later be used to train a model. In that scenario, what matters most is the ability to pull large datasets and clean them up afterward.
Retrieval-Augmented Generation (RAG) systems work completely differently. They're not trying to build a full copy of the internet. Their job is to find the right documents at the moment a user asks a specific question.
This distinction changes how you should think about llms.txt — whether the standard is useful at all depends heavily on which stage of a model's workflow you're talking about.
In RAG systems, speed is what matters
Modern AI search tools generally work in stages: first they find potentially relevant documents, then fetch their content, then hand that content to the language model that generates the actual answer.
That raises an obvious question: if a system already knows the answer lives at a specific URL, why would it make an extra HTTP request just to fetch llms.txt, read a list of recommended materials, and then go right back to the page it already found?
This is where the biggest doubts about the standard's practical usefulness come from. Every extra step adds latency and adds another operation for the system to perform. In an environment where milliseconds matter, that's not a trivial argument. If an AI search tool already knows the right document, an extra guide file may simply be redundant.
There are scenarios where llms.txt makes more sense — developer tools are the best example
The fact that Google is cautious doesn't mean the whole standard is worthless. Quite the opposite — research suggests the strongest interest in llms.txt isn't coming from classic search engines but from the developer-tools ecosystem.
Coding assistants, task-executing agents, and systems that parse technical documentation typically work with much smaller, more focused datasets than general web search does. In that context, a hand-curated guide to a documentation set really can cut down the time it takes to find the right information.
It's no coincidence that some of the earliest adopters of llms.txt were API documentation sites, developer libraries, and open-source projects — an environment that fits Howard's original premise far better than a news site or an e-commerce store does. The most compelling use cases for llms.txt today are in technical documentation, not broad-based SEO.
Separating the Standard's Potential From Its Current Role
Weighing the Ahrefs data, the OtterlyAI analysis, and Google's stated position together, it's hard to argue that llms.txt has become a widely used piece of AI infrastructure. It's just as hard to call it a pointless project.
The spec solves a specific, real problem its author described, and it has real applications in specific scenarios — especially where models are working with technical documentation or performing agentic tasks. That's not the same thing as confirming broad adoption by the AI search engines handling millions of queries a day.
That distinction is central to the whole debate. Instead of asking whether llms.txt "works," it's more useful to ask who it was designed for and in what environment. Only then can you fairly assess its actual potential.
The Case For and Against llms.txt: Why Is the SEO Industry Split?
The controversy around llms.txt isn't really about the spec itself — the format is simple and easy to implement. The disagreement is about whether it solves a real problem for today's language models.
You can think Howard's idea is logical and still doubt that today's AI systems actually need it. In practice, the question isn't whether llms.txt is a valid standard — it's whether models have any reason to use it.
Websites are built for people, not models
Supporters point out that modern web pages are full of elements that are just noise to a language model. llms.txt is meant to shorten the path to the most important material and flag priorities — not replace HTML or sitemap.xml. In this view, the standard acts as a guide, not a new publishing format.
Models are getting better at doing without it
Skeptics point out that modern models are increasingly good at recognizing document structure on their own and separating real content from navigation elements. In RAG-based systems, reading an extra file means another HTTP request and added processing cost. If a model has already found the right document, an extra guide may just be dead weight.
Both sides have a point
Current evidence suggests llms.txt may be worthwhile in specific use cases — technical documentation and agentic systems, for example — while there's no evidence it plays a meaningful role in mainstream AI search today. That makes it worth treating as a promising experiment, not a mandatory piece of technical SEO.
What Can We Actually Confirm?
The standard exists and is gaining traction quickly. There's no question llms.txt is an active, evolving project. Since the spec was published, more implementations, generators, and documentation-tool integrations have appeared, and adoption is climbing fast. The debate isn't about whether the standard exists — it's about how much it's actually being used.
There's no evidence it affects Google. Google hasn't confirmed that llms.txt influences rankings or AI Overviews. John Mueller has emphasized that Google's systems can analyze page structure on their own and don't need an extra file to understand what a page contains. There's currently no basis for treating llms.txt as a ranking factor.
What Should Be Considered a Myth?
"llms.txt works just like robots.txt."
This is one of the most repeated oversimplifications. Yes, both files live in a domain's root directory — but they serve completely different functions. robots.txt talks to crawlers and sets rules for what they can access. llms.txt doesn't block anything and has no effect on indexing; its only role is to flag the materials a site owner considers most important. The robots.txt comparison works fine as a loose metaphor, but technically the two solve entirely different problems.
"Implementing llms.txt improves SEO."
There's currently no data supporting this. No available study has shown an increase in organic visibility after implementing the standard, and Google hasn't indicated it treats llms.txt as a ranking signal. That could change down the road, but as of today, linking llms.txt to classic SEO outcomes goes beyond what the evidence actually shows.
"Without llms.txt, AI won't use your site's content."
This isn't supported either. Language models have been analyzing websites for years without llms.txt, relying on crawlers, their own indexes, and search mechanisms. When an AI system does draw on a site's content, it's almost always thanks to existing indexing infrastructure — not because it happened to find an llms.txt file. Not having llms.txt doesn't make a site invisible to AI systems.
What Should Publishers Take Away From This?
If you run a website, you can implement llms.txt — but right now there's no reason to expect it to move the needle. Putting the file together isn't hard and usually doesn't cost much. The issue is that there's currently no evidence tying its implementation to better Google visibility, more frequent AI citations, or more traffic.
That doesn't make llms.txt worthless. If you run an extensive documentation site, a knowledge base, or a resource-heavy expert content site, a curated guide like this could turn out to be useful down the road, especially as AI agents become more capable. For now, it's best treated as a low-cost experiment whose benefits haven't been proven yet.
For most publishers, the bigger wins will keep coming from tactics with a proven track record: producing genuinely valuable content, improving information architecture, using structured data well, and keeping your site fast. That's exactly where CMS 4Media earns its keep — with built-in SEO automation, structured data handled for you, and an AI assistant built into the editorial workflow, it takes care of the fundamentals that actually move the needle, so your team can spend less time on technical busywork and more time on the content itself.
