We Maintain a Static llms.txt. It Goes Stale Every Week.
Pixelmojo publishes 2 to 4 blog posts per week. Every time we ship a new article, our llms.txt file at public/llms.txt is immediately out of date. The file lists our blog URLs, product descriptions, and company context for any AI system that reads it. But since it is a static text file committed to git, someone has to remember to update it manually.
Nobody remembers.
This is the dirty secret behind the 844,000+ websites that have adopted llms.txt since Jeremy Howard proposed the standard in September 2024. Most of them deployed a static file once, and it has been decaying ever since. The content drifts. New pages are missing. Old pages that have been removed still appear. The file becomes a snapshot of a moment in time, not a representation of what the site actually contains.
We are going to fix this. But first, we need to talk about what llms.txt actually does, because the SEO industry has gotten this completely wrong. (If you are new to the AI search landscape, start with our primer on why Google traffic dropped 33% and what replaced it.)
Step 3 of the GEO Playbook covers the technical infrastructure for getting cited by ChatGPT, Perplexity, Claude, and Gemini. This post is the deep dive on the static-vs-dynamic decision.
What llms.txt Actually Is (and Is Not)
Jeremy Howard, the co-founder of Answer.AI and former president of fast.ai, proposed the /llms.txt standard in September 2024. The idea is simple: place a markdown file at your site root that gives LLMs a structured overview of your content. The official specification lives at llmstxt.org.
The format looks like this:
# Company Name
> Brief description
## Section
- [Page Title](https://example.com/page): Description
- [Another Page](https://example.com/other): Description
There is also llms-full.txt, which compiles every page on your site into a single markdown document. Think of llms.txt as the table of contents and llms-full.txt as the complete book.
The standard is intentionally minimal. No complex schemas. No XML namespaces. Just markdown that both humans and machines can read.
What the SEO industry thinks it does
The GEO (Generative Engine Optimization) community immediately latched onto llms.txt as a potential ranking signal. The reasoning: if AI search engines can read your llms.txt, they will cite your content more often. Blog posts and conference talks started framing it alongside robots.txt and schema markup as a "must-have" for AI visibility.
What the data actually shows
Three independent studies have tested this assumption. All three reached the same conclusion.
| Study | Sample Size | Duration | Finding |
|---|---|---|---|
| SE Ranking | 300,000 domains | Cross-sectional | No correlation between llms.txt and AI citations |
| ALLMO.ai | 94,000+ URLs | Cross-sectional | No statistically significant difference |
| Search Engine Land | 10 sites | 180 days (90 pre/post) | 8 of 10 sites saw no change; 2 grew from unrelated efforts |
| OtterlyAI | 1 site | 90 days | Only 0.1% of AI crawler requests touched llms.txt |
Sources: SE Ranking study, ALLMO.ai report, Search Engine Land 180-day study, OtterlyAI GEO experiment.
The Search Engine Land study is particularly telling. Of the two sites that saw AI traffic growth after adding llms.txt, one had launched 27 downloadable AI templates simultaneously, and the other had concurrent PR coverage from major publications, restructured product pages, and published 12 new FAQ pages. The llms.txt file had nothing to do with their results.
OtterlyAI's experiment adds the technical evidence: their llms.txt file received only 84 AI bot visits in 90 days, while the average page on the same site received 265 AI bot visits. The AI crawlers are not even looking at the file.
“None of the AI services have said they are using llms.txt, and you can tell when you look at your server logs that they do not even check for it.”
John Mueller, Google Search Advocate, June 2025
The Google Contradiction
Here is where the story gets interesting.
In June 2025, John Mueller was unequivocal: llms.txt is useless. No AI system uses it. Server logs prove it.
Six months later, in December 2025, SEO professional Lidia Infante spotted an llms.txt file on Google's own Search Central developer documentation portal. She posted her discovery on Bluesky with a pointed question: "Is this an endorsement of LLMs.txt or are you trolling us, John?"
Mueller's response was a cryptic "hmmn :-/".
The file disappeared within hours of being discovered.
The most likely explanation is that different teams within Google operate independently. A documentation platform engineer probably deployed llms.txt as part of a broader tooling update, completely unaware of Mueller's public stance. This is how large organizations work: the left hand rarely knows what the right hand is doing.
But the incident matters because it reveals something important. Even inside Google, individual teams see value in making their documentation LLM-friendly, even if the search team does not use it as a ranking signal.
Where llms.txt Actually Works: Developer Tooling
If llms.txt does not help with search rankings, why have Anthropic, Cloudflare, Stripe, Vercel, and 844,000 other sites implemented it?
Because the real consumer of llms.txt is not an AI search engine. It is a developer sitting in their IDE.
When you use Claude Code, Cursor, or Windsurf to write code, these tools need context about the libraries and APIs you are working with. They fetch documentation to understand function signatures, configuration options, and best practices. And the fastest way to load that context is to grab a single markdown file that summarizes everything. (We wrote about how context engineering shapes AI coding agent performance.)
This is why Mintlify's November 2024 rollout was so significant. When Mintlify auto-generated llms.txt across every documentation site it hosts, thousands of developer tools instantly gained LLM-friendly documentation. Anthropic's Claude docs. Cursor's docs. Stripe's API reference. All of them became loadable as a single context window for a coding assistant.
| Use Case | How llms.txt is Used | Who Benefits |
|---|---|---|
| Developer tooling | Coding assistants fetch it to load library/API context | Developers writing code |
| Documentation platforms | Auto-generated summary of all docs pages | Anyone using the docs with AI |
| Internal knowledge bases | LLM-friendly index of internal wikis and runbooks | Teams using AI for internal search |
| AI search citations | Not meaningfully used by any AI search engine | Nobody (yet) |
Yoast SEO recognized this shift when they added llms.txt auto-generation in version 25.3 (June 2025), making it the first major WordPress SEO plugin to support the standard. Their implementation automatically selects cornerstone content and updates weekly.
The Context7 pattern
There is another emerging pattern worth noting. Tools like Context7 (which we use in our own development) resolve library names to documentation and serve them directly to coding agents via MCP (Model Context Protocol). This is essentially a dynamic, API-driven version of what llms.txt does as a static file.
The difference: Context7 queries documentation on demand and returns only the relevant sections. llms.txt dumps everything into one file and hopes the model figures out what it needs.
Both approaches solve the same problem. But one scales and one does not.
The Real Problem: Static Files Decay
Here is what our static llms.txt looks like in practice:
- Monday: We publish a new blog post about Claude Code hooks.
- Tuesday: Someone asks Perplexity about Claude Code hooks. Perplexity does not check our llms.txt (no AI search engine does). But a developer using Claude Code to implement hooks might benefit from our documentation context, except the new post is not in llms.txt yet.
- Wednesday: We publish another post. llms.txt is now two posts behind.
- Thursday: We update our pricing page. llms.txt still references old pricing.
- Friday: Someone remembers to update llms.txt manually. They add the new posts but miss the pricing change.
This pattern repeats every week. The file is perpetually out of sync with reality.
For documentation sites like Mintlify or Vercel's Next.js docs, this problem is solved at the platform level. The documentation framework auto-generates the file. But for custom-built sites running Next.js with a CMS or markdown-based content system, you need to build this yourself.
Building a Dynamic llms.txt in Next.js
The solution is straightforward: instead of a static file in public/, create a Next.js route handler that generates llms.txt from your actual content at request time.
Option 1: Contentlayer-powered (what we use)
If your blog runs on Contentlayer (like ours does), you can pull all published posts and generate the file dynamically:
// app/llms.txt/route.ts
import { allPosts } from 'contentlayer/generated'
export async function GET() {
const posts = allPosts
.filter(post => post.date)
.sort((a, b) => new Date(b.date).getTime() - new Date(a.date).getTime())
const lines = [
'# Pixelmojo',
'',
'> AI product studio. Vector qualifies leads.',
'> Hive orchestrates agents. You own the code.',
'',
'## Products',
'',
'- [Vector](/vector): 12-dimension lead qualification',
'- [Hive](/hive): Multi-agent AI co-workers',
'',
'## Blog Posts',
'',
...posts.map(
post => `- [${post.title}](/blogs/${post.slug}): ${post.description}`
),
'',
'## Contact',
'',
'- Email: founders@pixelmojo.io',
'- Location: Makati, Philippines',
]
return new Response(lines.join('\n'), {
headers: {
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': 'public, max-age=3600, s-maxage=86400',
},
})
}
This route regenerates the file from Contentlayer's data every time it is requested, with a 1-hour client cache and 24-hour CDN cache. New blog posts appear automatically after the next Contentlayer build.
Option 2: CMS-powered (Sanity, Prisma, Supabase)
If your content lives in a headless CMS or database, query it directly:
// app/llms.txt/route.ts
import { client } from '@/lib/sanity'
export async function GET() {
const posts = await client.fetch(
`*[_type == "post" && !(_id in path("drafts.**"))]
| order(publishedAt desc)
{ title, slug, description }`
)
const lines = [
'# Your Company',
'',
'> Your description',
'',
'## Articles',
'',
...posts.map(
(p: { title: string; slug: { current: string }; description: string }) =>
`- [${p.title}](/blog/${p.slug.current}): ${p.description}`
),
]
return new Response(lines.join('\n'), {
headers: { 'Content-Type': 'text/plain; charset=utf-8' },
})
}
export const revalidate = 3600
Option 3: ISR with on-demand revalidation
For sites that need instant updates when content changes, combine the route handler with Next.js ISR and a webhook from your CMS:
// app/llms.txt/route.ts
export const revalidate = false // Only revalidate on demand
// app/api/revalidate/route.ts
import { revalidatePath } from 'next/cache'
export async function POST(request: Request) {
const secret = request.headers.get('x-revalidate-secret')
if (secret !== process.env.REVALIDATE_SECRET) {
return new Response('Unauthorized', { status: 401 })
}
revalidatePath('/llms.txt')
return Response.json({ revalidated: true })
}
Point your CMS webhook at /api/revalidate and your llms.txt updates within seconds of publishing new content.
Can AI Bots Find Your Content?
Check access policies for 17 crawlers, including OpenAI and Claude search, browsing, and training agents. Checks robots.txt, structured data, llms.txt, and content accessibility.
Try the AI Crawl CheckerNot ready to build it by hand? The free llms.txt Generator assembles a starter from your existing page signals (title, meta, schema, navigation) in seconds. Either way, once you have a file, validate it with the llms.txt Validator. For the full testing workflow (bot access, citations, Reddit monitoring, and llms.txt validation), see our free AI visibility tools guide.
The Ecosystem: Who Is Building What
The llms.txt ecosystem is splitting into two camps: platforms that generate it automatically and sites that maintain it manually. The automatic camp is winning.
| Platform | Approach | When Added | Coverage |
|---|---|---|---|
| Mintlify | Auto-generated for all hosted docs | November 2024 | Thousands of sites (Anthropic, Cursor, etc.) |
| Yoast SEO | Auto-generated, weekly updates | June 2025 (v25.3) | Any WordPress site with Yoast |
| Next.js (nextjs.org) | Custom implementation | 2025 | Official Next.js docs |
| Vercel | Platform-level support | 2025 | Vercel documentation |
| Cloudflare | Custom implementation | 2025 | Cloudflare developer docs |
| Stripe | Custom implementation | 2025 | Stripe API reference |
BuiltWith now tracks llms.txt adoption across the web, reporting over 844,000 implementations as of late 2025. But a significant percentage of these are static files that were deployed once and never updated.
Related standards
llms.txt is not the only file competing for this space:
- robots.txt handles crawler permissions (allow/block). AI-specific directives like
User-agent: GPTBotandUser-agent: ClaudeBotcontrol training data access. We covered AI crawler configuration in our GEO Playbook. - ai.txt (proposed by Spawning AI in 2023) addresses training consent and data permissions, complementing robots.txt rather than replacing it.
- sitemap.xml provides URL discovery and change frequency hints.
- Schema.org / JSON-LD adds structured semantic data that AI systems already consume.
llms.txt fills a specific gap: giving LLMs a human-readable, markdown-formatted summary of what your site actually does. None of the other standards do this well.
What You Should Actually Do
Based on the evidence, here is a practical framework for deciding how to handle llms.txt:
If you are a documentation site or developer tool
Implement llms.txt dynamically. This is where the standard delivers real value. Coding assistants will use it, and your developers (and your users' developers) benefit directly. Mintlify handles this automatically. If you are self-hosting docs, build a route handler.
If you are a B2B SaaS or content-heavy site
Build a dynamic llms.txt from your CMS as future infrastructure. It costs almost nothing to implement (one route handler), and it ensures your file stays current if and when AI systems start consuming it. Do not expect it to improve your AI search citations today.
If you are a small business or personal site
Skip it. Your time is better spent on clean HTML, accurate structured data and content quality, which help every reader, human or machine. No single factor is a guaranteed citation lever. We covered the full tactical playbook in our GEO guide.
What matters more than llms.txt
The irony of the llms.txt conversation is that it distracts from the things that actually work for AI visibility:
- Answer-first content structure. Start sections with direct answers, then expand with evidence. Readers, and answer engines quoting passages, find the answer faster.
- Structured data (JSON-LD). Google uses it to understand pages and for rich results, though it says no special schema is needed for AI Overviews.
- Clean, crawlable HTML. No JavaScript-rendered content that crawlers cannot read. No content hidden behind authentication.
- Topical authority. Publish deep, interconnected content clusters. Our AI Search Playbook series is an example of this approach.
- External citations. When other authoritative sites reference your content, AI systems pick up on those signals.
We covered all five tactics in detail in our GEO implementation case study.
Our Approach: We Shipped It
We replaced our static public/llms.txt with a dynamic route handler at app/llms.txt/route.ts that generates from Contentlayer data. The static file is gone. The route handler pulls every published blog post automatically, includes all company and product context, and caches at the CDN layer for 24 hours.
You can see the result at pixelmojo.io/llms.txt. The blog section at the bottom updates itself every time we publish a new article. No manual edits. No stale data. No forgetting to update a file at 11pm on a Friday.
The implementation took about 30 minutes. The maintenance burden dropped to zero.
That is the real value proposition of llms.txt done right: not as a search ranking hack, but as infrastructure that represents your site accurately to any AI system that asks. Whether that system is a coding assistant loading documentation context or something that does not exist yet.
Build it dynamically. Let it run. Focus your energy on the content itself.
llms.txt Implementation: Questions Teams Ask
Common questions about this topic, answered.
What is llms.txt and who proposed it?
llms.txt is a markdown file placed at the root of a website that is meant to help large language models understand your site structure and content. Data scientist Jeremy Howard proposed it in September 2024 through Answer.AI. Unlike robots.txt (which controls crawl permissions) or sitemap.xml (which lists URLs), llms.txt provides a human-readable, LLM-optimized summary of what your site offers, why it matters, and how its content is organized.
Does llms.txt improve AI search rankings or citations?
Current evidence says no. SE Ranking analyzed 300,000 domains and found no correlation between llms.txt presence and AI citations. ALLMO.ai studied 94,000+ URLs and reached the same conclusion. OtterlyAI monitored AI crawler behavior for 90 days and found that only 0.1% of AI bot requests touched the llms.txt file, while average pages received far more attention. The file does not currently function as a ranking signal for any major AI search engine.
Why did Google add llms.txt after publicly dismissing it?
In June 2025, Google Search Advocate John Mueller stated that no AI system uses llms.txt. In December 2025, an llms.txt file was spotted on Google Search Central documentation by SEO professional Lidia Infante. Mueller responded with a cryptic "hmmn :-/" before the file was quietly removed. The most likely explanation is that a Google documentation platform engineer deployed it independently of the search team. Large organizations often have teams that operate without awareness of each other's public positions.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a concise summary file that lists your site sections, key pages, and brief descriptions with links. llms-full.txt compiles the complete text content of every page into a single markdown document. Think of llms.txt as the table of contents and llms-full.txt as the entire book. Mintlify generates both automatically for documentation sites. For most implementations, llms.txt alone is sufficient.
Should I use a static or dynamic llms.txt?
If your site publishes new content regularly (blogs, documentation updates, product changes), a dynamic llms.txt generated from your CMS or data layer stays accurate automatically. Static files require manual updates every time content changes and inevitably go stale. For sites that rarely change (landing pages, simple portfolios), a static file in your public directory works fine. The implementation cost of a dynamic route handler in Next.js is minimal: one file, roughly 30 lines of code.
How do I build a dynamic llms.txt in Next.js?
Create an app/llms.txt/route.ts file that exports a GET function. Inside it, query your data layer (Contentlayer for markdown blogs, Sanity or Prisma for CMS content, or Supabase for database-driven sites). Format the results as markdown lines with your site description, sections, and page listings. Return the content as a text/plain Response with appropriate Cache-Control headers. For instant updates, combine with ISR on-demand revalidation triggered by a CMS webhook.
What is the real use case for llms.txt if not search rankings?
The primary value is developer tool context. Coding assistants such as Claude Code, Cursor and Windsurf can load an llms.txt file when a developer points them at it, which makes it useful documentation context. Check your own server logs for coding-agent user-agents to see whether yours is fetched; we have not seen a published measure of how often this happens.
How many websites have adopted llms.txt?
BuiltWith tracks over 844,000 websites with llms.txt implementations as of late 2025. Major adopters include Anthropic (Claude docs), Cloudflare, Stripe, Vercel, Next.js, Zapier, and Supabase. Mintlify's November 2024 rollout was the largest single driver: auto-generating llms.txt across thousands of developer documentation sites overnight. Yoast SEO added support in June 2025, extending reach to WordPress sites.
How does llms.txt relate to robots.txt, ai.txt, and sitemap.xml?
Each file serves a different purpose. robots.txt controls which crawlers can access which pages. ai.txt (proposed by Spawning AI in 2023) declares training consent permissions for AI model training. sitemap.xml provides URL discovery with change frequency hints. llms.txt fills a unique gap: providing a human-readable, markdown-formatted summary of your site for LLM consumption. They are complementary, not competing, standards.
What should I focus on instead of llms.txt for AI search visibility?
Based on verified data: answer-first content structure (direct answers at the top of sections), structured data in JSON-LD (used by Google to understand pages), clean crawlable HTML (no JS-rendered content behind authentication), topical authority through deep content clusters, and external citations from authoritative sources. They are sound practice, though Google says its AI features need no special formatting or schema and none of them is a guaranteed citation lever. llms.txt is low-cost infrastructure worth implementing, but should not be your primary optimization focus.
The Bottom Line
llms.txt is not what the SEO industry wanted it to be. It is not a ranking signal. It is not a citation driver. It is not the next robots.txt.
It is developer infrastructure. And like all infrastructure, it works best when it stays current automatically.
If you maintain a static llms.txt, it is already out of date. Build a dynamic version that pulls from your actual content, or accept that the file will always be a stale approximation of your site.
We retired the static file and serve llms.txt from a dynamic route handler that regenerates from Contentlayer. Total implementation time: 30 minutes. Ongoing maintenance: zero.
That is the kind of investment worth making.
Related reading from our AI Search Playbook series:
- Your Google Traffic Dropped 33%. Here's Where It Went (Part 1)
- SEO vs AEO vs GEO: From Ranking in Search to Becoming the Recommended Brand (Part 2)
- The GEO Playbook (Part 3)
- What We Changed for AI Search by February 2026 (Dated Note) (Part 4)
Building AI systems that need accurate, real-time context?
- Vector: Lead qualification that reads 12 dimensions of intent
- Hive: Multi-agent AI co-workers that share intelligence
- Contact Us: Talk to us about your AI infrastructure
