llms.txt Explained: The New robots.txt for AI
What is llms.txt?
llms.txt is a plain-text file, written in Markdown, that lives at the root of your domain (yoursite.com/llms.txt) and gives AI models a clean, curated map of your site: what you do, your most important pages, your docs, your pricing. Think of it as a guided tour written specifically for machines, so they don't have to reconstruct your site from cluttered HTML, navigation menus, cookie banners, and marketing animation. The idea borrows from a familiar pattern. For decades, robots.txt has told crawlers where they may go. llms.txt extends the spirit of that convention to a new audience — large language models — but flips the purpose from restriction to explanation. Instead of "here's what you can't touch," it says "here's what matters and here's how to understand it."
Is llms.txt the same as robots.txt?
No, and confusing them is common. robots.txt is about permission: it tells crawlers which paths they're allowed to access. llms.txt is about comprehension: it tells models what your site is and which content is worth understanding. One is a gate; the other is a guide. You want both configured correctly, and they solve different failure modes. A misconfigured robots.txt can accidentally block AI crawlers entirely — a hard failure where the model never sees you. A missing llms.txt is a soft failure — the model can reach your site but has to guess at what's important, and may guess wrong.
What goes in a good llms.txt?
A strong llms.txt is short, honest, and structured. At minimum it includes: a one-line description of what your product is and who it's for, in plain language; a curated list of your most important pages, each with a short note explaining what it covers (product, pricing, documentation, key guides); pointers to documentation and API references if you have them; and nothing misleading, because models cross-check llms.txt against your actual pages. A minimal shape looks like a title, a one-line summary, a "Core pages" list with a note per link, and a short "About" paragraph.
Does llms.txt actually help, or is it hype?
Honesty matters here, because trust is the product. Adoption of llms.txt is still early, and no major model guarantees it reads or honors yours. If someone promises that one file will make ChatGPT recommend you, be skeptical. What's true is more modest and still worth doing: llms.txt costs nothing, removes ambiguity about your site, and is part of a coherent signal set. When it lines up with your structured data, your on-page positioning, and consistent descriptions elsewhere, it helps models form a stable, correct picture of you — and a correct picture is the entire game. It's a cheap, low-risk point on the board, not a silver bullet.
How do I create and add an llms.txt?
You can write one by hand in a few minutes using the structure above, or generate a first draft with a tool and then edit it for accuracy. Save it as llms.txt, upload it to your domain root so it resolves at yoursite.com/llms.txt, and verify it loads in a browser. Then confirm it matches reality — every page you point to should exist and say what you claim. Re-check it whenever you materially change your site.
Frequently Asked Questions
- Where do I put the llms.txt file?
- At the root of your domain, so it's reachable at yoursite.com/llms.txt — the same location convention as robots.txt. Some teams also publish an expanded llms-full.txt with more detail.
- Do ChatGPT and Claude actually read llms.txt?
- Support is still emerging and not guaranteed by any provider. The upside is that it's free and low-risk, and it strengthens the overall set of signals that help models understand you correctly.
- Is llms.txt a replacement for a sitemap?
- No. A sitemap helps crawlers discover every URL; llms.txt curates and explains the handful that matter most. They complement each other.
- Can a bad llms.txt hurt me?
- It can if it's misleading — claiming pages or capabilities that don't exist undermines trust when models cross-check. Keep it accurate and modest and it's purely upside.
- Do I need technical skills to make one?
- Not really. It's a plain Markdown file. The harder part is deciding what actually matters on your site, which is a positioning question more than a technical one.
Ready when you are
Generate your llms.txt free with our tool, then scan your site.
▶ Scan your site free