The Developer's Guide to llms-full.txt: Exposing Your Site's Core Data Safely

1. What llms-full.txt Actually Is

llms-full.txt compiles all of a site's text into a single Markdown file, making it easier to paste a single URL to load complete context into an AI tool. This file structure was developed by Mintlify in collaboration with Anthropic and was subsequently included as part of the official llms.txt proposal. That lineage matters: this is not an unofficial hack. It is a recognised companion to the llms.txt standard, serving a distinct mechanical purpose in how LLMs retrieve and process site content. You can review the full specification at llmstxt.org, the canonical home of the proposal.

Instead of linking to individual pages, llms-full.txt contains the full text of those pages in Markdown format, so LLMs can access all the content directly without extra steps. This collapses what could be dozens of HTTP fetches into a single, predictable request. For AI agents working inside strict latency budgets or for users who manually paste a URL into a chat interface to load context, a single consolidated file eliminates the round-trip penalty of following each link from the index file.

Not only are AI agents actively visiting sites' llms.txt files — they are actually visiting llms-full.txt over twice as much. That statistic, surfaced by Mintlify from real-world agent traffic, tells you where the practical gravity of the specification sits. When a model has to choose between a map and the territory itself, it goes for the territory.

2. llms.txt vs llms-full.txt: The Architectural Difference

The reason there are two files instead of one comes down to a trade-off every LLM has to make: context window space is expensive, but fetching pages is slow. Different situations call for different sides of that trade-off. Understanding which side applies to your use case is the first architectural decision you need to make before writing a single byte.

llms.txt is a short index that links to your key pages with a note on each. llms-full.txt is an optional companion that inlines the full content of those pages into one Markdown file, so a tool can load everything in a single request without following links. The index file stays tiny — typically under 10 KB — while the full file can grow into the megabyte range depending on how much prose, code, and structured data your site carries.

If you have already worked through the Beginner's Guide to llms.txt: What It Is and Why Your Site Needs It, you know that llms.txt operates like a curated reading list. llms-full.txt is the library itself, open-stacks and all. Both files live at your domain root, coexist without conflict, and complement each other across different AI interaction patterns.

3. File Format and Structure

llms-full.txt follows the same structural convention as llms.txt but inlines fuller descriptions and, for content sites, much more of the actual page text. Where llms.txt outputs short pointers, llms-full.txt includes fuller text for the same URLs. The format is always plain Markdown — no HTML, no JSON envelope, no binary encoding. The file must be served with a plain-text MIME type and return an HTTP 200 at yourdomain.com/llms-full.txt.

Structural Pattern

The internal layout of a well-formed llms-full.txt follows a consistent page-by-page concatenation pattern. Each page section uses a level-2 Markdown heading as a separator, followed by the full prose of that page stripped of navigation chrome, scripts, ads, and boilerplate footers. The skeleton looks like this:

# YourSiteName
> One-sentence pitch: what this site is and who it's for.

## Page Title: Getting Started
URL: https://yourdomain.com/getting-started/

Full prose of the Getting Started page here.
All paragraphs, headings, lists, and code blocks — exactly
as authored, converted to clean Markdown.

---

## Page Title: API Reference
URL: https://yourdomain.com/api-reference/

Full prose of the API Reference page here.
...

The horizontal rule (---) between sections is a human-readability convention, not a spec requirement, but it dramatically improves how models segment context. Include a URL: annotation at the top of each section so a model can cite sources accurately. Strip every <script>, <style>, navigation block, sidebar, and cookie banner before concatenation. If you have already read How to Create and Format an llms.txt File: The Complete Syntax and Markdown Guide, the heading conventions translate directly — the difference is that you are appending body content rather than link lists.

4. Size Limits and Context Windows

There is no hard byte cap enforced by the llms.txt specification itself, but the practical ceiling is set by model context windows. While a large llms-full.txt can reduce fetch overhead and improve retrieval quality, the file can be extremely large, which may cause problems for larger docs sites. A model with a 200K-token context window — such as Gemini 2.5 Pro — can ingest roughly 150,000 words of plain prose. That sounds vast, but a documentation site with 300 pages averaging 500 words per page is already at 150,000 words before you add code blocks, which tokenise at a higher density than natural language.

For enterprise sites with thousands of pages, llms-full.txt is not realistic. Prioritisation of the 30–100 most important pages is the standard. For documentation sites such as Stripe, Vercel, and Anthropic, the full version is the primary use case. Treat the model's usable context as the hard limit, not theoretical window size. Models degrade in recall accuracy when content is pushed to the outer edges of a very long context, so keeping your file well inside the window — around 50–80% of the model's advertised limit — is a practical target.

llms-full.txt can grow large because it inlines page text, so keep it focused on genuinely high-value pages rather than the whole site. That keeps it maintainable and respects the limited context a model will actually use. A 2 MB ceiling expressed in bytes is a reasonable default for most deployments targeting a broad range of models.

5. What Content to Include — and Exclude

Include

• Core documentation pages — installation guides, API references, quickstarts
• Product or service descriptions that answer what the site does and who it serves
• High-traffic blog posts or guides that carry substantial factual value
• FAQ content and structured how-to pages
• Changelog entries if your product is developer-facing

Exclude

• Personally identifiable information (PII) — user-generated content, order history, profile data
• Internal or admin-only content — anything behind authentication must stay behind authentication
• Pricing or contractual data that is subject to change and could mislead an AI's responses
• Noindexed or canonically excluded pages — if the page is excluded from search crawling intentionally, exclude it here too
• Navigation, headers, footers, cookie banners — structural chrome burns tokens without adding signal
• Duplicate or thin content — tag archive pages, pagination shells, and near-identical product variants

6. Generating llms-full.txt Safely

The safest generation strategy is a static build script that runs at deploy time — never at request time. Runtime generation on every HTTP request for a file that could exceed 1 MB is a denial-of-service vector against your own infrastructure. Even with heavy caching, a cache miss under load will trigger the full build: scraping every page, parsing HTML, concatenating output, and writing a multi-megabyte string to disk. The fix is to move this entirely into your CI/CD pipeline.

Node.js Build Script (Static Site / Headless CMS)

// generate-llms-full.js  — run as a build step, not on request
const fs = require('fs');
const path = require('path');
const TurndownService = require('turndown');

const td = new TurndownService({ headingStyle: 'atx' });

// Provide your own page data source:
// JSON export from CMS, static HTML files, or a pre-built array.
const pages = require('./export/pages.json');

// Filter: only include pages explicitly flagged for AI exposure
const allowedPages = pages.filter(
  p => p.includeInLLMs === true && !p.isNoindex
);

let output = `# YourSiteName\n`;
output += `> Brief one-sentence description of your site.\n\n`;

for (const page of allowedPages) {
  const markdown = td.turndown(page.bodyHtml);
  output += `## ${page.title}\n`;
  output += `URL: ${page.canonicalUrl}\n\n`;
  output += markdown.trim();
  output += `\n\n---\n\n`;
}

// Hard size guard: warn if file exceeds 2 MB
const bytes = Buffer.byteLength(output, 'utf8');
if (bytes > 2 * 1024 * 1024) {
  console.warn(`[llms-full] WARNING: file is ${(bytes/1024/1024).toFixed(2)} MB`);
}

fs.writeFileSync(
  path.join(__dirname, 'public', 'llms-full.txt'),
  output,
  'utf8'
);
console.log(`[llms-full] Written: ${(bytes/1024).toFixed(1)} KB`);

Python Build Script (Django / Flask / Hugo)

# generate_llms_full.py — integrate into your build Makefile or CI step
import json, os, html2text

h2t = html2text.HTML2Text()
h2t.ignore_links = False
h2t.body_width = 0  # no line-wrapping

with open('export/pages.json', 'r', encoding='utf-8') as f:
    pages = json.load(f)

allowed = [p for p in pages if p.get('includeInLLMs') and not p.get('noindex')]

lines = ['# YourSiteName\n',
         '> Brief one-sentence description of your site.\n\n']

for page in allowed:
    md = h2t.handle(page['bodyHtml']).strip()
    lines.append(f"## {page['title']}\n")
    lines.append(f"URL: {page['canonicalUrl']}\n\n")
    lines.append(md + '\n\n---\n\n')

output = ''.join(lines)
byte_size = len(output.encode('utf-8'))

if byte_size > 2 * 1024 * 1024:
    print(f"[llms-full] WARNING: {byte_size / 1024 / 1024:.2f} MB — consider trimming.")

os.makedirs('public', exist_ok=True)
with open('public/llms-full.txt', 'w', encoding='utf-8') as f:
    f.write(output)

print(f"[llms-full] Written: {byte_size / 1024:.1f} KB")

WordPress / CMS Hook Approach

Yoast SEO and Rank Math both added native llms.txt generation in 2025. The setting appears in each plugin's dashboard, and once toggled on, the file is generated and updated automatically. For the full variant on WordPress, the cleanest approach is a save_post hook that appends or rewrites the relevant section of llms-full.txt in the server filesystem whenever a post is published. Do not regenerate the entire file on every save — maintain a per-post cache of Markdown output and only re-render the changed post's section, then stitch the final file during a scheduled WP-Cron job at off-peak hours.

For smaller sites — especially blogs, portfolios, or documentation sites under 50 pages — manual generation is the pragmatic route. The llms.txt Generator at ToolFast produces both the index file and a full-content variant from a single public URL, with semantic grouping and Markdown output ready to deploy. It analyses up to 20 pages, groups them semantically, and returns deployment instructions. Review and trim before publishing — generation is a strong first draft, not a final answer.

7. Serving It Efficiently

Because llms-full.txt is a static file, serving it efficiently is a solved problem — provided you treat it that way. The failure mode is allowing it to be dynamically assembled on each request. Once the file is written to disk at build time, configure your web server to serve it with aggressive caching headers and Gzip or Brotli compression. A 1.5 MB plain-text Markdown file compresses to roughly 200–350 KB with Brotli, which is an acceptable payload for the infrequent but high-value requests that AI agents make.

# nginx example — serve llms-full.txt as a static, compressed file
location = /llms-full.txt {
    root /var/www/public;
    default_type text/plain;
    charset utf-8;
    gzip_static on;
    brotli_static on;
    add_header Cache-Control "public, max-age=86400, stale-while-revalidate=3600";
    add_header X-Content-Type-Options "nosniff";
}

Set Cache-Control: max-age=86400 (24 hours) for a file that regenerates on each deployment. If your deploy frequency is higher — multiple times per day — reduce max-age accordingly and add a stale-while-revalidate directive to avoid latency spikes on cache busts. On CDN-backed infrastructure (Cloudflare, Fastly, CloudFront), purge the file path explicitly as part of your deploy pipeline so that AI agents never receive a stale version that references deleted or restructured pages.

Cross-reference your serving setup with the guidance in How AI Search Engines Crawl the Web: RAG, Vector Databases, and the Role of llms.txt to understand how retrieval-augmented pipelines interact with static content endpoints. The RAG layer at major AI search providers fetches content at query time, which means your file needs to be available at a stable, fast, always-200 URL.

8. Real-World Adoption Patterns

Major names including Anthropic, Cloudflare, and Hugging Face have implemented their own llms.txt and, in some cases, llms-full.txt. Anthropic's implementation is particularly instructive: their documentation site uses llms-full.txt as the primary ingestion point for Claude itself when users ask questions about the API. The file is generated from their documentation CMS at build time and CDN-cached globally.

Having both files works well when your documentation is extensive and wouldn't fit in a single context window, or when you serve different AI use cases. A code editor might use llms.txt for a structured overview with links to sections on installation, features, and API documentation, while its llms-full.txt contains complete documentation including all code examples and tutorials.

Publish llms.txt first on almost every site because it is the lightweight index defined at llmstxt.org. Add llms-full.txt only when your content is text-heavy enough to benefit from inlined page text — documentation sites, API references, and guide libraries are the strongest fit. Marketing sites with under 10 key pages rarely need the fuller edition.

One final note on expectations: as of early 2026, only about 2% of the most-cited domains in AI search have an llms.txt file. The specification is still a proposal, not a ratified standard, and no major AI company has formally committed to honouring it. Implement llms-full.txt as a low-risk, low-overhead signal layer — not as a replacement for clean crawlable HTML, structured data, or a well-maintained sitemap.xml. The infrastructure cost of a static file is near zero; the potential upside as AI agents mature is real.

FAQ