Dynamic llms.txt: Automating Markdown Generation with Python and Node.js

1. Why Generate llms.txt Dynamically?

A hand-crafted llms.txt file works perfectly for a ten-page brochure site. It becomes a maintenance headache the moment your site publishes content at any real pace. If you ship a new article every day, or your CMS holds thousands of records, manually updating a Markdown file is not a sustainable process — entries go stale, new pages get omitted, and the whole point of giving AI crawlers a curated index collapses.

The better approach is to treat llms.txt the same way you treat a sitemap: as a machine-generated artifact that reflects your database state at any given moment. Your script queries the database or the CMS API, converts stored HTML content into clean Markdown descriptions, assembles the correct llms.txt structure, and writes the file to disk. A cron job or a publish webhook triggers the script automatically so the file is always current.

If you are new to the standard itself, read the Beginner's Guide to llms.txt first. For the exact syntax rules the output must follow, see How to Create and Format an llms.txt File. This guide assumes you already understand the format and focuses entirely on production code.

2. The File Structure You Are Building

Before writing a single line of code, it helps to be clear on the output target. A valid llms.txt file always opens with a level-one Markdown heading containing the site name, followed by a blockquote description, then one or more level-two sections grouping related links. Each link follows the format - [Title](URL): short description. The file is plain UTF-8 text, no HTML, no frontmatter.

A minimal but complete example looks like this:

llms.txt output target

# ToolFast

> A collection of free developer and SEO tools for modern web workflows.

## Articles

- [Beginner's Guide to llms.txt](https://www.toolfast.net/2026/07/beginners-guide-to-llmstxt-what-it-is.html): Introduction to the llms.txt standard and why AI crawlers use it.
- [How to Create and Format an llms.txt File](https://www.toolfast.net/2026/07/how-to-create-and-format-llmstxt-file.html): Step-by-step syntax guide with examples.

## Tools

- [llms.txt Generator](https://www.toolfast.net/2026/07/llmstxt-generator.html): Free online tool to build a compliant llms.txt file without coding.
- [HTML Formatter](https://www.toolfast.net/2026/03/html-formatter-beautify-and-format.html): Beautify and indent HTML in the browser.

Your script's job is to reproduce this structure by querying real data. The sections, the link titles, the descriptions — all of it comes from the database. Nothing is hardcoded except the site name and the opening blockquote, which you pass in as environment variables or a config file.

3. Python Script: Generate llms.txt from a Database

Dependencies

The script uses three libraries. psycopg2 connects to PostgreSQL (swap for pymysql or sqlite3 if your stack differs). html2text converts any stored HTML excerpts into clean Markdown description text. markdownify is available as a heavier alternative when you need to preserve heading and list structure inside descriptions, but for single-sentence excerpts html2text is faster and requires no configuration.

bash – install dependencies

pip install psycopg2-binary html2text markdownify

Full Python Script

Python – generate_llms.py

"""
generate_llms.py
Queries a PostgreSQL database and writes a valid llms.txt file
to the web root. Designed to run as a cron job or be called
from a deploy hook.
"""

import os
import html2text
import psycopg2
from datetime import datetime, timezone

# ── Configuration ────────────────────────────────────────────
SITE_NAME    = os.environ.get("SITE_NAME", "My Site")
SITE_DESC    = os.environ.get("SITE_DESC", "A brief description of what this site covers.")
OUTPUT_PATH  = os.environ.get("LLMS_OUTPUT", "/var/www/html/llms.txt")
DB_DSN       = os.environ.get("DATABASE_URL", "postgresql://user:pass@localhost/mydb")

# Configure html2text for short, clean excerpt conversion
_h = html2text.HTML2Text()
_h.ignore_links  = True   # keep text only; URLs already go in the link field
_h.ignore_images = True
_h.body_width    = 0      # no line wrapping

def html_to_plain(html_str: str) -> str:
    """Convert an HTML excerpt to a single clean line of plain text."""
    if not html_str:
        return ""
    text = _h.handle(html_str).strip()
    # Collapse newlines so the description stays on one line
    return " ".join(text.split())


def fetch_sections(conn) -> dict:
    """
    Returns a dict of { section_label: [(title, url, excerpt), ...] }
    Adjust the SQL to match your actual schema.
    """
    sections = {}
    cur = conn.cursor()

    # ── Published blog posts ─────────────────────────────────
    cur.execute("""
        SELECT title, canonical_url, excerpt
        FROM   posts
        WHERE  status = 'published'
          AND  noindex IS FALSE
        ORDER  BY published_at DESC
        LIMIT  50
    """)
    sections["Articles"] = cur.fetchall()

    # ── Tool / product pages ─────────────────────────────────
    cur.execute("""
        SELECT title, canonical_url, excerpt
        FROM   pages
        WHERE  page_type = 'tool'
          AND  status    = 'active'
        ORDER  BY sort_order ASC
    """)
    sections["Tools"] = cur.fetchall()

    cur.close()
    return sections


def build_llms_txt(sections: dict) -> str:
    """Assemble the llms.txt Markdown string from section data."""
    lines = []

    # Header block
    lines.append(f"# {SITE_NAME}\n")
    lines.append(f"> {SITE_DESC}\n")

    for section_name, rows in sections.items():
        if not rows:
            continue
        lines.append(f"\n## {section_name}\n")
        for title, url, excerpt in rows:
            description = html_to_plain(excerpt)
            entry = f"- [{title}]({url})"
            if description:
                entry += f": {description}"
            lines.append(entry)

    # Optional: append a generation timestamp as a comment
    ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
    lines.append(f"\n")

    return "\n".join(lines)


def main():
    conn = psycopg2.connect(DB_DSN)
    try:
        sections = fetch_sections(conn)
        content  = build_llms_txt(sections)
        with open(OUTPUT_PATH, "w", encoding="utf-8") as f:
            f.write(content)
        print(f"[OK] llms.txt written to {OUTPUT_PATH} ({len(content)} bytes)")
    finally:
        conn.close()


if __name__ == "__main__":
    main()

The script separates concerns cleanly: fetch_sections() owns all database interaction, html_to_plain() handles the HTML-to-text conversion, and build_llms_txt() does nothing but string assembly. This makes unit testing each piece straightforward without mocking the file system or the database connection simultaneously.

Swapping html2text for markdownify

If your stored excerpts contain structural HTML — lists, headings, bold text — and you want that structure preserved in the description field, replace the conversion step with markdownify. The markdownify library does a full DOM-to-Markdown pass, which produces richer output at the cost of slightly more verbose descriptions. For most llms.txt use cases, the plain-text approach from html2text is sufficient because descriptions are meant to be short and scannable.

Python – markdownify alternative for richer excerpts

from markdownify import markdownify as md_convert

def html_to_markdown_excerpt(html_str: str) -> str:
    """Use markdownify when you need to preserve inline formatting."""
    if not html_str:
        return ""
    result = md_convert(
        html_str,
        heading_style="ATX",   # Use # headings
        strip=["img", "a"],    # Drop images and links from descriptions
        bullets="-"
    )
    # Flatten to a single paragraph for llms.txt compatibility
    return " ".join(result.split())

4. Node.js Script: Generate llms.txt via a CMS API

Dependencies

This version targets teams running a headless CMS — Contentful, Sanity, or a custom REST API — rather than a direct database connection. It uses the turndown package for HTML-to-Markdown conversion and the native fetch API available in Node.js 18+. If you are on an older Node version, swap fetch for node-fetch.

bash – install dependencies

npm install turndown
# Optional for Node < 18:
# npm install node-fetch

Full Node.js Script

Node.js – generateLlms.mjs

/**
 * generateLlms.mjs
 * Fetches published content from a headless CMS REST API,
 * converts HTML body excerpts to Markdown with Turndown,
 * and writes llms.txt to the project root.
 *
 * Run: node generateLlms.mjs
 * Or schedule with: crontab -e  →  0 * * * * /usr/bin/node /app/generateLlms.mjs
 */

import TurndownService from "turndown";
import fs              from "fs/promises";
import path            from "path";

// ── Configuration ────────────────────────────────────────────
const SITE_NAME   = process.env.SITE_NAME   ?? "My Site";
const SITE_DESC   = process.env.SITE_DESC   ?? "A short description of what this site covers.";
const API_BASE    = process.env.CMS_API_URL ?? "https://cms.example.com/api";
const API_KEY     = process.env.CMS_API_KEY ?? "";
const OUTPUT_FILE = process.env.LLMS_OUTPUT ?? path.resolve("./public/llms.txt");

// Turndown instance — strip links and images from short descriptions
const td = new TurndownService({ headingStyle: "atx", bulletListMarker: "-" });
td.remove(["script", "style", "nav", "footer", "figure", "img"]);

function htmlToPlain(html = "") {
  if (!html) return "";
  const md = td.turndown(html);
  // Collapse whitespace to keep descriptions on one line
  return md.replace(/\s+/g, " ").trim();
}

async function fetchSection(endpoint, params = {}) {
  const url = new URL(`${API_BASE}/${endpoint}`);
  Object.entries(params).forEach(([k, v]) => url.searchParams.set(k, v));
  const res = await fetch(url.toString(), {
    headers: {
      "Authorization": `Bearer ${API_KEY}`,
      "Content-Type":  "application/json"
    }
  });
  if (!res.ok) throw new Error(`CMS API error: ${res.status} on ${url}`);
  const json = await res.json();
  // Expect { data: [{ title, slug, excerpt_html }, ...] }
  return json.data ?? [];
}

function buildEntry(title, url, excerptHtml) {
  const desc = htmlToPlain(excerptHtml);
  return desc
    ? `- [${title}](${url}): ${desc}`
    : `- [${title}](${url})`;
}

async function buildLlmsTxt() {
  const [articles, tools] = await Promise.all([
    fetchSection("posts",  { status: "published", noindex: "false", limit: 50 }),
    fetchSection("pages",  { type:   "tool",      status:  "active"           })
  ]);

  const lines = [
    `# ${SITE_NAME}\n`,
    `> ${SITE_DESC}\n`,
    "\n## Articles\n",
    ...articles.map(({ title, canonical_url, excerpt_html }) =>
      buildEntry(title, canonical_url, excerpt_html)
    ),
    "\n## Tools\n",
    ...tools.map(({ title, canonical_url, excerpt_html }) =>
      buildEntry(title, canonical_url, excerpt_html)
    )
  ];

  return lines.join("\n");
}

async function main() {
  try {
    const content = await buildLlmsTxt();
    await fs.writeFile(OUTPUT_FILE, content, "utf8");
    console.log(`[OK] llms.txt written → ${OUTPUT_FILE} (${content.length} bytes)`);
  } catch (err) {
    console.error("[ERROR]", err.message);
    process.exit(1);
  }
}

main();

The Promise.all() call fetches both the articles section and the tools section concurrently rather than sequentially, which halves the network round-trip time. The td.remove() call strips elements that pollute short descriptions — navigation, scripts, images — before Turndown processes the HTML. This mirrors what you would do manually in BeautifulSoup on the Python side.

5. Automating with Cron Jobs

Both scripts are designed to be run non-interactively, which makes them ideal cron targets. For a site that publishes once or twice a day, running the generator every hour is more than enough. For a high-volume news site, you might want to run it every ten minutes. For most blogs and product sites, a daily run at a low-traffic time is the right call.

crontab – schedule examples

# Python – run every day at 02:30 UTC
30 2 * * * /usr/bin/python3 /app/scripts/generate_llms.py >> /var/log/llms_gen.log 2>&1

# Node.js – run every hour at :00
0 * * * * /usr/bin/node /app/generateLlms.mjs >> /var/log/llms_gen.log 2>&1

# Run after every deploy (add to your CI/CD pipeline's post-deploy step)
# Example for a shell deploy script:
# node /app/generateLlms.mjs && echo "llms.txt regenerated"

Log the output on every run. If the CMS API is temporarily unavailable, the script will exit with a non-zero code and the error lands in the log file rather than silently corrupting the existing llms.txt. The process.exit(1) in the Node script and Python's unhandled exception propagation both achieve this behaviour. A simple Slack or email alert on cron failure keeps you informed without any extra monitoring infrastructure.

6. Webhook-Driven Regeneration

Cron jobs regenerate the file on a fixed schedule regardless of whether anything changed. Webhooks are smarter: your CMS fires a POST request to your server the moment a post is published or updated, and the server regenerates llms.txt immediately. The result is a file that is always accurate within seconds of a content change, with zero polling overhead.

Express.js Webhook Endpoint

Node.js – webhook handler (Express)

import express        from "express";
import crypto         from "crypto";
import { execFile }   from "child_process";

const app    = express();
const PORT   = process.env.PORT           ?? 3001;
const SECRET = process.env.WEBHOOK_SECRET ?? "change-me";

app.use(express.json());

function verifySignature(req) {
  const sig  = req.headers["x-hub-signature-256"] ?? "";
  const hmac = crypto
    .createHmac("sha256", SECRET)
    .update(JSON.stringify(req.body))
    .digest("hex");
  return crypto.timingSafeEqual(
    Buffer.from(`sha256=${hmac}`),
    Buffer.from(sig)
  );
}

app.post("/webhooks/llms-regenerate", (req, res) => {
  if (!verifySignature(req)) {
    return res.status(401).json({ error: "Invalid signature" });
  }

  // Fire-and-forget – respond immediately, regenerate in background
  res.status(202).json({ message: "Regeneration queued" });

  execFile("node", ["/app/generateLlms.mjs"], (err, stdout, stderr) => {
    if (err) {
      console.error("[WEBHOOK] Regeneration failed:", stderr);
    } else {
      console.log("[WEBHOOK] llms.txt regenerated:", stdout.trim());
    }
  });
});

app.listen(PORT, () => console.log(`Webhook server on :${PORT}`));

The HMAC-SHA256 signature check is non-negotiable in production. Without it, anyone who discovers your webhook URL can trigger arbitrary script execution on your server. The crypto.timingSafeEqual call avoids timing-based attacks that a naive string comparison would be vulnerable to. Store the shared secret in an environment variable, never hardcode it.

Python Flask Equivalent

Python – webhook handler (Flask)

import os, hmac, hashlib, subprocess
from flask import Flask, request, jsonify, abort

app    = Flask(__name__)
SECRET = os.environ.get("WEBHOOK_SECRET", "change-me").encode()

def verify(req):
    sig      = req.headers.get("X-Hub-Signature-256", "")
    digest   = "sha256=" + hmac.new(SECRET, req.data, hashlib.sha256).hexdigest()
    return hmac.compare_digest(sig, digest)

@app.post("/webhooks/llms-regenerate")
def regenerate():
    if not verify(request):
        abort(401)
    subprocess.Popen(["python3", "/app/scripts/generate_llms.py"])
    return jsonify({"message": "Regeneration queued"}), 202

if __name__ == "__main__":
    app.run(port=3001)

Configure your CMS to POST to this endpoint on every publish event. Most headless CMS platforms — Contentful, Sanity, Strapi, Directus — include a webhook delivery section in their settings panel where you set the target URL and the shared secret. WordPress users can trigger the same pattern on the publish_post hook using a simple wp_remote_post() call in a custom plugin. See the How to Configure llms.txt on WordPress guide for the WordPress-specific implementation.

7. Library Reference and Comparison

The choice between html2text and markdownify in Python usually comes down to whether you need to subclass the converter for custom tag handling. For llms.txt generation, where descriptions are kept short and structural HTML in excerpts is rare, html2text is the leaner default. On the Node.js side, turndown is the dominant choice with over 2,500 dependent packages on npm and active maintenance through 2026.

8. No-Code Alternative

If scripting is not your focus or you need a quick proof of concept before investing in automation, the free llms.txt Generator produces a correctly formatted file in the browser in under a minute. You supply the site name, description, and a list of URLs with labels — the tool handles the Markdown structure and lets you download the finished file. It is the fastest path from zero to a live llms.txt and a practical starting point you can later replace with the automated scripts above.

For teams managing a WordPress site specifically, the plugin-based workflow covered in How to Configure llms.txt on WordPress handles regeneration on publish automatically without requiring any server-side scripting. The dynamic script approach in this guide is the right call when you are running a custom stack, a headless architecture, or need fine-grained control over which content appears in the file and in what order.