llmstxt

About llmstxt

llmstxt turns a site's existing sitemap into a maintained llms.txt file — the plain-text index that AI crawlers and assistants increasingly read in preference to guessing a site's structure from its HTML. The pipeline is deliberately narrow: verify that you actually own the domain by DNS TXT record or by file upload, let the service crawl the sitemap, and it emits a grouped llms.txt at a served URL that stays current without anyone editing it by hand. Ownership verification first is the part that separates this from a one-off generator, because an unverified generator is an abuse vector and the resulting file is nobody's canonical anything. Freshness is handled incrementally rather than by full re-crawl: the service reprocesses only new or changed URLs using the sitemap's lastmod hints, which keeps crawl cost proportional to actual change and avoids hammering a large site weekly for nothing. Delivery is engineered rather than assumed — ETag headers, CDN-ready caching and a last-known-good fallback so a failed regeneration serves the previous good file instead of a 404 or an empty document. The generation itself is rules-first with AI capped rather than AI-first, and the caps are published per tier as a Premium Top-N count, which is an honest way to expose where the model spend actually goes. Tiers scale on domains, URL count and update frequency.

Key Features

Tags