LLMs.txt Validator Rules, Size Limits, and Audits
A technical validation guide explaining the structural thresholds, CommonMark blocks, and HTTP discovery headers required to pass an llms.txt validator.
Chrome Lighthouse audits sites for an llms.txt file as part of its agentic browsing checks, while online validators trigger size warnings whenever a file exceeds 500KB. Although only 10.13 percent of domains have implemented the file across an audit of over 500 sites, passing automated checks requires meeting specific CommonMark syntax rules and HTTP discovery standards.123
Specification Hierarchy: BOM, H1 Title, and Blockquote Rules
The llms.txt specification defines an ordered markdown hierarchy that parses top to bottom. The file starts with an optional byte-order mark (BOM), followed immediately by an H1 heading. This H1 title naming the site or project is the only required section in the standard. Public checkers flag validation failures if the file lacks an H1 header starting with '# Title'.124
Directly after the H1 title, the file expects a blockquote summary starting with the '>' syntax. Checkers validate that this quote block contains key project context. Following the blockquote, publishers can include zero or more non-heading markdown sections, such as paragraphs or lists, to guide models on interpreting files.145
H2 File Lists, Protocol URLs, and the Reserved Optional Section
Following non-heading markdown blocks, the document organizes resources into zero or more markdown sections delimited by H2 headings. Checkers verify these sections as file lists containing markdown hyperlinks formatted as name. Each link can optionally feature a colon and explanatory notes about the file.145
Validators flag broken, malformed, or duplicate URLs. Missing protocol prefixes will fail checks, as all links must be full URLs specifying http:// or https://. Furthermore, the 'Optional' H2 heading is reserved by convention for secondary links that an agent can skip when it requires a shorter context window.12
Validator Thresholds: 50KB Limits and llms-full.txt Rules
Automated validators enforce distinct structural boundaries between index discovery and full documentation payloads. For regular llms.txt index files, exceeding 500KB triggers large file size warnings because large files can breach AI context limits.2
| File Type | Target Size | Warning Threshold | Content Scope |
|---|---|---|---|
| llms.txt | Under 50KB | Exceeding 500KB | Structured index under 100 links |
| llms-full.txt | Can exceed 500KB | None | Complete documentation and reference guides |
Best practices dictate keeping the core llms.txt file under 50KB and capping link volume below 100 links. In contrast, the companion /llms-full.txt file embeds the complete markdown of every listed page per the specification and can exceed 500KB without issue.26
CommonMark Parsing and Checker Errors
Online checkers evaluate underlying CommonMark compliance. In CommonMark parsing, document structures separate into leaf blocks and container blocks. ATX headings and paragraphs function as leaf blocks, whereas block quotes and lists serve as container blocks.52
Validators reject files with missing required sections, incorrect Markdown formatting, absent H1 titles, missing '>' blockquote syntax, or missing H2 sections. Automated generators also filter out posts with fewer than 200 words, password-protected pages, private posts, and noindex content before building the index.26
Path Specificity, Clean Markdown, and HTTP Headers
An llms.txt file can reside at the site root or at any subpath, covering the pages beneath that location. Tools also generate discovery files at /.well-known/llms.txt. When multiple files apply across paths, client agents must use the most specific file.16
The specification recommends serving clean markdown directly rather than HTML pages wrapped in navigation and scripts. Pages providing clean markdown should use the same URL with .md appended, such as page.html.md, or with the extension replaced by .md, such as page.md. URLs without file names should append index.html.md or index.md.16
Servers can broadcast these relationships via an HTTP Link: response header using rel='alternate' type='text/markdown' for the markdown version and rel='describedby' pointing to the covering llms.txt file. Under standard HTTP header rules, if an Accept header is present and the server cannot provide an acceptable response, it should return a 406 Not Acceptable status code.17
Questions
What should be in an llms.txt file?
An llms.txt file must contain an H1 heading with the site or project name as its only required element. It typically includes an optional byte-order mark, a blockquote summary starting with '>', non-heading project notes, and H2 sections containing markdown links with full http:// or https:// URLs.
Is llms.txt mandatory?
The llms.txt file is not mandatory for regular web functionality, but Chrome Lighthouse audits sites for its presence as part of agentic browsing checks. Only 10.13 percent of audited domains currently serve the file.
How to generate LLMs full txt?
A companion /llms-full.txt file is generated by embedding the complete markdown documentation of every page listed in the core index. Unlike the primary llms.txt file, which should stay under 50KB, /llms-full.txt can exceed 500KB.