🎉 New: Generate llms.txt programmatically using our free API!
View Docs →Create llms.txt and llms-full.txt files for any website in seconds
âš
Get the complete content in clean markdown format - no HTML parsing needed
LLMs.txt was conceived by Jeremy Howard, co-founder of Answer.AI, to address a fundamental challenge in AI-human interaction.
Modern AI systems face a significant limitation: they can only process a finite amount of context at once. This makes it challenging for them to comprehend extensive documentation websites effectively. While traditional SEO practices work well for search engines, they aren't optimized for AI systems that need to reason about content.
When AI assistants attempt to process standard web pages, they struggle with non-essential elements like navigation menus, scripts, and styling. These elements consume valuable context space without contributing to the actual content understanding.
LLMs.txt provides a elegant solution: it delivers precisely curated information in a format that AI systems can efficiently process and understand.
And soon, leading AI companies began implementing LLMs.txt...
LLMs.txt is a standardized markdown file that helps AI systems better understand your website. The specification defines two key files:
/llms.txt
: A concise navigation guide that helps AI systems quickly grasp your site's structure/llms-full.txt
: A comprehensive file containing all documentation in one accessible locationThe /llms.txt
file follows a specific structure:
# Your Project Name > A concise summary of what your project does Key information and context about the project ## Main Documentation - [Getting Started](https://docs.example.com/start): Quick setup guide - [Core Concepts](https://docs.example.com/concepts): Essential principles ## Optional Resources - [Advanced Topics](https://docs.example.com/advanced): In-depth guides - [Examples](https://docs.example.com/examples): Code samples
The format starts with an H1 heading for the project name, followed by a blockquote summary. Additional sections use H2 headers to organize documentation links, with an "Optional Resources" section specifically marking supplementary content.
Check out these real-world examples:
While /llms.txt
provides navigation and structure,
/llms-full.txt
contains the complete documentation content in markdown format.
Here's an example from Cursor's /llms-full.txt
:
# AI Review (Beta) AI Review is a feature that allows you to review your recent changes in your codebase to catch any potential bugs. <Frame> <img src="https://mintlify.s3-us-west-1.amazonaws.com/cursor/images/advanced/review.png" alt="AI Review" /> </Frame> You can click into individual review items to see the full context in the editor, and chat with the AI to get more information. ### Custom Review Instructions In order for AI Review to work in your favor, you can provide custom instructions for the AI to focus on. For example, if you want the AI to focus on performance-related issues, you could put: ``` focus on the performance of my code ``` This way, AI Review will focus on the performance of your code when scanning through your changes. ### Review Options Currently, you have a several options to choose from to review: * `Review Working State` * This will review your uncommitted changes. * `Review Diff with Main Branch` * This will review the diff between your current working state and the main branch. * `Review Last Commit` * This will review the last commit you made.
While LLMs.txt might seem similar to existing web standards, it addresses a unique set of challenges specific to AI interaction.
Traditional sitemaps simply list all indexable pages on your site. However, they don't help AI systems process the actual content. Without LLMs.txt, AI assistants would need to parse complex HTML and handle unnecessary markup, wasting valuable context space.
While robots.txt guides search crawlers on what they can access, it doesn't help with content comprehension. It's focused on permissions rather than understanding.
LLMs.txt is specifically designed for AI interaction. It addresses context window limitations by providing clean, structured content without the clutter of HTML, JavaScript, or CSS. This makes it easier for AI systems to process and understand your site's content efficiently.
Unlike search engines that actively crawl the web, current AI systems don't automatically discover and process LLMs.txt files.
To use LLMs.txt with your AI assistant, you'll need to provide the content manually. Here are the common methods:
Provide the direct URL to your LLMs.txt file in your conversation with the AI.
Copy the file contents directly into your prompt when asking questions about the documentation.
Use your AI tool's file upload feature if available to share the LLMs.txt content.
To use with ChatGPT, simply navigate to your documentation's /llms-full.txt file, copy either the URL or its contents, and paste it into your chat. Then you can ask specific questions about the documentation.
Since Claude doesn't have direct web access, you'll need to provide the documentation manually. Simply copy the contents of the /llms-full.txt file and paste it into your conversation, or upload it as a text file. Once loaded, Claude will have the complete context to answer any questions about the documentation.
Cursor makes it easy to integrate documentation context into your development workflow.
Simply type @Docs > Add new doc
to open the documentation modal,
then paste the URL of your /llms-full.txt file. Once indexed, you can reference this documentation
naturally in your conversations with Cursor's AI.
Here's how it works: