🎉 New: Generate llms.txt programmatically using our free API!

View Docs →

llms.txt Generator

Create llms.txt and llms-full.txt files for any website in seconds

The Story Behind LLMs.txt

LLMs.txt was conceived by Jeremy Howard, co-founder of Answer.AI, to address a fundamental challenge in AI-human interaction.

Modern AI systems face a significant limitation: they can only process a finite amount of context at once. This makes it challenging for them to comprehend extensive documentation websites effectively. While traditional SEO practices work well for search engines, they aren't optimized for AI systems that need to reason about content.

When AI assistants attempt to process standard web pages, they struggle with non-essential elements like navigation menus, scripts, and styling. These elements consume valuable context space without contributing to the actual content understanding.

LLMs.txt provides a elegant solution: it delivers precisely curated information in a format that AI systems can efficiently process and understand.

And soon, leading AI companies began implementing LLMs.txt...

Understanding LLMs.txt

LLMs.txt is a standardized markdown file that helps AI systems better understand your website. The specification defines two key files:

  • /llms.txt: A concise navigation guide that helps AI systems quickly grasp your site's structure
  • /llms-full.txt: A comprehensive file containing all documentation in one accessible location

The /llms.txt file follows a specific structure:

# Your Project Name

> A concise summary of what your project does

Key information and context about the project

## Main Documentation
- [Getting Started](https://docs.example.com/start): Quick setup guide
- [Core Concepts](https://docs.example.com/concepts): Essential principles

## Optional Resources
- [Advanced Topics](https://docs.example.com/advanced): In-depth guides
- [Examples](https://docs.example.com/examples): Code samples

The format starts with an H1 heading for the project name, followed by a blockquote summary. Additional sections use H2 headers to organize documentation links, with an "Optional Resources" section specifically marking supplementary content.

Check out these real-world examples:

Understanding llms-full.txt

While /llms.txt provides navigation and structure, /llms-full.txt contains the complete documentation content in markdown format.

Here's an example from Cursor's /llms-full.txt:

# AI Review (Beta)

AI Review is a feature that allows you to review your recent changes in your codebase to catch any potential bugs.

<Frame>
  <img src="https://mintlify.s3-us-west-1.amazonaws.com/cursor/images/advanced/review.png" alt="AI Review" />
</Frame>

You can click into individual review items to see the full context in the editor, and chat with the AI to get more information.

### Custom Review Instructions

In order for AI Review to work in your favor, you can provide custom instructions for the AI to focus on. For example,
if you want the AI to focus on performance-related issues, you could put:

```
focus on the performance of my code
```

This way, AI Review will focus on the performance of your code when scanning through your changes.

### Review Options

Currently, you have a several options to choose from to review:

*   `Review Working State`
    *   This will review your uncommitted changes.
*   `Review Diff with Main Branch`
    *   This will review the diff between your current working state and the main branch.
*   `Review Last Commit`
    *   This will review the last commit you made.

See the full file on Cursor's docs →

LLMs.txt vs sitemap.xml vs robots.txt

While LLMs.txt might seem similar to existing web standards, it addresses a unique set of challenges specific to AI interaction.

sitemap.xml

Traditional sitemaps simply list all indexable pages on your site. However, they don't help AI systems process the actual content. Without LLMs.txt, AI assistants would need to parse complex HTML and handle unnecessary markup, wasting valuable context space.

robots.txt

While robots.txt guides search crawlers on what they can access, it doesn't help with content comprehension. It's focused on permissions rather than understanding.

llms.txt

LLMs.txt is specifically designed for AI interaction. It addresses context window limitations by providing clean, structured content without the clutter of HTML, JavaScript, or CSS. This makes it easier for AI systems to process and understand your site's content efficiently.

Using LLMs.txt with AI Systems

Unlike search engines that actively crawl the web, current AI systems don't automatically discover and process LLMs.txt files.

To use LLMs.txt with your AI assistant, you'll need to provide the content manually. Here are the common methods:

Share the Link

Provide the direct URL to your LLMs.txt file in your conversation with the AI.

Copy & Paste

Copy the file contents directly into your prompt when asking questions about the documentation.

File Upload

Use your AI tool's file upload feature if available to share the LLMs.txt content.

Using with Different AI Tools

ChatGPT

To use with ChatGPT, simply navigate to your documentation's /llms-full.txt file, copy either the URL or its contents, and paste it into your chat. Then you can ask specific questions about the documentation.

Example of using LLMs.txt with ChatGPT

Claude

Since Claude doesn't have direct web access, you'll need to provide the documentation manually. Simply copy the contents of the /llms-full.txt file and paste it into your conversation, or upload it as a text file. Once loaded, Claude will have the complete context to answer any questions about the documentation.

Example of using LLMs.txt with Claude

Cursor IDE

Cursor makes it easy to integrate documentation context into your development workflow. Simply type @Docs > Add new doc to open the documentation modal, then paste the URL of your /llms-full.txt file. Once indexed, you can reference this documentation naturally in your conversations with Cursor's AI.

Example of using LLMs.txt with Cursor IDE
View Generation History