About llms.txt Generator

The llms.txt Generator API allows you to programmatically generate llms.txt files for any website. This service was inspired by Jeremy Howard's proposal for standardizing AI-friendly documentation.

Output

  • llms.txt - A concise navigation guide for AI systems following the llms.txt specification

API Quick Reference

Base URL

https://llmstxtgenerator.org/api/llmstxt

Endpoints

GET /api/llmstxt/{website}

Generate llms.txt for specified website

Parameters

  • website (required) - Domain to generate llms.txt for (e.g., example.com)

Response Codes

Status Description
200 OK Successfully generated llms.txt
400 Bad Request Invalid domain format
413 Payload Too Large Website exceeds URL limit (1000)
429 Too Many Requests Rate limit exceeded
500 Server Error Internal server error

Response Format

Success Response

# example.com

> A description of the website

## Main Section
- [Page Title](https://example.com/page): Description

Error Response

Error: [error message]

Limitations

  • 60 second timeout per request
  • Maximum 1000 URLs per website
  • 50 URLs processed per batch
  • Rate limit: 100 requests per hour per IP

Implementation Examples

cURL

curl https://llmstxtgenerator.org/api/llmstxt/example.com

Python

Using requests

import requests

response = requests.get('https://llmstxtgenerator.org/api/llmstxt/example.com')
print(response.text)

Using FastAPI

from fastapi import FastAPI, Response
import httpx

app = FastAPI()

@app.get("/generate/{domain}")
async def generate_llms(domain: str):
    async with httpx.AsyncClient() as client:
        response = await client.get(
            f'https://llmstxtgenerator.org/api/llmstxt/{domain}'
        )
        return Response(content=response.text, media_type="text/plain")

Using Flask

from flask import Flask, Response
import requests

app = Flask(__name__)

@app.route('/generate/')
def generate_llms(domain):
    response = requests.get(
        f'https://llmstxtgenerator.org/api/llmstxt/{domain}'
    )
    return Response(response.text, mimetype='text/plain')

JavaScript

Using Fetch API

fetch('https://llmstxtgenerator.org/api/llmstxt/example.com')
  .then(response => response.text())
  .then(data => console.log(data));

Using Node.js

const axios = require('axios');

async function getLLMS(domain) {
    const response = await axios.get(
        `https://llmstxtgenerator.org/api/llmstxt/${domain}`
    );
    return response.data;
}

Using Express.js

const express = require('express');
const axios = require('axios');
const app = express();

app.get('/generate/:domain', async (req, res) => {
    try {
        const response = await axios.get(
            `https://llmstxtgenerator.org/api/llmstxt/${req.params.domain}`
        );
        res.type('text').send(response.data);
    } catch (error) {
        res.status(500).send(error.message);
    }
});

PHP

Using cURL

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://llmstxtgenerator.org/api/llmstxt/example.com");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);

echo $response;

Using Laravel

use Illuminate\Support\Facades\Http;

Route::get('/generate/{domain}', function ($domain) {
    $response = Http::get("https://llmstxtgenerator.org/api/llmstxt/{$domain}");
    return response($response->body())
        ->header('Content-Type', 'text/plain');
});

Go

package main

import (
    "io"
    "net/http"
    "fmt"
)

func main() {
    resp, err := http.Get("https://llmstxtgenerator.org/api/llmstxt/example.com")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        panic(err)
    }
    
    fmt.Println(string(body))
}

Ruby

Using Net::HTTP

require 'net/http'
require 'uri'

uri = URI('https://llmstxtgenerator.org/api/llmstxt/example.com')
response = Net::HTTP.get(uri)
puts response

Using Ruby on Rails

require 'net/http'

class LlmsController < ApplicationController
  def generate
    uri = URI("https://llmstxtgenerator.org/api/llmstxt/#{params[:domain]}")
    response = Net::HTTP.get(uri)
    render plain: response
  end
end