# content
I am excited to announce the release of llm-docs-builder, a library that transforms Markdown documentation into an AI-optimized format for Large Language Models.

TL;DR: Open source tool that strips 85-95% of noise from documentation for AI systems. Transforms Markdown, generates llms.txt indexes, and serves optimized docs to AI crawlers automatically. Reduces RAG costs significantly.

View on GitHub

If you find it interesting or useful, don’t forget to star the repo – it helps others discover the tool!

Table of Contents

Toggle

The Problem

What llm-docs-builder Does

Before and After Example

Why This Matters

How It Works

1. Hierarchical Context Preservation

2. Semantic Noise Removal

3. Enhanced llms.txt Generation

Getting Started

Installation

Transform Your Documentation

Generate an llms.txt Index

Measure Your Savings

Configuration

Serving Optimized Docs to AI Crawlers

Why Markdown for RAG Systems

Section Title

When NOT to Use This

Next Steps

The Problem

If you have watched an AI assistant confidently hallucinate your library API – suggesting methods that do not exist or mixing up versions – you’ve experienced this documentation problem firsthand. When AI systems like Claude, ChatGPT, and GitHub Copilot try to understand your docs using RAG (Retrieval-Augmented Generation), they drown in noise.

Beautiful HTML documentation with navigation bars, CSS styling, and JavaScript widgets becomes a liability. The AI retrieves your “Getting Started” page, but 90% of what it processes is HTML boilerplate and formatting markup. The actual content? Buried in that mess.

Context windows are expensive and limited. Research shows that typical HTML documents waste up to 90% of tokens on pure noise: CSS styles, JavaScript code, HTML tag overhead, comments, and meaningless markup. This waste adds up fast across thousands of pages and millions of queries.

What llm-docs-builder Does

This tool transforms your markdown documentation to eliminate 85-95% of the noise compared to the HTML version, letting AI assistants focus on the actual content. I have extracted it from the Karafka framework’s documentation build system, where it has served thousands of developers in production for months.

See also  How to set up a firewall on Ubuntu 18.04

Real metrics from Karafka documentation:

Page | HTML | Markdown | Reduction
Getting Started | 82.0 KB | 4.1 KB | 95% (20x)
Monitoring | 156 KB | 6.2 KB | 96% (25x)
Configuration | 94.3 KB | 3.8 KB | 96% (25x)

Average: 93% fewer tokens, 20-36x smaller files

Before and After Example

Before transformation (98 tokens):

---title: Getting Starteddescription: Learn how to get startedtags: [tutorial, beginner]updated: 2024-01-15---[![Build](https://img.shields.io/badge/build-passing-green.svg)](https://ci.example.com)[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)# Getting Started> **Note**: Requires Ruby 3.0+Welcome to our framework! Let's get you up and running...

After transformation (18 tokens, 81% reduction):

# Getting StartedWelcome to our framework! Let's get you up and running.

Why This Matters

Cleaner documentation means AI assistants spend less time processing noise and more time understanding your actual content. This translates to lower costs per query, fewer hallucinations as shown in the HtmlRAG study, and much faster response times.

How It Works

llm-docs-builder applies several transformations to your markdown documentation to make it RAG-friendly, then generates an llms.txt index that helps AI agents discover and navigate your content efficiently. Below are examples of these transformations in action.

1. Hierarchical Context Preservation

When documents are chunked for RAG, context loss leads to hallucinations. Consider:

# Configuration## Consumer Settings### auto_offset_resetControls how consumers handle missing offsets...

Chunked independently, ### auto_offset_reset loses all parent context. llm-docs-builder preserves hierarchy:

### Configuration / Consumer Settings / auto_offset_resetControls how consumers handle missing offsets...

Now the chunk is self-contained even when retrieved in isolation.

2. Semantic Noise Removal

Strips YAML/TOML frontmatter.

Removes HTML comments and build badges.

Expands relative links to absolute URLs.

Normalizes whitespace while preserving code blocks.

Preserves code syntax highlighting markers.

See also  What to Expect at the Ubuntu Summit 2022

3. Enhanced llms.txt Generation

This feature creates llms.txt index files – the emerging standard for AI-discoverable documentation, adopted by Anthropic, Cursor, Pinecone, LangChain, and 200+ projects.

Generated llms.txt includes token counts and timestamps and provides AI-readable documentation for your:

# Llms.txt## Documentation- [Getting Started](https://myproject.io/docs/getting-started.md): 1,024 tokens, updated 2024-03-15- [API Reference](https://myproject.io/docs/api-reference.md): 5,420 tokens, updated 2024-03-18- [Configuration Guide](https://myproject.io/docs/configuration.md): 2,134 tokens, updated 2024-03-12Total documentation: 8,578 tokens across 3 core pages

AI agents can prioritize which documents to fetch based on token budgets, their needs and freshness.

Getting Started

Installation

docker pull mensfeld/llm-docs-builder:latestalias llm-docs-builder='docker run -v $(pwd):/workspace mensfeld/llm-docs-builder'

Transform Your Documentation

llm-docs-builder bulk-transform --docs ./docs --base-url https://myproject.io

This single command can reduce your RAG system’s token usage by 85-95%.

Generate an llms.txt Index

llm-docs-builder generate --docs ./docs

Measure Your Savings

llm-docs-builder compare --url https://karafka.io/docs/Getting-Started/

Example output:

============================================================Context Window Comparison============================================================Human version:  82.0 KBAI version:     4.1 KBReduction:      77.9 KB (95%)Factor:         20.1x smaller============================================================

Configuration

Create llm-docs-builder.yml:

docs: ./docsbase_url: https://myproject.io# Optimization optionsconvert_urls: trueremove_comments: trueremove_badges: trueremove_frontmatter: truenormalize_whitespace: true# RAG enhancementsnormalize_headings: trueinclude_metadata: trueinclude_tokens: trueexcludes:  - "**/internal/**"

Serving Optimized Docs to AI Crawlers

Configure your web server to automatically serve markdown to LLM crawlers while continuing to serve HTML to human visitors. Detect AI user agents (ChatGPT-User, GPTBot, anthropic-ai, claude-web, PerplexityBot, meta-externalagent) and serve .md files instead of .html.

Implement this feature to automatically detect AI agents and serve them raw markdown, as shown in the following example:

Apache (.htaccess):

SetEnvIf User-Agent "(?i)(openai|anthropic|claude|gpt|chatgpt|perplexity)" IS_LLM_BOTRewriteCond %{ENV:IS_LLM_BOT} !^$RewriteCond %{REQUEST_FILENAME}.md -fRewriteRule ^(.*)$ $1.md [L]

Nginx:

map $http_user_agent $is_llm_bot {    default 0;    "~*(?i)(openai|anthropic|claude|gpt|chatgpt|perplexity)" 1;}location ~ ^/docs/ {    if ($is_llm_bot) {        try_files $uri.md $uri $uri/ =404;    }}

Benefits:

Zero disruption to human users

Automatic cost savings on every AI query

No separate documentation sites needed

See also  How to check memory usage on Ubuntu 22.04

Why Markdown for RAG Systems

Tokenization efficiency matters for both cost and performance. The following table shows a simple heading comparison:

Format | Example | Token Count

HTML

Section Title

7-8 tokens

Markdown | ## Section Title | 3-4 tokens

HTML requires opening and closing tags for every element (2x overhead), special characters (, >,  ) consume multiple tokens each, and attributes add 2-3 tokens per occurrence. Markdown uses single characters for formatting (**, *, -, |) that often tokenize to single tokens, requires no closing tags, and maintains semantic structure without attribute bloat.

Format efficiency comparison:

Plain text: 96% reduction vs raw HTML

Cleaned HTML (CSS/JS removed): 94% reduction

Markdown: 90% reduction

While cleaned HTML can match Markdown's efficiency, the preprocessing required is complex and error-prone. Markdown provides the optimal balance: simple to generate, efficient to tokenize, and preserves semantic structure naturally. For RAG systems that chunk and retrieve documents independently, a clear hierarchy of Markdown ensures that each chunk remains interpretable without its surrounding context.

When NOT to Use This

My llm-docs-builder will not be of great use when:

Your docs rely heavily on visual diagrams that cannot be described in Markdown.

You are already serving pure Markdown without HTML noise.

Your documentation is primarily an API reference with minimal prose (consider OpenAPI/Swagger instead).

Next Steps

Your documentation is already being consumed by LLMs. The question is whether you're serving optimized content or forcing them to parse megabytes of HTML boilerplate.

Install llm-docs-builder via Docker.

Run compare on your existing docs to measure potential savings.

Configure llm-docs-builder.yml for your project.

Run bulk-transform to generate optimized versions.

Use server configuration to serve markdown to AI crawlers.

Every query you optimize saves money and improves the quality of AI-assisted development with your framework.

llm-docs-builder is open source under the MIT License. It is extracted from production code powering the Karafka framework documentation.

The post Announcing llm-docs-builder: An Open Source Tool for Making Documentation AI-Friendly appeared first on Closer to Code.

# contentMap.en
I am excited to announce the release of llm-docs-builder, a library that transforms Markdown documentation into an AI-optimized format for Large Language Models.

TL;DR: Open source tool that strips 85-95% of noise from documentation for AI systems. Transforms Markdown, generates llms.txt indexes, and serves optimized docs to AI crawlers automatically. Reduces RAG costs significantly.

View on GitHub

If you find it interesting or useful, don’t forget to star the repo – it helps others discover the tool!

Table of Contents

Toggle

The Problem

What llm-docs-builder Does

Before and After Example

Why This Matters

How It Works

1. Hierarchical Context Preservation

2. Semantic Noise Removal

3. Enhanced llms.txt Generation

Getting Started

Installation

Transform Your Documentation

Generate an llms.txt Index

Measure Your Savings

Configuration

Serving Optimized Docs to AI Crawlers

Why Markdown for RAG Systems

Section Title

When NOT to Use This

Next Steps

The Problem

If you have watched an AI assistant confidently hallucinate your library API – suggesting methods that do not exist or mixing up versions – you’ve experienced this documentation problem firsthand. When AI systems like Claude, ChatGPT, and GitHub Copilot try to understand your docs using RAG (Retrieval-Augmented Generation), they drown in noise.

Beautiful HTML documentation with navigation bars, CSS styling, and JavaScript widgets becomes a liability. The AI retrieves your “Getting Started” page, but 90% of what it processes is HTML boilerplate and formatting markup. The actual content? Buried in that mess.

Context windows are expensive and limited. Research shows that typical HTML documents waste up to 90% of tokens on pure noise: CSS styles, JavaScript code, HTML tag overhead, comments, and meaningless markup. This waste adds up fast across thousands of pages and millions of queries.

What llm-docs-builder Does

This tool transforms your markdown documentation to eliminate 85-95% of the noise compared to the HTML version, letting AI assistants focus on the actual content. I have extracted it from the Karafka framework’s documentation build system, where it has served thousands of developers in production for months.

See also  What changing vehicle ownership habits and mobility trends mean for the future of the automotive industry

Real metrics from Karafka documentation:

Page | HTML | Markdown | Reduction
Getting Started | 82.0 KB | 4.1 KB | 95% (20x)
Monitoring | 156 KB | 6.2 KB | 96% (25x)
Configuration | 94.3 KB | 3.8 KB | 96% (25x)

Average: 93% fewer tokens, 20-36x smaller files

Before and After Example

Before transformation (98 tokens):

---title: Getting Starteddescription: Learn how to get startedtags: [tutorial, beginner]updated: 2024-01-15---[![Build](https://img.shields.io/badge/build-passing-green.svg)](https://ci.example.com)[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)# Getting Started> **Note**: Requires Ruby 3.0+Welcome to our framework! Let's get you up and running...

After transformation (18 tokens, 81% reduction):

# Getting StartedWelcome to our framework! Let's get you up and running.

Why This Matters

Cleaner documentation means AI assistants spend less time processing noise and more time understanding your actual content. This translates to lower costs per query, fewer hallucinations as shown in the HtmlRAG study, and much faster response times.

How It Works

llm-docs-builder applies several transformations to your markdown documentation to make it RAG-friendly, then generates an llms.txt index that helps AI agents discover and navigate your content efficiently. Below are examples of these transformations in action.

1. Hierarchical Context Preservation

When documents are chunked for RAG, context loss leads to hallucinations. Consider:

# Configuration## Consumer Settings### auto_offset_resetControls how consumers handle missing offsets...

Chunked independently, ### auto_offset_reset loses all parent context. llm-docs-builder preserves hierarchy:

### Configuration / Consumer Settings / auto_offset_resetControls how consumers handle missing offsets...

Now the chunk is self-contained even when retrieved in isolation.

2. Semantic Noise Removal

Strips YAML/TOML frontmatter.

Removes HTML comments and build badges.

Expands relative links to absolute URLs.

Normalizes whitespace while preserving code blocks.

Preserves code syntax highlighting markers.

See also  Restart Nginx on Ubuntu 22.04

3. Enhanced llms.txt Generation

This feature creates llms.txt index files – the emerging standard for AI-discoverable documentation, adopted by Anthropic, Cursor, Pinecone, LangChain, and 200+ projects.

Generated llms.txt includes token counts and timestamps and provides AI-readable documentation for your:

# Llms.txt## Documentation- [Getting Started](https://myproject.io/docs/getting-started.md): 1,024 tokens, updated 2024-03-15- [API Reference](https://myproject.io/docs/api-reference.md): 5,420 tokens, updated 2024-03-18- [Configuration Guide](https://myproject.io/docs/configuration.md): 2,134 tokens, updated 2024-03-12Total documentation: 8,578 tokens across 3 core pages

AI agents can prioritize which documents to fetch based on token budgets, their needs and freshness.

Getting Started

Installation

docker pull mensfeld/llm-docs-builder:latestalias llm-docs-builder='docker run -v $(pwd):/workspace mensfeld/llm-docs-builder'

Transform Your Documentation

llm-docs-builder bulk-transform --docs ./docs --base-url https://myproject.io

This single command can reduce your RAG system’s token usage by 85-95%.

Generate an llms.txt Index

llm-docs-builder generate --docs ./docs

Measure Your Savings

llm-docs-builder compare --url https://karafka.io/docs/Getting-Started/

Example output:

============================================================Context Window Comparison============================================================Human version:  82.0 KBAI version:     4.1 KBReduction:      77.9 KB (95%)Factor:         20.1x smaller============================================================

Configuration

Create llm-docs-builder.yml:

docs: ./docsbase_url: https://myproject.io# Optimization optionsconvert_urls: trueremove_comments: trueremove_badges: trueremove_frontmatter: truenormalize_whitespace: true# RAG enhancementsnormalize_headings: trueinclude_metadata: trueinclude_tokens: trueexcludes:  - "**/internal/**"

Serving Optimized Docs to AI Crawlers

Configure your web server to automatically serve markdown to LLM crawlers while continuing to serve HTML to human visitors. Detect AI user agents (ChatGPT-User, GPTBot, anthropic-ai, claude-web, PerplexityBot, meta-externalagent) and serve .md files instead of .html.

Implement this feature to automatically detect AI agents and serve them raw markdown, as shown in the following example:

Apache (.htaccess):

SetEnvIf User-Agent "(?i)(openai|anthropic|claude|gpt|chatgpt|perplexity)" IS_LLM_BOTRewriteCond %{ENV:IS_LLM_BOT} !^$RewriteCond %{REQUEST_FILENAME}.md -fRewriteRule ^(.*)$ $1.md [L]

Nginx:

map $http_user_agent $is_llm_bot {    default 0;    "~*(?i)(openai|anthropic|claude|gpt|chatgpt|perplexity)" 1;}location ~ ^/docs/ {    if ($is_llm_bot) {        try_files $uri.md $uri $uri/ =404;    }}

Benefits:

Zero disruption to human users

Automatic cost savings on every AI query

No separate documentation sites needed

See also  What to Expect at the Ubuntu Summit 2022

Why Markdown for RAG Systems

Tokenization efficiency matters for both cost and performance. The following table shows a simple heading comparison:

Format | Example | Token Count

HTML

Section Title

7-8 tokens

Markdown | ## Section Title | 3-4 tokens

HTML requires opening and closing tags for every element (2x overhead), special characters (, >,  ) consume multiple tokens each, and attributes add 2-3 tokens per occurrence. Markdown uses single characters for formatting (**, *, -, |) that often tokenize to single tokens, requires no closing tags, and maintains semantic structure without attribute bloat.

Format efficiency comparison:

Plain text: 96% reduction vs raw HTML

Cleaned HTML (CSS/JS removed): 94% reduction

Markdown: 90% reduction

While cleaned HTML can match Markdown's efficiency, the preprocessing required is complex and error-prone. Markdown provides the optimal balance: simple to generate, efficient to tokenize, and preserves semantic structure naturally. For RAG systems that chunk and retrieve documents independently, a clear hierarchy of Markdown ensures that each chunk remains interpretable without its surrounding context.

When NOT to Use This

My llm-docs-builder will not be of great use when:

Your docs rely heavily on visual diagrams that cannot be described in Markdown.

You are already serving pure Markdown without HTML noise.

Your documentation is primarily an API reference with minimal prose (consider OpenAPI/Swagger instead).

Next Steps

Your documentation is already being consumed by LLMs. The question is whether you're serving optimized content or forcing them to parse megabytes of HTML boilerplate.

Install llm-docs-builder via Docker.

Run compare on your existing docs to measure potential savings.

Configure llm-docs-builder.yml for your project.

Run bulk-transform to generate optimized versions.

Use server configuration to serve markdown to AI crawlers.

Every query you optimize saves money and improves the quality of AI-assisted development with your framework.

llm-docs-builder is open source under the MIT License. It is extracted from production code powering the Karafka framework documentation.

The post Announcing llm-docs-builder: An Open Source Tool for Making Documentation AI-Friendly appeared first on Closer to Code.

# name
Announcing llm-docs-builder: An Open Source Tool for Making Documentation AI-Friendly

# nameMap.en
Announcing llm-docs-builder: An Open Source Tool for Making Documentation AI-Friendly

# preview.content
I am excited to announce the release of llm-docs-builder, a library that transforms Markdown documentation into an AI-optimized format for Large Language Models.

TL;DR: Open source tool that strips 85-95% of noise from documentation for AI systems. Transforms Markdown, generates llms.txt indexes, and serves optimized docs to AI crawlers automatically. Reduces RAG costs significantly.

 View on GitHub

If you find it interesting or useful, don't forget to star  the repo - it helps others […]

# summary
I am excited to announce the release of llm-docs-builder, a library that transforms Markdown documentation into an AI-optimized format for Large Language Models.

TL;DR: Open source tool that strips 85-95% of noise from documentation for AI systems. Transforms Markdown, generates llms.txt indexes, and serves optimized docs to AI crawlers automatically. Reduces RAG costs significantly.

 View on GitHub

If you find it interesting or useful, don't forget to star  the repo - it helps others […]

# summaryMap.en
I am excited to announce the release of llm-docs-builder, a library that transforms Markdown documentation into an AI-optimized format for Large Language Models.

TL;DR: Open source tool that strips 85-95% of noise from documentation for AI systems. Transforms Markdown, generates llms.txt indexes, and serves optimized docs to AI crawlers automatically. Reduces RAG costs significantly.

 View on GitHub

If you find it interesting or useful, don't forget to star  the repo - it helps others […]