Search and SEO

Writing•3 min read||

DocsLit generates search indexes, sitemaps, and AI-readable files automatically. You do not need to configure anything — these features work out of the box when you run docslit build.

Built-in search

Every static build generates a search-index.json file that powers the client-side search bar. The index includes page titles, descriptions, group names, and full body text.

Readers press / or click the search icon to open the search dialog. Results update as they type, with matches ranked by relevance.

Add a `description` field to your frontmatter. It improves search relevance and appears in the `llms.txt` output for AI agents.

Search engine optimization

HTML pages

Static builds generate individual HTML pages at docs/{slug}.html for every page in your sidebar. These pages contain the fully rendered content, making them crawlable by search engines.

Sitemap

When you set the url field in docslit.json, the build generates a sitemap.xml:

{
  "name": "My Docs",
  "url": "https://docs.example.com",
  "sidebar": [...]
}

Submit this sitemap to Google Search Console and other search engines to improve indexing.

Robots.txt

The build generates a robots.txt file that allows all major search engines and AI crawlers by default.

AI agent discovery

DocsLit generates several files to make your documentation accessible to AI agents:

A structured index of all your documentation pages with titles and descriptions. AI agents use this to understand your site structure and decide which pages to read. Follows the llmstxt.org specification.

The complete text content of every page concatenated into a single file. AI agents can consume your entire documentation in one request.

A machine-readable discovery file listing all available endpoints — `llms.txt`, search index, Markdown URLs, and content negotiation support. Agents can fetch this single file to learn how to interact with your docs.

A standalone MCP server that agents like Claude Desktop can connect to over stdio. Provides list_pages, get_page, and search_docs tools with no additional dependencies.

These files are generated automatically from your page content and frontmatter descriptions. Every page also includes an agent directive (in HTML and Markdown) pointing to your llms.txt index, following the Agent-Friendly Documentation Spec.

Versioned sites

For versioned sites (using the versions block in docslit.json), the build generates a root /llms.txt that acts as a version index:

# My Docs > Version-specific documentation indexes are listed below. ## Versions - [0.2.0 (default)](https://docs.example.com/0.2.0/llms.txt): Documentation for version 0.2.0 - [0.1.0](https://docs.example.com/0.1.0/llms.txt): Documentation for version 0.1.0

Each version's own llms.txt then lists the individual pages for that version. AI agents can start at the root index and navigate to the relevant version automatically.

Validate your deployed site with afdocs:

npx afdocs check https://docs.example.com

For versioned sites, pass the version-specific index if needed:

npx afdocs check https://docs.example.com --llms-txt-url https://docs.example.com/0.1/llms.txt --doc-version 0.1

Every page includes a Copy page split button in the page action bar. Readers can use it to:

Copy page — copy the page as Markdown, ready to paste into any LLM chat
Open in ChatGPT — opens ChatGPT pre-seeded with a question about the page URL
Open in Claude — opens Claude pre-seeded with the page URL
Copy MCP install command — copies the npx command to install your site's MCP server

No configuration is needed — the button appears automatically on every page.

MCP server

The generated mcp-server.js is a self-contained Node script that implements the Model Context Protocol over stdio. It requires no additional dependencies — just Node.js. It provides three tools:

Tool	Description
`list_pages`	List all pages with titles and descriptions
`get_page`	Get the full Markdown content of a page by slug
`search_docs`	Search documentation by keyword with excerpts

After building your site with docslit build, the MCP server is ready to use at dist/mcp-server.js.

Add it to your project's .claude/settings.json:

{
  "mcpServers": {
    "my-docs": {
      "command": "node",
      "args": ["dist/mcp-server.js"]
    }
  }
}

Then ask Claude Code to search or read your docs directly from the conversation.

Add it to your Claude Desktop configuration:

{
  "mcpServers": {
    "my-docs": {
      "command": "node",
      "args": ["/absolute/path/to/dist/mcp-server.js"]
    }
  }
}

Restart Claude Desktop to pick up the new server.

Add it to your project's .cursor/mcp.json:

{
  "mcpServers": {
    "my-docs": {
      "command": "node",
      "args": ["dist/mcp-server.js"]
    }
  }
}

Any MCP client that supports stdio transport can use the server. Point it at node dist/mcp-server.js as the command.

Commit `dist/mcp-server.js` to your repository so that contributors and AI tools can use it without needing to rebuild. The server reads from the sibling `.md` and `search-index.json` files in the same directory.

Content negotiation

AI agents can request any page URL with an Accept: text/markdown header to receive raw Markdown instead of HTML. This works automatically in the dev server and in static builds deployed to platforms that support edge middleware.

# Returns raw Markdown instead of the HTML page
curl -H "Accept: text/markdown" https://docs.example.com/quickstart

For static builds, DocsLit generates platform-specific files to handle this negotiation automatically.

DocsLit generates functions/_middleware.js at your project root on each build. Cloudflare Pages picks this up automatically when you deploy — no additional configuration is needed. The build also writes _headers into dist/ so .md files are served with Content-Type: text/markdown.

DocsLit generates a vercel.json with header-based rewrites. Copy it to your project root before deploying:

cp dist/vercel.json vercel.json

Or if you already have a vercel.json, merge the rewrites array into it.

Move _middleware.js into your netlify/edge-functions/ directory, or reference it in your netlify.toml:

[[edge_functions]]
  path = "/*"
  function = "markdown-negotiation"

On platforms without middleware support (GitHub Pages, S3), agents can request Markdown directly at /{slug}.md. The llms.txt file lists these URLs for discovery.

Offline search

The offline build mode (docslit build --offline) produces a search-index.js file that loads on demand when search is first used. Search works without a server — even when opening the file from your desktop via file://.

Best practices

Write a one-sentence description for every page. This improves search results, llms.txt quality, and helps readers decide if a page is relevant.

Add "url" to your docslit.json to enable sitemap.xml generation with absolute URLs.

Write headings that describe what the section covers. Headings are indexed for search and appear in the table of contents.

Next steps

Start adding rich content with callouts and alerts
Deploy your site with static hosting or DocsLit Cloud