Developer Implementation Guide
How to Configure
How to Configure /llms.txt & AI Crawler Rules in 2026
Learn how to give AI answer engines clean Markdown context via /llms.txt while separating live search citation bots (OAI-SearchBot, PerplexityBot) from bulk training scrapers.
1. The Official /llms.txt File Structure
An /llms.txt file lives at the root of your domain (https://yourdomain.com/llms.txt) and follows a strict, token-efficient Markdown format:
# Acme Cloud Platform > Acme Cloud is an enterprise serverless Postgres and vector database platform with sub-10ms global read replicas. ## Core Documentation - [Quickstart Guide](https://acme.com/docs/quickstart): Deploy your first cluster in 60 seconds - [Pricing & Compute Tiers](https://acme.com/pricing): Pay-as-you-go storage and compute pricing - [REST & GraphQL API Reference](https://acme.com/docs/api): OpenAPI 3.1 endpoints and authentication ## Key Product Facts - Supports pgvector HNSW indexing up to 4,096 dimensions - SOC2 Type II, HIPAA, and GDPR compliant
2. Recommended 2026 robots.txt Template for Maximum AI Citations
Use this configuration to allow real-time AI Search engines that link back to your site while blocking unattributed dataset scrapers:
# Allow OpenAI ChatGPT Live Search Citations User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Allow Perplexity Real-Time Answer Engine User-agent: PerplexityBot Allow: / # Allow Anthropic Claude Web Grounding User-agent: ClaudeBot Allow: / # Block Unattributed Bulk Scrapers User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: /
Generate Your Custom llms.txt Automatically
Enter your URL in GeoRank Pro to scan your live site and download a tailored llms.txt file in one click.