The Ultimate Guide to Managing GPTBot, PerplexityBot, and the New /llms.txt
Why Blocking All AI Bots in 2026 is Career Suicide
Between 2023 and 2024, the SEO industry panicked. Fearing content theft, millions of publishers added blanket Disallow: / rules for every AI crawler they could find. GPTBot, ClaudeBot, Google-Extended, PerplexityBot—all blocked in one sweeping move.
By 2026, in the full Answer Engine Optimization (AEO) era, that decision has proven to be digital suicide.
AI platforms now drive billions of referrals. Studies aggregate that citations from engines like Perplexity or ChatGPT result in conversion rates 5–11× higher than traditional organic search. If you block the wrong bot today, you become invisible in ChatGPT answers, Perplexity footnotes, and Gemini summaries.
To understand the broader shift from traditional search to AI answers, read our Complete 2026 AEO Guide. In this technical guide, we focus purely on server-side crawler management.
One policy does not fit all bots
OpenAI, Anthropic, and Google often use different user-agents for different jobs: one may crawl to surface your URL in answers; another may crawl only to expand training corpora. Your robots.txt should reflect that split—not a single blunt rule for “AI.”
The 2026 AI Bot Matrix: Traffic Heroes vs. Content Leeches
The secret to modern technical SEO is realizing that major AI vendors separate retrieval / citation crawlers from training crawlers. One bot can help you earn traffic; another can ingest your pages for model weights with no citation upside.
Here is a practical 2026 classification based on official documentation and server-log patterns—always verify user-agent strings and IP ranges against each vendor’s current docs:
| Bot name | Owner | Purpose | 2026 recommendation | Reason |
|---|---|---|---|---|
| PerplexityBot | Perplexity AI | Search indexing & citations | Allow | Highest-conversion citations; direct traffic. Pair with our Perplexity SEO Guide. |
| OAI-SearchBot | OpenAI | ChatGPT Search results | Allow | Required for visibility in ChatGPT answer surfaces that link out to sources. |
| Claude-SearchBot | Anthropic | Claude search & user actions | Allow | Growing citation volume for B2B/SaaS and research-style queries. |
| GPTBot | OpenAI | Foundation model training | Block | Pure training crawl—no substitute for OAI-SearchBot for in-product citations. See also Does ChatGPT use backlinks? |
| Google-Extended | Gemini-family model training | Block (if policy) | Training token control. Standard Googlebot still handles traditional Search and AI Overviews—do not confuse the two. | |
| CCBot | Common Crawl | Massive open datasets | Block (typical) | Often burns crawl budget with zero direct citation ROI for most publishers. |
Note: For a deep dive into how OpenAI-related discovery intersects with classic authority signals, read our guide: Does ChatGPT use backlinks?
Your Perfect 2026 robots.txt (Copy-Paste Ready)
Copy and paste this template into your root robots.txt file. It is designed to welcome citation-oriented crawlers while opting out of common training-only agents—adjust paths (/wp-admin/, /private/, /api/) and the sitemap URL to match your stack.
# =============================================
# 2026 AI-OPTIMIZED robots.txt
# Allow citation engines → Block pure trainers
# =============================================
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /private/
Disallow: /api/
# === TRAFFIC & CITATION HEROES (ALLOW) ===
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
# === TRAINING LEECHES (BLOCK) ===
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: CCBot
Disallow: /
# === OPTIONAL: AGGRESSIVE SCRAPERS ===
User-agent: Bytespider
Disallow: /
Sitemap: https://yourwebsite.com/sitemap_index.xmlReplace https://yourwebsite.com/sitemap_index.xml with your real sitemap index. If you use a CDN or staging domain, keep robots.txt consistent with the canonical host you want cited.
Advanced Optimizations Beyond robots.txt
Updating robots.txt is only the first layer. To compete in AEO, align edge security, machine-readable policy files, and measurement.
1. The new standard: /llms.txt
In 2026, progressive sites adopt /llms.txt—a root-level markdown file that functions like a machine-oriented site brief: who you are, which sections matter, and how you prefer attribution. It does not replace robots.txt; it complements crawl rules with explicit guidance for LLM-powered tools that honor it.
# LLM Instructions
Business: [Your Brand Name]
Primary Audience: Technical SEOs and Marketers
Key Pages: /blog/, /services/, /about/
Preferred Citation Style: Always include the author's name and exact URL.2. Cloudflare & WAF configuration
Many publishers accidentally kill AI referral traffic because the CDN ships a “block AI scrapers” preset. Those rules often lump citation bots together with aggressive scrapers.
Prefer explicit allowlists: permit requests where User-Agent matches documented citation bots and source IPs fall in vendor-published ranges—tighten over time as documentation updates.
3. Monitor your logs
Configure analytics segments for referrers such as perplexity.ai and UTM patterns like utm_source=chatgpt.com. Periodically grep raw access logs (or use a log-based crawler tool) to confirm citation bots return 200 responses—not chained 403 or challenge pages from the WAF.
Is Your Technical Foundation AI-Ready?
If your robots.txt is outdated, you may be bleeding qualified AI referrals every day. When technical SEO, CDN rules, and schema are aligned with 2026 answer engines, citation surfaces and referral quality tend to improve measurably—often within weeks after fixes ship, depending on crawl frequency and authority.
We offer a technical AEO audit: server log sampling, WAF/CDN checks, robots.txt and sitemap review, and structured data validation against your priority templates.
Frequently asked questions
- Should I block GPTBot in robots.txt?
-
Yes, you should block GPTBot if you want to prevent OpenAI from using your content for foundation model training. However, you MUST allow OAI-SearchBot to ensure your site appears in real-time ChatGPT Search citations.
- What is PerplexityBot?
-
PerplexityBot is the web crawler for Perplexity AI. It fetches real-time web pages to generate footnoted, citation-heavy answers. You must Allow this bot to get cited in Perplexity.
- What does Google-Extended do?
-
Google-Extended is a user-agent token that allows publishers to manage whether their site content helps train standalone Google generative AI models like Gemini. Standard Googlebot still crawls for traditional Search and AI Overviews.
- What is an llms.txt file?
-
An llms.txt file is a new standard for 2026. It is a markdown file placed in a website’s root directory that provides explicit instructions to Large Language Models on how to read, interpret, and cite the website’s content.
Request your free technical AEO audit
Server logs, WAF posture, robots/sitemaps, and schema—reviewed for 2026 answer engines and citation bots.
Request your free technical AEO auditAllow the bots that cite you; block the ones that only train on you.
Comments
No comments yet. Be the first to reply.