AI Bot robots.txt Guide: OAI-SearchBot, GPTBot & More

What robots.txt can—and cannot—control

Google’s robots.txt documentation makes the key distinction: robots.txt manages crawler access, not guaranteed removal from search. If sensitive content must stay private, require authentication or remove public access.

AI crawler matrix based on current official documentation

Check the live source of truth before copying a token: OpenAI crawlers, Perplexity crawlers, Anthropic crawlers and Google-Extended. User-agent strings, IP lists and product behaviour can change.

Do you need to list every AI bot explicitly?

Usually not. An empty robots.txt file, or a wildcard group that does not disallow your public pages, already permits compliant crawlers. An explicit Allow: / can make policy easier for humans to audit, but it does not create a ranking boost.

There is also a syntax trap: a crawler that matches a more specific user-agent group may use that group instead of inheriting your wildcard protections. If you create a dedicated search-bot group, repeat any paths that must remain disallowed. Otherwise a copied “allow AI” block can unintentionally expose crawl access to admin, search-result or internal utility paths.

Copy-paste WordPress policy: allow AI search crawlers

This baseline keeps public content crawlable, repeats protected WordPress paths in the specific AI search group, and leaves training crawlers unchanged. Replace the sitemap and remove any path that does not exist on your site.

User-agent: *
Allow: /wp-content/uploads/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /?s=
Disallow: /*?s=
Disallow: /search/

User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /?s=
Disallow: /*?s=
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

If the wildcard group already allows the same public paths, the dedicated group is optional. Its value is policy clarity, not extra authority.

Optional policy: allow search, block training

Add only the training controls that match your publishing policy. These rules are independent from the search-crawler group:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Blocking GPTBot does not, by itself, block OAI-SearchBot. Blocking Google-Extended does not block Googlebot and, according to Google, does not affect Google Search inclusion or ranking. These are publisher-policy decisions, not universal SEO recommendations.

How WordPress virtual robots.txt works

WordPress can serve /robots.txt dynamically even when no physical file exists in the web root. SEO plugins can filter that response. Before uploading a new file:

1. Open the public https://example.com/robots.txt URL and save the exact response.
2. Identify the owner: physical file, WordPress core, SEO plugin, host or CDN.
3. Change the existing owner rather than creating two competing configurations.
4. Verify that the served response remains plain text, returns HTTP 200 and contains the intended sitemap.
5. Clear page, object, server and CDN caches before testing again.

On maksut.net, MiniSEO currently generates the virtual robots.txt response. Its wildcard rules allow public content and list both the main MiniSEO sitemap and the temporary Turkish services sitemap. No physical robots.txt file is required. This guide therefore does not silently change the site’s training policy.

Test robots.txt, WAF access and crawler identity

1. Check the public file: confirm one authoritative response on the canonical HTTPS host.
2. Parse the exact bot token: test the target URL against the wildcard group and any more-specific group.
3. Check the application response: a user-agent test can reveal obvious 403s or redirects.
4. Verify legitimate traffic: compare the request IP with the vendor’s current published ranges where available; a user-agent string alone can be spoofed.
5. Inspect logs: check response code, requested URL, crawl date and whether CDN challenges intercepted the request.
6. Recheck after propagation: vendors may take time to process a robots.txt change. OpenAI and Perplexity currently mention roughly a 24-hour adjustment window.

curl -sS https://example.com/robots.txt

curl -sS -D - -o /dev/null -A "OAI-SearchBot" https://example.com/important-page/

The second command tests how the stack reacts to the label; it does not prove that the request came from OpenAI. For a real allowlist, combine the documented user agent with the vendor’s current IP data and monitor changes rather than copying a stale IP list into an article.

robots.txt vs llms.txt

llms.txt is a community proposal for a concise Markdown map of useful resources. The llms.txt proposal describes it as a complement to existing standards. It does not grant access, block training, replace a sitemap or guarantee an AI citation. Keep crawler permission in robots.txt and treat llms.txt as an optional discovery aid—especially useful for documentation-heavy sites.

To build and validate the separate file without duplicating this crawler-policy page, use the Maksut llms.txt generator.

Country and language considerations

robots.txt applies to a host, not to a marketing language. A directory-based English and Turkish site normally shares the root file; separate subdomains must each publish their own. Do not translate user-agent tokens. Localise the pages bots can reach—the offers, currency, spelling, claims and evidence—rather than creating different bot names for the US, UK or Turkey.

The current page has early US and Netherlands impressions, including the Dutch query robots.txt optimaliseren. That is evidence of international technical intent, not a reason to place machine-translated Dutch copy on the English URL.

Common AI crawler mistakes

• Blocking every AI-named bot: training, search and user-triggered retrieval are different use cases.
• Assuming explicit allow rules are required: a permissive wildcard group already allows public paths.
• Forgetting specific-group precedence: repeat protected paths when a dedicated bot group could supersede wildcard rules.
• Blocking Googlebot instead of Google-Extended: this can damage Google Search crawling.
• Using robots.txt for privacy or deindexing: use authentication, removal or supported noindex controls.
• Ignoring the CDN: robots.txt may allow a bot while Cloudflare, Akamai or another WAF returns 403.
• Treating llms.txt as an access standard: it is a proposed discovery format, not a permission or ranking file.
• Reporting crawler hits as AI traffic: logs show requests; analytics and CRM data show human visits and customers.

Turn crawler policy into measurable AI search access

A safe implementation needs more than a copied code block: a documented publishing policy, one robots.txt owner, verified crawler identity, WAF alignment, sitemap continuity and conversion measurement. Maksut.net’s AI SEO and search visibility service audits those layers and connects access decisions to the pages and markets that can generate qualified demand.