Can AI Tools Read Your Website? A 20-Site Test With Claude and Gemini

How the test worked

On 30 October 2026 we ran the same check on each homepage with the Direct AI Access test in AI Perspective. Three fetchers requested the same URL within roughly a minute of each other:

• Plain crawler: an ordinary HTTP request from our server, which sits on a hosting datacenter IP in Europe. This is what most “can bots reach my site?” checkers do.
• Claude: Anthropic’s web fetch tool, called through the Claude API.
• Gemini: Google’s URL context tool, called through the Gemini API.

OpenAI’s API has no equivalent URL-fetch tool that we could call directly, so ChatGPT is not part of this test. Each site was tested once, on its homepage only.

Results for all 20 sites

“Read” for Claude means the tool returned the page text. For Gemini it means the tool reported a successful retrieval. “Challenge” means the response carried bot-challenge signals such as a challenge header, a challenge page or an interstitial title.

What the numbers show

1. 14 of 20 sites treated every fetcher the same. Twelve served all three, and two (one behind DataDome, one behind a Cloudflare challenge) blocked all three.
2. 4 sites blocked the plain crawler but served both AI tools. A generic bot-access check run from a datacenter would report a problem that Claude and Gemini never hit.
3. 2 sites served the plain crawler but refused AI tools. One refused both, one refused Gemini only. A normal crawl test would have said everything was fine.
4. Cloudflare alone predicted nothing. Ten of the 20 sites were behind Cloudflare: six served everyone, three challenged only the datacenter crawler, and one challenged everyone. The outcome depends on each site’s own bot and WAF settings, not on the CDN.

What this means for your site

• Test with the fetcher you care about. A request from your laptop, a monitoring service or a datacenter is a different client with a different reputation. It can be blocked when AI tools are not, and the reverse.
• Look beyond robots.txt. In the four “crawler blocked, AI allowed” cases the decision came from the WAF or bot management layer: challenge headers, challenge pages, 403 and 429 responses. For the two “AI refused” cases we cannot tell from outside whether a robots.txt rule for AI agents or a WAF rule made the call. On your own site you can: check your robots.txt groups for AI agents and your CDN’s security event log for the same minute.
• Check after every security change. Bot-fighting toggles, new WAF rules and rate limits change who gets through. Re-run the test whenever you change them.
• Fetchable is not cited. This test only answers whether a tool can open the page right now. Whether a model understands the page, and whether AI search mentions you, are separate questions.

Limits of this test

One homepage per site, one run, at one moment. Deeper pages, other regions and other times can behave differently, and sites change their bot rules often. Our plain crawler runs from a datacenter IP, which many security systems treat as suspicious by default. Treat the table as a snapshot of 30 October 2026, not as a permanent label for any site.