CRCrawlRanker
Back
Blog

How to check if AI crawlers can access your site

GPTBot, ClaudeBot, Google-Extended, PerplexityBot — three free ways to find out which AI crawlers can actually read your site, and what each answer means.

Published

Whether AI crawlers can reach your site now affects two separate things: whether your content shows up in AI answers with a citation, and whether it gets used as training data. Most sites have never checked which of those they are opted into — and the answer lives in one small file you can inspect in under a minute.

This guide shows three ways to check, from fastest to most thorough: read your robots.txt yourself, run it through a validator that knows the AI user agents, and confirm what your server actually returns to a bot. None of them needs an account or a crawl budget.

The AI crawlers that matter in 2026

Each company runs more than one bot, and they do different jobs. Blocking one does not block the others — this is the single most common misunderstanding we see in real robots.txt files.

  • GPTBot (OpenAI) — collects content for model training. Blocking it keeps your pages out of future training sets; it does not affect ChatGPT search answers.
  • OAI-SearchBot (OpenAI) — builds the index behind ChatGPT search. Block it and your site stops being findable — and citable — there.
  • ChatGPT-User (OpenAI) — fetches a page live when a user asks ChatGPT to read a specific URL. It acts on behalf of a person, not a scheduled crawl.
  • ClaudeBot (Anthropic) — training-data crawler, the equivalent of GPTBot. Anthropic also operates Claude-User and Claude-SearchBot for live fetches and search.
  • Google-Extended (Google) — not a separate crawler at all. It is a robots.txt control token that tells Google whether regular Googlebot crawls may be used for Gemini training. Blocking it does not remove you from Google Search.
  • PerplexityBot (Perplexity) and CCBot(Common Crawl) — Perplexity’s search index and the open crawl corpus many models train on.

Method 1: read your robots.txt yourself

Your robots.txt lives at one fixed address: yourdomain.com/robots.txt. Open it in a browser and look for the user agents above. Three rules govern what you see:

  • Bots follow their most specific group. If there is a User-agent: GPTBot group, GPTBot obeys that group and ignores User-agent: * entirely — even if the * group is stricter.
  • No group means the wildcard applies. If ClaudeBot is never named, it follows the * rules like any other crawler.
  • An empty Disallow: line allows everything. Disallow: with no path is full access; Disallow: / with a slash is a total block. One character changes the meaning completely.

The manual read gets slow the moment a file has several groups, wildcards, or conflicting Allow and Disallow lines — which is exactly when mistakes hide.

Method 2: paste your domain into a validator that knows AI bots

Our free robots.txt validator fetches your live file, checks the syntax, and — the part most testers skip — prints an AI crawler report: a per-bot verdict for GPTBot, ClaudeBot, Google-Extended, and the other agents above, resolved through the same group-matching rules the bots use. No signup, and it takes about ten seconds.

A validator matters here because AI-crawler rules are where hand-edited files go wrong most often: a group added for GPTBot that accidentally unblocks everything else it used to inherit from *, or a blanket block copied from a blog post that quietly removed the site from ChatGPT search citations.

Method 3: confirm what your server actually returns

robots.txt is a request, not a lock — compliant crawlers obey it, but your CDN or firewall may also be blocking bots at the network level, sometimes without anyone remembering it was configured. To see what a bot actually receives, request a page with its user agent from a terminal:

curl -A "GPTBot" -I https://yourdomain.com/

A 200 means the server serves that user agent; a 403 or an endless challenge page means something upstream (Cloudflare bot rules are the usual suspect) blocks it regardless of what robots.txt says. Two honest caveats:

  • This tests the user-agent string, not the bot itself. Some firewalls verify crawler IP ranges too, so a curl from your laptop can be blocked while the real bot passes — or vice versa.
  • The reverse check is your server logs: search them for the agent names above. Seeing GPTBot in your logs is the only definitive proof it visits — and not seeing it after weeks, while robots.txt allows it, usually just means your site has not reached its crawl radar yet.

Can ChatGPT read my website?

The question usually mixes three different mechanisms, and the answer can differ for each:

  • Live reading: paste your URL into ChatGPT and it fetches via ChatGPT-User. If that agent is allowed (or simply not named) in your robots.txt and your firewall serves it, yes — ChatGPT can read the page on demand.
  • Search answers: whether your pages appear in ChatGPT search depends on OAI-SearchBot having indexed them.
  • Training: whether your content shapes future models depends on GPTBot (and CCBot, since public crawl corpora feed many models).

Run the AI crawler report from method 2 and you get all three answers for OpenAI, Anthropic, Google, and Perplexity in one pass.

Should you block AI crawlers?

That is a policy decision, not a syntax one, and it involves a real trade: blocking training bots protects your content from model training, while blocking search and user-agent bots makes you invisible in an answer surface a growing share of people use instead of Google. Whatever you decide, decide it per bot — the blanket block-everything recipe circulating since 2023 also removes the citations that send traffic. We cover the SEO side of that trade-off in Does AI-generated content hurt your SEO?, and the checking side is free: validate first, then edit with intent.

Keep going

Or scan any URL with the free CrawlRanker SEO audit — seven SEO checks plus an AI-content score in about 30 seconds.