AI crawler checker
Can ChatGPT, Claude, Perplexity and Google's AI actually read your website? Check your robots.txt and firewall in about ten seconds.
- robots.txt rules for 13 AI and search crawlers
- Firewall or CDN blocks, by visiting as each AI crawler
- Noindex instructions that hide the page
- Whether you have an llms.txt file
Example result for a made-up dental practice. Enter your site above to check yours.
- Foundrobots.txt
Read and checked against every crawler below.
- OKIndexing
No noindex instruction on this page.
- Nonellms.txt
Optional. A short llms.txt summary of your site is cheap to add, though few AI tools rely on it yet.
AI search
These decide whether AI search tools can find and recommend you. Keep them allowed.
| Crawler | What it does | robots.txt | Firewall test |
|---|---|---|---|
| GooglebotGoogle | Google Search, including AI Overviews and AI Mode | Allowed | Verified by IP address, so it can't be tested from outside |
| BingbotMicrosoft | Bing and Copilot. ChatGPT search also draws on Bing's index | Allowed | Verified by IP address, so it can't be tested from outside |
| OAI-SearchBotOpenAI | Lets ChatGPT search show and link to your site | Allowed | Got in |
| Claude-SearchBotAnthropic | Lets Claude's search find and cite your site | Allowed | Got in |
| PerplexityBotPerplexity | Perplexity's search index | Allowed | Possibly blockedGot 403; a browser got in |
Fetches on request
These fetch a page when a person asks an AI tool about it, for example by pasting your link. Because a person starts the visit, OpenAI says robots.txt rules may not apply to ChatGPT-User.
| Crawler | What it does | robots.txt | Firewall test |
|---|---|---|---|
| ChatGPT-UserOpenAI | Fetches a page when someone asks ChatGPT about it | Allowed | Got in |
| Claude-UserAnthropic | Fetches a page when someone asks Claude about it | Allowed | Got in |
| Perplexity-UserPerplexity | Fetches a page when someone asks Perplexity about it | Allowed | Possibly blockedGot 403; a browser got in |
Model training
Blocking these is a fair choice and doesn't remove you from AI search. It only affects what future models learn from your pages.
| Crawler | What it does | robots.txt | Firewall test |
|---|---|---|---|
| GPTBotOpenAI | Collects pages to train OpenAI's models | BlockedDisallow: / | Got in |
| ClaudeBotAnthropic | Collects pages to train Anthropic's models | Allowed | Possibly blockedGot 403; a browser got in |
| Google-ExtendedGoogle | Controls use of your pages for Gemini. It doesn't affect Google Search or AI Overviews | BlockedDisallow: / | A robots.txt setting, not a separate crawler |
| Applebot-ExtendedApple | Controls use of your pages to train Apple Intelligence | Allowed | A robots.txt setting, not a separate crawler |
| CCBotCommon Crawl | An open web archive that many AI models are trained on | BlockedDisallow: / | Got in |
The firewall test visits your page using each crawler's name. Real crawlers are also checked by their IP address, so "possibly blocked" means: look at your firewall or CDN bot settings (Cloudflare's AI bot blocking is the usual cause).
Questions
If I block GPTBot, will ChatGPT stop recommending me?
Not directly. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, and fetches pages on request with ChatGPT-User. Block GPTBot and allow OAI-SearchBot if you want to be found without your pages being used for training.
Does Google-Extended affect AI Overviews?
No. Google says Google-Extended only controls use of your pages for Gemini. AI Overviews and AI Mode use the normal Googlebot, so blocking Googlebot is what would remove you.
My robots.txt allows everything but the firewall test says blocked. Why?
Many CDNs and hosting firewalls, Cloudflare included, have settings that block AI crawlers regardless of robots.txt. Check the bot or security settings in your hosting or CDN dashboard.
Why can't you test Googlebot and Bingbot?
Google and Bing confirm their crawlers by IP address, and good firewalls block anything pretending to be them. A test from our servers would be blocked for the wrong reason, so we only check your robots.txt for those two.
Do I need an llms.txt file?
It's optional. It's a short plain-text summary of your site for AI tools. It's cheap to add, but few AI tools rely on it yet, so it won't make up for blocked crawlers or thin pages.