Check if your site blocks GPTBot, ClaudeBot and other AI crawlers
Free tool that reads your robots.txt and shows which AI crawlers are allowed or blocked, including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot.
AI assistants are becoming a real source of traffic. Many sites block their crawlers without knowing it, through a CMS setting, a security plugin or a robots.txt copied from somewhere else.
Crawlers this tool checks
| User agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT search |
| ClaudeBot | Anthropic | Model training |
| PerplexityBot | Perplexity | Perplexity search |
| Google-Extended | Gemini training and grounding (not Google Search) | |
| CCBot | Common Crawl | Open web dataset |
How to allow or block them
Add a group per crawler to robots.txt, for example:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Checks included
| Check | Severity | How to fix |
|---|---|---|
| AI crawlers are blocked in robots.txt Blocked AI crawlers cannot read your site, so it is less likely to be cited by ChatGPT, Claude, Perplexity or Google's AI features. Fine if intentional. | low | If you want AI visibility, allow these user agents in robots.txt. |
This free tool runs the full passive Wardzo scan and shows these checks first. Nothing is stored in full: secrets are masked, and anonymous reports are deleted after 30 days.
Frequently asked questions
Should I block AI crawlers?
It is a business decision. Blocking training crawlers such as GPTBot or CCBot keeps your content out of future training data. Blocking search crawlers such as OAI-SearchBot or PerplexityBot can remove you from AI search answers. Wardzo shows you the current state so the choice is deliberate.
Does robots.txt actually stop AI crawlers?
The major AI companies say their crawlers respect robots.txt. It is a request, not an enforcement mechanism.
Related
- What is robots.txt?
robots.txt is a text file at the root of a website that tells crawlers, including search engines and AI crawlers, which paths they may fetch.
- SEO health monitoring
Catch the SEO mistakes that silently remove you from Google and AI search, including accidental noindex, robots.txt blocks, missing sitemaps, wrong canonicals, broken pages and blocked AI crawlers.
- Noindex and indexability checker
Free indexability checker. Find accidental noindex tags and X-Robots-Tag headers, robots.txt rules that block Google, missing sitemaps, off-site canonicals and broken pages.