You probably check Google Search Console now and then. But do you ever check what AI crawlers are doing on your site? Most owners I talk to haven’t, and that’s a gap worth closing. If GPTBot, ClaudeBot or PerplexityBot can’t get in, AI tools may never mention your pages.
The worst part is how quiet it is. Your site loads. Your Google traffic looks fine. And a newer kind of search simply can’t see you.
What AI crawlers are
AI crawlers are not all doing the same job. Some collect content to help train AI models, while others look for information that can be used in AI search results. There are also crawlers that only fetch a page when a user asks an AI service for specific information. Understanding these differences makes it easier to decide which bots you want to allow and which ones you may want to block.
Training crawlers
These collect text that a company may use to train its models. Your page goes into a big dataset, and nobody sees it in a live answer. GPTBot (OpenAI) and ClaudeBot (Anthropic) are the main examples.
Google-Extended works a little differently. You’ll never see it in your logs. It’s a switch in robots.txt that tells Google whether your content can feed Gemini and other AI features.
AI search crawlers
These build the index behind AI search. When someone asks ChatGPT or Perplexity a question and gets an answer with links, a bot like this found those pages first. OAI-SearchBot handles ChatGPT search, and PerplexityBot handles Perplexity. Block them, and you lose your chance at those links.
User-triggered fetchers
These crawlers work a little differently. For example, someone might paste your website URL into a chat and ask the AI to summarize the page. The service may then fetch that specific page to get the information it needs.
The way these fetchers work can vary from one AI company to another, including how they handle robots.txt. So before blocking or allowing one, check the relevant company’s documentation to understand how its crawler works.
Why the wrong block costs you
Plenty of owners don’t want their work used for training. That’s a fair position. The trouble starts when the fix is “block every AI bot,” because that also blocks the search crawlers that could send you readers.
You can block GPTBot and still allow OAI-SearchBot. That’s a common middle path. Blocking both is a different choice with different results. Pick on purpose.
How robots.txt handles these bots
Your `robots.txt` file sits at the top level of your website, usually at an address like `yoursite.com/robots.txt`. It is simply a text file that tells crawlers which parts of your site they can access and which ones they should stay away from.
The rules are grouped by bot. Each group starts with a `User-agent` line that identifies the crawler, followed by instructions about which pages or folders it can or cannot access.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /private/
User-agent: *
Allow: /
Here, GPTBot gets nothing. OAI-SearchBot gets everything. ClaudeBot can crawl all but the private folder. Every other bot is allowed in.
PerplexityBot and Google-Extended work in much the same way. You identify the bot with a `User-agent` line and then add the access rule you want. If a crawler doesn’t have its own set of rules, it falls under the User-agent: * section.
One thing to keep in mind is that robots.txt is more like a set of instructions than a security lock. Crawlers that follow the rules will respect them, but the file can’t physically stop a bot from visiting your site. It also doesn’t control what happens at your CDN or firewall level, which is where some unexpected blocking can happen.
How to check your own access
You can read the file yourself. Open it in a browser and search for each bot name. If the file is five lines long, that’s plenty.
Longer files are harder. Wildcards, overlapping paths and rules from three years ago can make the result unclear. For those, Mencoro’s free AI crawler checker saves time. Enter your domain and see whether GPTBot, ClaudeBot, PerplexityBot and the rest show as allowed or blocked. Run it now, and run it again after any change.
Common mistake
1. One blanket Disallow
This is one of the easiest ways to block more crawlers than you intended. A `User-agent: *` group with `Disallow: /` tells crawlers without their own specific rules not to access any part of the site.
It often happens when a website is still on a staging server, where blocking crawlers makes sense, and the same setting accidentally stays in place when the site goes live. Plugins can also add broad rules without making it obvious. So even if you don’t remember changing your `robots.txt` file, it’s worth checking what’s actually in it.
2. CDN rules
Many CDNs and bot-protection tools challenge automated traffic by default. Picture a small blog whose robots.txt welcomes every bot. The owner later switches on a CDN with strict bot protection and never looks at it again. From then on, AI crawlers get a challenge page instead of articles. The robots.txt file still says yes, so the owner has no reason to suspect anything.
3. Firewall settings
Firewalls sometimes flag unfamiliar user agents. Rate limits can also cut a crawler off midway if it requests pages too fast. If you run a security plugin, check its bot rules and limits.
4. Assuming it’s all open
Nobody remembers adding a block, so people assume there isn’t one. Yet plugins, hosting defaults and old developers all edit these settings. Test instead of guessing.
5. After you change something
Once you’ve updated robots.txt, a CDN setting, or a firewall rule, give the changes a little time to take effect. Then open yoursite.com/robots.txt and check that the live version actually shows your changes.
Run the checker again and go through the bots one by one. If a crawler is still showing as blocked even though your robots.txt looks right, check your CDN and firewall settings next. That’s often where the block is coming from.
The takeaway
Open your robots.txt today, decide which AI bots you want, and test the result. If a bot you wanted is blocked, trace it past robots.txt to your CDN and firewall.
