Should Your Small Business Block AI Crawlers in 2026?

For the first time in the web's history, most of the visitors to your website aren't people. Cloudflare reported that automated bots crossed 57.5% of all HTML requests across its network in mid-2026, and the AI-specific crawlers inside that number are growing faster than anything else — AWS puts AI bot traffic growth at more than 300% year over year.

This stopped being a niche concern on September 15, 2026, when Cloudflare began blocking “mixed-use” AI crawlers by default on any page that serves ads — a change that reaches millions of sites whose owners never opened a settings panel. Whether you run a plumbing company, a Shopify store, or a local news site, you now have to decide on purpose what AI bots are allowed to do with your content.

The instinct for many owners is to slam the door. But blocking AI crawlers is a trade-off, not a free win: done carelessly, it removes your business from ChatGPT, Perplexity, and Google's AI answers at the exact moment those surfaces are starting to send real traffic. This guide breaks down what these bots actually do, what blocking changes, and the four policies that make sense for a small business.

Not every AI bot wants the same thing

The single most important thing to understand in 2026 is that the big AI companies now run separate crawlers for separate jobs, and those crawlers respond very differently to being blocked.

Training crawlers

GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended, and Bytespider (TikTok) collect text to train or improve large language models. Blocking them keeps your content out of future model training. It does not remove you from any search product, and it has zero effect on your Google rankings — these bots are completely separate from Googlebot.

Search and assistant crawlers

OAI-SearchBot powers ChatGPT's search feature, PerplexityBot feeds Perplexity's answers, and Googlebot plus Google's AI indexing drive AI Overviews and AI Mode. Block these and your business simply cannot be cited when someone asks an AI assistant for a recommendation in your category. For most local and e-commerce businesses, that is the traffic you actually want.

The catch

Some crawlers do double duty, and vendors have been slow to document which is which. That ambiguity is exactly why Cloudflare's September default targets “mixed-use” bots — and why a hand-tuned policy beats a blanket rule.

What blocking actually does (and doesn't do)

• It won't hurt your Google ranking. AI crawlers like GPTBot and ClaudeBot are separate from Googlebot, so blocking them is invisible to classic search.
• It will remove you from the answer you blocked. No crawl, no citation. Blocking OAI-SearchBot means ChatGPT search can't surface your pages.
• It's voluntary. robots.txt is an honor system. Reputable bots from OpenAI, Anthropic, Google, and Perplexity obey it; scrapers that ignore it need a firewall rule, not a text file.
• It's reversible and granular. You can allow one bot, block another, and change your mind next week with no lasting damage.

The four policies that make sense

1. Allow everything

The right call if your content is mostly marketing and promotional pages: you want maximum reach and have little proprietary text to protect. You should still rate-limit abusive crawlers at the edge (policy 3), but you don't need robots.txt blocks.

2. Split policy — block training, allow search

Roughly 30% of the top 1,000 sites now run this configuration: block GPTBot, CCBot, ClaudeBot, Google-Extended, and Bytespider, while explicitly allowing OAI-SearchBot, PerplexityBot, and Googlebot. You keep your visibility in AI answers while keeping your original writing, research, or photography out of training sets. This is the best default for any business that publishes genuinely original content.

3. CDN-level rate limiting

If your problem is server load or bandwidth cost rather than principle, handle it at the edge. Cloudflare's AI Crawl Control, bot-management rules, or a simple WAF rate limit will throttle aggressive crawlers without the bluntness of a full block. Keep the search-and-assistant bots on the allowlist.

4. Pay-per-crawl

Cloudflare's pay-per-crawl returns an HTTP 402 “Payment Required” to AI crawlers and lets you set a price or block them outright; it is now evolving toward “pay per use” pricing tied to how often your content is actually used in answers. Realistically, only sites with large, in-demand content libraries will earn meaningful money — but it costs nothing to enable a charge on training bots while allowlisting the assistants that send referrals.

Setting your policy in 30 minutes

Edit robots.txt

Add a block like this near the top of your robots.txt and adjust it to the policy you picked. Reputable AI crawlers read the file on their next visit.

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Check your CDN defaults before you rely on the file

If you are on Cloudflare, open the AI Crawl Control dashboard and review what changed around September 15, 2026. Confirm the AI search bots you want are still allowed and that the default managed setting matches your intent — a CDN rule overrides robots.txt.

Skip llms.txt for now (mostly)

llms.txt — a proposed file that hands AI models a curated map of your site — sits at roughly 8–10% adoption but almost no measured usage: one study of 500 million AI bot visits found only 408 touched an llms.txt file, and Google has said it will not support it. It is harmless to add, but it is not where your first hour should go.

Watch your logs

Give it two weeks, then check server logs or analytics for crawler activity and any change in AI referral traffic. Adjust one bot at a time so you can tell what each change did.

What most small businesses should do

If you sell local services or products, allow the AI search and assistant crawlers and spend your energy on clean structured data so those bots can actually understand your hours, location, and catalog. If you publish original content, layer the split policy on top. Handle bandwidth pain at your CDN, not with a sledgehammer in robots.txt. And revisit the decision every quarter — the crawler landscape in 2026 changes month to month.

Not sure what is crawling your site, or whether your pages are even readable to an AI assistant? Clariola's free website audit checks your robots.txt, structured data, Core Web Vitals, and AI-answer readiness in one pass and tells you exactly what to fix first. You can also read our guide on getting cited in Google AI Overviews and our walkthrough of generative engine optimization.

Frequently asked questions

Will blocking AI crawlers hurt my Google search ranking? No. GPTBot, ClaudeBot, CCBot, and Google-Extended are all separate from Googlebot. Blocking them has no effect on classic Google rankings or on whether Google can index your site.

If I block GPTBot, can ChatGPT still recommend my business? Only if you leave OAI-SearchBot allowed. GPTBot is for training; OAI-SearchBot is what ChatGPT's search feature uses to find and cite live pages, so block only GPTBot and you stay citable.

Do I need llms.txt to get cited by AI? No. Adoption is under 10% and measured crawler usage is close to zero. Accurate schema markup and crawlable content matter far more.