Separate AI search access from training

Choose the policy before the rule

The question “should I block AI crawlers?” has no single answer, because the bots are not one thing. The documentation from the vendors divides them into roughly three jobs, and each job has a different consequence for you.

Each setting is independent. OpenAI says a site can allow OAI-SearchBot for search and disallow GPTBot for training, and that when both are allowed it may use a single crawl for both purposes.^[1] The Web Almanac found these bots appearing more often in robots.txt files: GPTBot was named in about 4.5% of desktop sites’ files in 2025, up from 2.9% in 2024, with ClaudeBot at 3.6%, Google-Extended at 3.4% and PerplexityBot at 2.8%.^[7]

Review the whole delivery path

Robots.txt is only one layer. A crawler’s request passes through a CDN or firewall, then your server, then your robots rules and finally any page-level directives. A block at any layer produces the same outcome, and it is easy to set one and forget another. Cloudflare, for example, announced on 1 July 2025 that it was changing its default to block AI crawlers unless they pay creators for content, so check that setting if your site sits behind it and you meant to allow some bots.^[8] Vercel’s analysis of AI crawler traffic notes that a platform firewall rule can block AI bots separately from robots.txt.^[9]

Check also that robots.txt itself is healthy. Across the 2025 Web Almanac crawl, about 85% of sites returned a valid 200 for it and about 13% returned a 404, which means no restrictions at all.^[7] Google’s robots.txt report in Search Console shows which files Google found for your hosts, when they were last crawled and any errors.^[10]

Keep private content private

Robots.txt is a convention, not a lock. The standard says its rules are not a form of access authorization,^[11] and the file is public, so a Disallow line lists the very paths you wanted to hide. Anything confidential belongs behind authentication. For pages that should not appear in results, use noindex and leave the page crawlable so the directive can be read, as the crawl and index chapter explains. OpenAI notes that a page blocked for its search crawler can still appear as a navigational link.^[1]

Bear in mind what blocking cannot undo. A crawler you disallow today does not remove what it fetched before, and user-initiated fetchers may not follow robots rules at all, since a person asked for the page. If material must not be fetched by anyone, it should not be publicly reachable.

Write a policy matrix

Write the decision down per service before touching the file. A business can reasonably choose any of these, and each trade-off is yours to weigh. This is not legal advice, and licensing or copyright concerns may point to a different choice.

The second policy looks like this. Note the repeated rule in the search group: Google’s crawlers apply only the one group that most specifically matches their user agent and ignore the others.^[12] A bot given its own group no longer reads the wildcard group, so anything you want to apply to it must be repeated.

# Search crawlers welcome, with the same exclusions as everyone
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /cart/

# Training crawlers blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else
User-agent: *
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

Test a change before broad release

After editing, fetch the file you actually serve, since caches, redirects and CDNs can change it: curl -s https://example.com/robots.txt. Keep it under 500 KiB, because Google ignores content beyond that limit.^[12] Remember the delays: Google generally caches robots.txt for up to 24 hours,^[12] and OpenAI says changes can take about 24 hours to be reflected.^[1] Then check the layers in front of it by requesting a page with each crawler’s name in the user agent string.

for ua in OAI-SearchBot GPTBot ClaudeBot PerplexityBot; do
  printf "%s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://example.com/
done

A 403 or 429 for a bot you meant to allow points to a firewall or rate limit. This test is a spot check, not proof: some firewalls verify bots by IP address and not by user agent. The vendors publish their address ranges, and Anthropic warns that blocking its IPs may not opt you out reliably because the bots could then not read your robots.txt.^[2]

Interpret the logs narrowly

Server logs show which bots visited, how often and what they requested, which is useful for confirming that a rule works. They do not show that the content was used, cited or ranked, and a visit by an AI crawler is not an answer appearance. User agent strings can be spoofed, so verify important cases against the vendors’ published IP ranges. For scale: in Vercel’s December 2024 analysis, GPTBot made about 569 million requests in a month across the Vercel network, against 4.5 billion for Googlebot, and a third of ChatGPT’s and Claude’s fetches returned 404s.^[9] Use logs to check access, not to guess at visibility; for that, use the method in the citation audit chapter.

Apply it to your website

• You have written down a policy for search, training and user-initiated access, per vendor.
• robots.txt names each crawler’s documented token, and repeated rules appear in every group that needs them.
• CDN, firewall and hosting rules were checked and match the policy.
• Private material is behind authentication, and pages to exclude from results carry noindex and stay crawlable.
• robots.txt returns 200, stays under 500 KiB and has been tested after each change.
• You verify bots against published IP ranges and do not treat crawl volume as evidence of citations.
• The policy record has an owner and a review date, because bots are added and renamed.