AI Search & Emerging

ClaudeBot

Anthropic's training crawler; restricting it signals that a site's future content should be excluded from Claude model training datasets.

By Brandon MinaCEO, BrandPilot AIUpdated November 2026

Short answer

ClaudeBot is Anthropic's web crawler used to collect content for training its Claude models. Restricting ClaudeBot in a site's robots.txt signals that the site's future content should be excluded from Claude training datasets. It is distinct from the crawlers and agents Anthropic uses for search and for fetching pages at users' request.