Claude-SearchBot

Anthropic crawler that indexes web content to improve Claude's search results; blocking it can reduce a site's visibility and citations in Claude answers.

By Brandon MinaCEO, BrandPilot AIUpdated November 2026

Short answer

Claude-SearchBot is an Anthropic web crawler that indexes content to improve the quality of Claude's search results. It is separate from Anthropic's training crawler. Blocking Claude-SearchBot in robots.txt can reduce a site's visibility and citations in Claude's answers, so publishers weigh search exposure separately from training-data preferences.