AgentBlocking it costs you visibility
Claude-Code
Coding agents read docs live. Relevant if developers are your audience.
- robots.txt token
Claude-Code- Operator
- Anthropic
- Powers
- Claude Code documentation lookups
- Honours robots.txt
- Not documented
Allow it
User-agent: Claude-Code
Allow: /Block it
User-agent: Claude-Code
Disallow: /Need the whole file, with every crawler and your private paths — and the llms.txt to go with it? The generator writes them. AI config generator →
User-Agent
Mozilla/5.0 (compatible; Claude-Code/1.0; +https://www.anthropic.com) Crawlable/1.0; +https://crawlable.fr/probeThis is the string we send when probing. The Crawlable suffix identifies us in your logs.
Anthropic
Check your own site
robots.txt is only half the answer: your CDN can refuse Claude-Code before it ever reads the file. A scan checks both.
Free, no account, about 5 seconds.