DatasetYour call — not scored
CCBot
One crawl, many models: Common Crawl is the training corpus behind a large share of open-weight LLMs.
- robots.txt token
CCBot- Operator
- Common Crawl ↗
- Powers
- the Common Crawl dataset used by most open models
- Honours robots.txt
- Yes, documented
Allow it
User-agent: CCBot
Allow: /Block it
User-agent: CCBot
Disallow: /Need the whole file, with every crawler and your private paths — and the llms.txt to go with it? The generator writes them. AI config generator →
User-Agent
CCBot/2.0 (https://commoncrawl.org/faq/) Crawlable/1.0; +https://crawlable.fr/probeThis is the string we send when probing. The Crawlable suffix identifies us in your logs.
Check your own site
robots.txt is only half the answer: your CDN can refuse CCBot before it ever reads the file. A scan checks both.
Free, no account, about 5 seconds.