Skip to content
Crawlable
frSign in
DatasetYour call — not scored

Diffbot

Builds a commercial knowledge graph resold to AI companies.

robots.txt token
Diffbot
Diffbot-User
Operator
Diffbot ↗
Powers
Diffbot Knowledge Graph
Honours robots.txt
Not documented

Allow it

User-agent: Diffbot
Allow: /

Block it

User-agent: Diffbot
User-agent: Diffbot-User
Disallow: /

Need the whole file, with every crawler and your private paths — and the llms.txt to go with it? The generator writes them. AI config generator →

User-Agent

Mozilla/5.0 (compatible; Diffbot/1.0; +https://www.diffbot.com) Crawlable/1.0; +https://crawlable.fr/probe

This is the string we send when probing. The Crawlable suffix identifies us in your logs.

Check your own site

robots.txt is only half the answer: your CDN can refuse Diffbot before it ever reads the file. A scan checks both.

Free, no account, about 5 seconds.