Skip to content
Crawlable
frSign in
DatasetYour call — not scored

AI2Bot

Non-profit research crawler feeding open, documented datasets.

robots.txt token
AI2Bot
Ai2Bot-Dolma
Operator
Allen Institute for AI ↗
Powers
open research datasets (Dolma)
Honours robots.txt
Yes, documented

Allow it

User-agent: AI2Bot
Allow: /

Block it

User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
Disallow: /

Need the whole file, with every crawler and your private paths — and the llms.txt to go with it? The generator writes them. AI config generator →

User-Agent

Mozilla/5.0 (compatible; AI2Bot/1.0; +https://allenai.org/crawler) Crawlable/1.0; +https://crawlable.fr/probe

This is the string we send when probing. The Crawlable suffix identifies us in your logs.

Check your own site

robots.txt is only half the answer: your CDN can refuse AI2Bot before it ever reads the file. A scan checks both.

Free, no account, about 5 seconds.