Batch check

One domain per line, up to 25. Checks run one at a time to stay polite to the archives; a fresh batch takes roughly a minute per domain, cached domains are instant.

Crawl reference: , from Common Crawl's public archives

Gate 1 Permissionrobots.txt is a small text file on your site that tells crawlers which paths to stay out of. Access is presumed unless you disallow it, so if it disallows CCBot you have opted out of the open dataset that trains AI models.

checking

Does robots.txt allow CCBot?

Gate 2 EdgeYour CDN or web application firewall (WAF) sits in front of your site and can turn bots away before they reach your pages, even when robots.txt says yes. This gate shows the real responses CCBot got: a 200 means it was let in, a 403 or 429 means it was blocked.

checking

What did CCBot actually receive from your CDN and firewall?

Gate 3 RenderingCrawlers save the raw HTML your server sends, before any JavaScript runs. If your homepage assembles itself in the browser with JavaScript, the crawler stores an almost-empty shell, present in the dataset but saying nothing.

checking

How many real words are in the raw HTML the crawler stored?

Stored homepage words, last 12 crawls

The persistent core versus the churning shell: how consistently the homepage lands in each monthly snapshot, and at what depth. An × marks a crawl with no homepage capture.

reading 12 monthly indexes, this can take a minute…

Every AI crawler, all three gates

allowed / served blocked no response / partial not applicable

Permission is the effective robots.txt policy for the homepage; "named" means the bot appears by name, "default" means it inherits the wildcard rules. Edge and rendering come from a live homepage fetch sending each crawler's own User-Agent from this server, which catches UA-based blocking and cloaking. These probes come from this server's IP, not the crawlers' verified ranges, so an edge that checks IPs may 403 a probe while admitting the real bot, or admit a probe while blocking the real bot. The browser control row helps you read the difference: bot blocked while the control passes means the edge is UA-selective; both blocked is marked inconclusive. Only CCBot has ground truth here, and it is in the gates above, straight from the corpus.

probing crawlers…

Crawler	Permission (robots.txt)	Edge (live probe)	Rendering (words served)

How this works

Permission. A live fetch of your robots.txt, reading every AI-crawler directive, not just CCBot.
Edge. The real HTTP status codes CCBot received, read from Common Crawl's public archives for the latest monthly crawl. A site can allow CCBot on paper and still show nothing but 403s and 429s here. That gap is the whole point.
Rendering. The visible-word count of your homepage exactly as the crawler stored it in the WARC archive, with no JavaScript executed.

Crawled, trained, and retrieved are three different things, and all three start with being crawled. Methodology from The AI Visibility Audit, Stephen Burns, Web Intelligence, Common Crawl Foundation. Every number comes from Common Crawl's public archives.

Glossary

Common Crawl: A nonprofit that crawls the open web every month and publishes the results as a free, public dataset. It is the single largest public source of the text that trains AI models.
CCBot: The web crawler Common Crawl uses to fetch pages. If CCBot cannot read your site, your words never enter that public dataset.
robots.txt: A small text file at the root of your site that tells crawlers which paths to stay out of. It is an exclusion protocol, not a lock: access is presumed unless you disallow it, and only well-behaved crawlers obey it.
Edge (CDN / WAF): Your content delivery network and web application firewall sit in front of your site and can block bots before they reach your pages, even when robots.txt allows them. A 403 (forbidden) or 429 (too many requests) here means the crawler was turned away at the door.
Rendering: Crawlers save the raw HTML your server sends, before any JavaScript runs. A page built entirely in JavaScript looks full in a browser but reaches the crawler as an almost-empty shell.
Empty shell: A page that is technically in the dataset but carries almost no readable words, because its content is assembled by JavaScript the crawler never runs.
WARC: The archive format Common Crawl stores each captured page in: the exact raw response the crawler received, headers and HTML, with no JavaScript executed.
Web Graph: Common Crawl's map of how sites link to one another, at the domain and host level. It is used to decide which sites to crawl more deeply and often.
Harmonic Centrality: A score derived from the Web Graph that measures how central a site is, how few clicks it sits from the well-connected core of the web. A rank closer to #1 means higher crawl priority. It rewards being linked to from central sites, not just being linked to a lot.
Crawl (CC-MAIN-YYYY-NN): One monthly snapshot of the web. The code names the year and the ISO week it began, so CC-MAIN-2026-25 is the crawl that started in week 25 of 2026 (June).
AI crawler: A bot that fetches web pages for an AI company, whether to train a model (GPTBot, ClaudeBot, Google-Extended) or to answer a live question (ChatGPT-User, Perplexity-User). Each can be allowed or blocked separately in robots.txt.
Parametric memory: What a model learned during training and now carries in its weights, what it "knows" without looking anything up. It depends on your pages having been crawled and trained on before the model's cutoff date. This is the layer Common Crawl feeds, and the one this tool measures.
Retrieval (RAG): Content a model fetches live at the moment it answers, instead of recalling it from memory. It depends on being reachable by retrieval and search bots right now, a separate permission from the training crawlers. A brand can be strong in one layer and absent from the other.
Why you can still appear even if you block crawlers: Blocking CCBot stops your pages from entering future training data, but it does not make you vanish from AI answers. A model may already hold you in parametric memory from earlier crawls, it can retrieve live third-party pages about you, and it can describe you from resellers, marketplaces, directories, and reviews. The real cost of blocking is not disappearing, it is being represented by everyone except yourself, in their words and on their terms.