AI Visibility Audit

Common Crawl Data

Is your site visible to AI? Check it.

AI can't cite what it never read.

AI assistants learn what brands, businesses, and people exist from training data, and the largest public source of that data is Common Crawl. This tool checks whether its crawler, CCBot, actually sees your site, at all three gates where visibility fails. Most audits stop at the first.

How this works

  1. Permission. A live fetch of your robots.txt, reading every AI-crawler directive, not just CCBot.
  2. Edge. The real HTTP status codes CCBot received, read from Common Crawl's public archives for the latest monthly crawl. A site can allow CCBot on paper and still show nothing but 403s and 429s here. That gap is the whole point.
  3. Rendering. The visible-word count of your homepage exactly as the crawler stored it in the WARC archive, with no JavaScript executed.

Crawled, trained, and retrieved are three different things, and all three start with being crawled. Methodology from The AI Visibility Audit, Stephen Burns, Web Intelligence, Common Crawl Foundation. Every number comes from Common Crawl's public archives.

Glossary

Common Crawl
A nonprofit that crawls the open web every month and publishes the results as a free, public dataset. It is the single largest public source of the text that trains AI models.
CCBot
The web crawler Common Crawl uses to fetch pages. If CCBot cannot read your site, your words never enter that public dataset.
robots.txt
A small text file at the root of your site that tells crawlers which paths to stay out of. It is an exclusion protocol, not a lock: access is presumed unless you disallow it, and only well-behaved crawlers obey it.
Edge (CDN / WAF)
Your content delivery network and web application firewall sit in front of your site and can block bots before they reach your pages, even when robots.txt allows them. A 403 (forbidden) or 429 (too many requests) here means the crawler was turned away at the door.
Rendering
Crawlers save the raw HTML your server sends, before any JavaScript runs. A page built entirely in JavaScript looks full in a browser but reaches the crawler as an almost-empty shell.
Empty shell
A page that is technically in the dataset but carries almost no readable words, because its content is assembled by JavaScript the crawler never runs.
WARC
The archive format Common Crawl stores each captured page in: the exact raw response the crawler received, headers and HTML, with no JavaScript executed.
Web Graph
Common Crawl's map of how sites link to one another, at the domain and host level. It is used to decide which sites to crawl more deeply and often.
Harmonic Centrality
A score derived from the Web Graph that measures how central a site is, how few clicks it sits from the well-connected core of the web. A rank closer to #1 means higher crawl priority. It rewards being linked to from central sites, not just being linked to a lot.
Crawl (CC-MAIN-YYYY-NN)
One monthly snapshot of the web. The code names the year and the ISO week it began, so CC-MAIN-2026-25 is the crawl that started in week 25 of 2026 (June).
AI crawler
A bot that fetches web pages for an AI company, whether to train a model (GPTBot, ClaudeBot, Google-Extended) or to answer a live question (ChatGPT-User, Perplexity-User). Each can be allowed or blocked separately in robots.txt.
Parametric memory
What a model learned during training and now carries in its weights, what it "knows" without looking anything up. It depends on your pages having been crawled and trained on before the model's cutoff date. This is the layer Common Crawl feeds, and the one this tool measures.
Retrieval (RAG)
Content a model fetches live at the moment it answers, instead of recalling it from memory. It depends on being reachable by retrieval and search bots right now, a separate permission from the training crawlers. A brand can be strong in one layer and absent from the other.
Why you can still appear even if you block crawlers
Blocking CCBot stops your pages from entering future training data, but it does not make you vanish from AI answers. A model may already hold you in parametric memory from earlier crawls, it can retrieve live third-party pages about you, and it can describe you from resellers, marketplaces, directories, and reviews. The real cost of blocking is not disappearing, it is being represented by everyone except yourself, in their words and on their terms.