For any domain: how many pages Common Crawl captured per crawl, when an AI-crawler block started, whether the archived copy has drifted, how the edge treats CCBot, sitemap coverage, and web-graph crawl priority. Built on the static Common Crawl data, no signup.
New to this? Try nytimes.com (a major site that blocks AI crawlers), bbc.com, or example.com. You can paste a full page URL too.
First check takes about 20 seconds - it scans the last 12 monthly crawls across 100M+ domains. After that, repeat checks are cached and instant.
How this works, and what it does not mean
Every number here comes from Common Crawl's own public files - a free, monthly crawl of billions of pages that most AI training sets are built from. For your domain the tool reads each crawl's index to count what CCBot (Common Crawl's crawler) stored, walks robots.txt backward month by month to date any block, pulls the exact bytes CCBot archived for your homepage, and fetches your homepage live as CCBot, a real browser, and a fake Googlebot to see how your edge treats bots.
Being in Common Crawl is not the same as being in a model. Every lab filters the data, so coverage here is a strong signal, not a guarantee.
Thin coverage is not proof of a block. Common Crawl is a sample of the web, not a full copy. When the signals disagree, the panels say so instead of forcing one answer.
Lower crawl-priority rank is better. It means your domain sits closer to the center of the web graph and gets crawled more often.
Counts for very large sites are estimates (accurate to about 3,000 pages), and the panel says so. Finished crawls never change, so repeat checks are cached and instant.