Internet-scale bot-vs-human telemetry: per-domain, per-crawler-class request data distinguishing AI-training crawlers from search crawlers and human visitors, with crawler identity (GPTBot, ClaudeBot, Bytespider, PerplexityBot, Googlebot), declared crawl purpose (training / search / aggregation), crawl volume, and the matched referral traffic back to the crawled property — yielding a crawl-to-click ratio per AI platform per publisher. Aggregated in Radar AI Insights; underlying form is edge request telemetry across millions of customer domains.
Equities in US. Related tickers NET, GOOGL, TTD, NWSA, NEWS, DLB, CRWV.
Tabular time series (timestamped hourly/daily aggregates by domain, country, ASN, crawler user-agent, purpose) queryable through the Radar API; underlying form is structured HTTP edge request records
Enormous: bot/human classification events in the 10^12+/month range raw, since a large share of global HTTP traffic passes the edge. The realistically licensable slice — per-domain daily crawl-purpose counts plus matched referral counts — is still billions of aggregate rows per year across millions of domains
Free labels: crawler identity and declared purpose per request (training vs search vs action-taking agent); bot vs verified-human; AI-engine referral source per visit; crawled-but-never-referred publisher set (the crawl-to-click imbalance); content-producer vs AI-consumer classification of every domain on the network; per-platform crawl-share time series
Cloudflare — Internet-scale bot-vs-human telemetry: per-domain, per-crawler-class request data distinguishing AI-training crawlers from search crawlers and human visitors, with crawler identity (GPTBot, ClaudeBot, Bytespider, PerplexityBot, Googlebot), declared crawl purpose (training / search / aggregation), crawl volume, and the matched referral traffic back to the crawled property — yielding a crawl-to-click ratio per AI platform per publisher. Aggregated in Radar AI Insights; underlying form is edge request telemetry across millions of customer domains.. Very large: ~5,200 employees, ~$2.…
Not yet on file: Sample · Data dictionary · Data frequency · Reporting lag · Point-in-time · License