Brickroad
Brickroad
IndexSignalsBlog

Product

  • Information Frontier Agent
  • Vendor Management Agent
  • Product tour
  • Pricing

Solutions

  • Utility estimation
  • Context assembly
  • Market intelligence

Suppliers

  • Sell your data
  • Start onboarding
  • Programmatic Data License

Community

  • Public index
  • Signals
  • Blog

Company

  • About
  • Team
  • Careers
  • Brand

Connect

  • LinkedIn
  • X
PrivacyTermsCookies

© 2026 Brickroad

/ Index / Cloudflare / Internet-scale bot-vs-human telemetry

Internet-scale bot-vs-human telemetry

Cloudflareblog.cloudflare.com Information TechnologyAlternativeSentimentUSAPI

Internet-scale bot-vs-human telemetry: per-domain, per-crawler-class request data distinguishing AI-training crawlers from search crawlers and human visitors, with crawler identity (GPTBot, ClaudeBot, Bytespider, PerplexityBot, Googlebot), declared crawl purpose (training / search / aggregation), crawl volume, and the matched referral traffic back to the crawled property — yielding a crawl-to-click ratio per AI platform per publisher. Aggregated in Radar AI Insights; underlying form is edge request telemetry across millions of customer domains.

History
5 years
Delivery
API
Format
Tabular time series
Availability
A3 · email-gated
Are you Cloudflare?Manage this listing
OverviewSampleAt a glanceInventory and coverageSupplier
SampleSign in to see the sample, or to ask for one.Sign in

At a glance

From the index
Sector
Information Technology
Data
AlternativeSentimentGeospatial
Industry
IT Services
Themes
AlternativeSentimentGeospatial
Instruments
Equities
Coverage
US
Delivery
API
Direct access
A3 · email-gated
Procurement
MEDIUM

Inventory and coverage

Coverage

Equities in US. Related tickers NET, GOOGL, TTD, NWSA, NEWS, DLB, CRWV.

Format

Tabular time series (timestamped hourly/daily aggregates by domain, country, ASN, crawler user-agent, purpose) queryable through the Radar API; underlying form is structured HTTP edge request records

Volume

Enormous: bot/human classification events in the 10^12+/month range raw, since a large share of global HTTP traffic passes the edge. The realistically licensable slice — per-domain daily crawl-purpose counts plus matched referral counts — is still billions of aggregate rows per year across millions of domains

Structure

Free labels: crawler identity and declared purpose per request (training vs search vs action-taking agent); bot vs verified-human; AI-engine referral source per visit; crawled-but-never-referred publisher set (the crawl-to-click imbalance); content-producer vs AI-consumer classification of every domain on the network; per-platform crawl-share time series

Supplier

Cloudflareblog.cloudflare.com

Cloudflare — Internet-scale bot-vs-human telemetry: per-domain, per-crawler-class request data distinguishing AI-training crawlers from search crawlers and human visitors, with crawler identity (GPTBot, ClaudeBot, Bytespider, PerplexityBot, Googlebot), declared crawl purpose (training / search / aggregation), crawl volume, and the matched referral traffic back to the crawled property — yielding a crawl-to-click ratio per AI platform per publisher. Aggregated in Radar AI Insights; underlying form is edge request telemetry across millions of customer domains.. Very large: ~5,200 employees, ~$2.…

Supplier page

Not yet on file: Sample · Data dictionary · Data frequency · Reporting lag · Point-in-time · License

On this page
  • Overview
  • Sample
  • At a glance
  • Inventory and coverage
  • Supplier