Known Agents

Known Agents — Attributed bot / AI-agent web-traffic logs: per-request records identifying which of 2,000+ known crawlers, scrapers and AI agents visited which page, when, and with what outcome (error, redirect, blocked), plus spoof-verification flags. Small company, outsized label quality. Single-employee NYC startup, but the data asset is a cross-site labeled corpus of AI-agent访问 identity + intent that no web-scale analytics vendor publishes. Scale should be valued on taxonomy uniqueness and cross-domain coverage rather than headcount. Use cases: Training classifiers that distinguish human …

Known Agents offers (Alternative, Reference, Sentiment) — Potentially billions of agent-visit events cumulatively — the page claims half of web visitors are bots/agents, so an instrumented mid-size retail site alone can emit millions of classified agent requests per month; the vendor's own 2,000+ agent reference directory is a fixed ~2,000-row labeled entity table.

Training classifiers that distinguish human vs. agent vs. scraper traffic; building AI-agent attribution and content-licensing-metering models; retailer competitive-pricing monitoring detection; LLM training-data provenance forensics (which AI Data Scraper pulled which content); bot-mitigation RL environments; web-crawl scheduling and robots-policy compliance models

The data is with 2 years of history.

Coverage spans US; Application Software; alternative, reference, sentiment; equities.