Datasets
Buyer room
AEO.TECHSHU.AIManage supplier listing
AEO Optima (TechShu AI / INT TechShu) — Longitudinal AI-crawler permission + activity ledger per monitored site, built on an unusually rigorous hand-curated crawler registry. Three facts kept deliberately separate: (1) what a site's robots.txt / ai.txt / llms.txt actually declares for each named crawler (allowed / blocked, with a 'Change since last check' diff column and check timestamps); (2) which crawlers genuinely fetched the site, classified against a ~19-entry registry that maps each user-agent to its operator AND its purpose (GPTBot/ClaudeBot/Amazonbot/CCBot/Bytespider/Meta-ExternalAge…
supplier index read from their site
Longitudinal AI-crawler permission + activity ledger per monitored site, built on an unusually rigorous hand-curated crawler registry. Three facts kept deliberately separate: (1) what a site's robots.txt / ai.txt / llms.txt actually declares for each named crawler (allowed / blocked, with a 'Change since last check' diff column and check timestamps); (2) which crawlers genuinely fetched the site, classified against a ~19-entry registry that maps each user-agent to its operator AND its purpose (GPTBot/ClaudeBot/Amazonbot/CCBot/Bytespider/Meta-ExternalAgent = training; OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Perplexity-User, Amzn-User, MistralAI-User, DuckAssistBot, Meta-WebIndexer = answering a live query; Google-Extended and Applebot-Extended = permissions that never fetch; OAI-AdsBot and Google-CloudVertexBot = explicitly not AI crawlers); (3) recommended policy state, which surfaces conflicts between robots.txt and AI-specific directives. The registry itself — with operator statements about training use, including Meta's stated willingness to bypass robots.txt — is a maintained reference asset.
Coverage Information Technology · Communication ServicesAsset class EquitiesTickers GOOGL · META · AMZN
Sample, licence terms, pricing and eval results when Techshu publishes them. Until then, discover alternatives today with a 7-day trial.
Start 7-day trial →What buyers ask about Techshu, answered from this page.
AEO Optima (TechShu AI / INT TechShu) — Longitudinal AI-crawler permission + activity ledger per monitored site, built on an unusually rigorous hand-curated crawler registry. Three facts kept deliberately separate: (1) what a site's robots.txt / ai.txt / llms.txt actually declares for each named crawler (allowed / blocked, with a 'Change since last check' diff column and check timestamps); (2) which crawlers genuinely fetched the site, classified against a ~19-entry registry that maps each user-agent to its operator AND its purpose (GPTBot/ClaudeBot/Amazonbot/CCBot/Bytespider/Meta-ExternalAge…
Techshu offers (Alternative, Reference, Sentiment) — Low per-site but structurally complete: each monitored site yields ~19 crawler rows per check with change history, and checks are point-in-time on demand rather than continuous, so hundreds of records per site per year rather than millions. Absolute portfolio size is not published; the ~19-crawler registry × monitored-site × check-date matrix is the deliverable unit — request the site count in outreach..
Content-licensing addressable-market models (which publishers permit training crawlers by vertical, over time — the direct denominator for AI content licensing); AI-copyright and text-and-data-mining compliance evidence (permission state plus observed fetches, i.e. permission-vs-actual-crawl discrepancy records); measuring AI-lab crawl-intensity shifts as a leading indicator of model-training cycles; training-data provenance and consent auditing; GEO/AEO benchmarking feeds.
Coverage spans US, APAC; Application Software, Publishing; alternative, reference, sentiment; equities.
Techshu has not added their own details yet. Not yet on file: