Blog

Practical guides, technical explainers, and field notes for developers building search, research, and retrieval into AI products with WebSearchAPI.ai.

James Bennett

AI Crawlers

Monthly AI Crawler Report: June 2026 — ClaudeBot Surges to #2 as Bytespider Reverses

Analysis of AI crawler traffic trends from June 2026. Anthropic's ClaudeBot exploded +66% to become the #2 AI crawler at 20.0%, leapfrogging Meta-ExternalAgent and GPTBot in the largest single-crawler surge on record. Bytespider's surge reversed, falling from 10.1% to 7.3%. Googlebot dropped below 25% (24.9%) to a new all-time low. Training crawling climbed to 47.4% while Search hit a record 10.5% and Claude-SearchBot rose to 3.3%, extending Anthropic's lead as the largest dedicated AI search crawler. Includes new E-E-A-T insights for AI citability, plus a full breakdown of crawler market share, industry targeting, crawl-to-refer ratios, robots.txt directives, and a Workers AI shift toward embeddings — all from Cloudflare Radar data.

James Bennett

AI Crawlers

The Search Engine Referral Report: Google 91.6% Dominance & the ChatGPT Referral Surge (July 2026)

Google controls 91.6% of search engine referrals and 88.1% of all web traffic. ChatGPT referrals 5x'd since January to 1.05% — now the largest AI referrer — while OpenAI's crawl-to-refer ratio collapsed from 1,284:1 to 217:1. Claude-User became the #2 bot on the web, and the three AI bot categories combined (35.4%) overtook search-engine crawlers for the first time. Analysis of fresh Cloudflare Radar data (June 23 – July 21, 2026) with production E-E-A-T insights from running WebSearchAPI.ai infrastructure.

Give your AI a wider view of the live web.

Search multiple engines and public communities, fetch the evidence, and return one ranked, model-ready result set.