Blog

Practical guides, technical explainers, and field notes for developers building search, research, and retrieval into AI products with WebSearchAPI.ai.

James Bennett

AI Crawlers

Monthly AI Crawler Report: July 2026 — ClaudeBot's Surge Reverses as Meta Rebounds

Analysis of AI crawler traffic trends from July 2026. Anthropic's record June surge reversed: ClaudeBot fell -22% from 20.0% to 15.6%, though it held #2 — exactly the partial retracement this report forecast. Meta-ExternalAgent rebounded +27% to 12.9%, ending a three-month decline, while Bytespider kept sliding to 4.7%. Search crawling hit a new record 11.8% as Training fell to 43.8% and Mixed Purpose rose for the first time in seven months. Mistral's crawl-to-refer ratio collapsed to effectively zero referrals, displacing Anthropic as the most extractive operator. Includes robots.txt adoption measured against a real denominator for the first time, a Workers AI shakeup as Z.ai's GLM-4.7-Flash unseats Gemma 4, plus E-E-A-T guidance for AI citability — all from Cloudflare Radar data.

James Bennett

AI Crawlers

The Search Engine Referral Report: Google 91.6% Dominance & the ChatGPT Referral Surge (July 2026)

Google controls 91.6% of search engine referrals and 88.1% of all web traffic. ChatGPT referrals 5x'd since January to 1.05% — now the largest AI referrer — while OpenAI's crawl-to-refer ratio collapsed from 1,284:1 to 217:1. Claude-User became the #2 bot on the web, and the three AI bot categories combined (35.4%) overtook search-engine crawlers for the first time. Analysis of fresh Cloudflare Radar data (June 23 – July 21, 2026) with production E-E-A-T insights from running WebSearchAPI.ai infrastructure.

Give your AI a wider view of the live web.

Search multiple engines and public communities, fetch the evidence, and return one ranked, model-ready result set.