Monthly AI Crawler Report: February 2026 Traffic Trends

This report is updated monthly with fresh Cloudflare Radar data. Bookmark this page to track how AI crawlers are reshaping web traffic each month.

Something flipped in February. For the first time since I've been tracking this data, dedicated AI training crawlers now generate more traffic than mixed-purpose bots. Training hit 45.4% of all AI bot requests. Mixed Purpose dropped to 43.9%. That crossover didn't happen gradually, it happened fast.

Meta-ExternalAgent leapfrogged GPTBot to become the world's second-largest AI crawler at 15.6%. Googlebot's share fell another 3.5 percentage points to 34.6%. I pulled 30 days of data covering February 11 through March 13, 2026, and the pattern is clear: the AI crawling game has entered a different phase entirely.

📊 Stats Alert: Dedicated AI training crawlers now generate 45.4% of all AI bot traffic, surpassing Mixed Purpose crawlers (43.9%) for the first time ever.

Who Are the Top AI Crawlers in February 2026?

Googlebot still sits at the top with 34.6% share of all identified AI bot requests. But its lead keeps shrinking. The real story this month is what happened behind it: Meta-ExternalAgent surged past GPTBot to claim second place at 15.6%, followed by GPTBot at 12.1% and ClaudeBot at 11.1% (Anthropic's crawler that powers the Claude Web Search API and their model training pipeline). Those four crawlers alone account for 73.5% of all AI bot traffic.

The top five companies (Google, Meta, OpenAI, Anthropic, Microsoft) control 82.8% of all AI crawler traffic, down from 84.5% in January. They're not crawling less. It's that second-tier crawlers like Amazonbot (+15.0%), Applebot (+17.0%), and OAI-SearchBot (+16.8%) are all growing fast enough to chip away at that concentration.

If you're managing AI bot access on your site, your robots.txt decisions about these five operators still control most of your AI traffic exposure. For context on how crawling translates into actual referral traffic back to websites, check our companion Search Engine Referral Report on crawl-to-refer ratios.

Which AI Crawlers Gained or Lost Ground This Month?

Meta-ExternalAgent keeps accelerating. It jumped from 11.9% to 15.6% of global AI bot traffic this month, a 30.8% relative increase and +3.7 percentage points. That's even bigger than January's +3.1 pp gain. Over just two months, Meta-ExternalAgent has roughly doubled from 8.5% to 15.6%.

I didn't expect this pace to hold. It did.

Four things jumped out at me:

1. Meta has overtaken OpenAI. Meta-ExternalAgent (15.6%) now generates more AI bot traffic than GPTBot (12.1%). Two months ago, GPTBot was comfortably ahead. Meta's Llama training pipeline expansion, including Llama 4 Scout which already shows up on Workers AI, is driving this. At the current trajectory, Meta-ExternalAgent could hit 20% by April 2026.

2. Googlebot's decline is picking up speed. It lost 3.9 pp in January, then another 3.5 pp in February. Over two months, its share dropped from 42.6% to 34.6%, a cumulative 8-point decline. Google probably isn't crawling less in raw volume. But competitors are crawling so much more that Google's share keeps getting diluted.

3. ClaudeBot flatlined. After dropping from 13.0% to 11.1% over December-January, ClaudeBot held at exactly 11.1% in February. Anthropic seems to have found a steady crawling cadence while everyone else ramps up or down around them.

4. The mid-tier grew in lockstep. Amazonbot (+15.0%), Applebot (+17.0%), and OAI-SearchBot (+16.8%) all grew at nearly identical relative rates. That parallel movement tells me this is an industry-wide expansion of AI data collection, not any one company going rogue. If you're building apps that rely on real-time retrieval, understanding what a web search API is and how these bots work under the hood matters more than ever.

What Are AI Bots Actually Doing With the Content They Crawl?

This is the big finding of February's data. Dedicated AI training crawlers now generate more traffic than mixed-purpose bots. First time that's happened.

What does this crossover actually mean if you run a website?

Training is now #1 at 45.4%. GPTBot, ClaudeBot, and Meta-ExternalAgent have crossed the line. The +2.8 pp gain came almost entirely from Meta's ramp-up. The AI industry is deploying more purpose-built training crawlers than it's relying on dual-purpose bots. In plain terms: the content being taken from your site is increasingly going toward model weights, not search indexing. And you can block these crawlers individually, which gives you more control than you had when Googlebot carried the bulk of the traffic.

Mixed Purpose dropped to #2 at 43.9%. Googlebot and Bingbot, which handle both search indexing and AI training simultaneously, lost ground. The -3.7 pp decline mirrors Googlebot's falling share. You still can't separate the AI training from the search indexing with these crawlers. Block Googlebot and you disappear from Google Search. For developers looking at how to ground AI responses with Google Search alternatives, this bundling problem keeps coming up.

Search crawling grew fastest at 8.2%. AI search bots like OAI-SearchBot gained +1.1 pp, a 15.5% relative increase. Second straight month as the fastest-growing category. These bots are tied directly to the growing ecosystem of AI search API alternatives that power real-time retrieval systems.

User Action dipped to 2.0%. Bots like ChatGPT-User that fetch pages when users ask questions slipped -0.2 pp. Could be seasonal. Could be a temporary plateau.

If Training keeps growing at this rate, it'll pass 50% of all AI bot traffic by Q2 2026. That would be a different world from where we were six months ago, when mixed-purpose crawlers dominated.

Which Industries Are AI Bots Targeting Most?

Retail keeps absorbing the most AI crawling activity globally. The targeting profile barely moved month-over-month.

Retail at 31.3% (January was 31.2%) makes sense when you think about it. Users asking ChatGPT "what's the best laptop under $1,000" or "compare Nike and Adidas running shoes" generate crawl requests against product pages, reviews, and pricing data. That's real bandwidth cost for e-commerce operators.

One shift I noticed: Travel and Tourism climbed from 3.8% to 4.1%, jumping past Professional Services (3.5% to 3.3%). Probably seasonal. Spring travel planning drives more AI queries toward booking sites and hotel reviews.

The industry-level breakdown gets more granular:

New entry this month: Adult Entertainment appeared at 2.6%, pushing out Marketing and Advertising (which held 2.4% in January). Computer Software at 13.8% is the second-largest target, and there's something recursive about it. AI companies are crawling software docs and technical content to train models that generate software. If you maintain API references or technical tutorials, AI crawlers are probably some of your most frequent visitors. That's exactly why picking the right AI web search API for your apps matters, since these crawlers are the infrastructure behind the search results your users actually see.

AI training crawlers aren't the only bots scanning the web at this scale, either. Technology detection platforms like Technologychecker.io crawl and fingerprint over 50 million domains using HTTP header analysis, JavaScript fingerprinting, DNS lookups, and headless browser rendering to identify 40,000+ technologies. Unlike AI training crawlers that take content for model weights, technology intelligence crawlers need to re-crawl frequently to track stack changes and new tech adoptions.

How Are Websites Fighting Back Against AI Crawlers?

After the robots.txt blocking wave plateaued in January, I expected the numbers to stay flat. They didn't. February showed a modest but clear uptick in blocking activity across almost every AI crawler.

Most Blocked AI Crawlers (% of Domains Referencing in robots.txt)

GPTBot is still the most commonly referenced AI crawler in robots.txt files at 5.45%, up from 5.24% in January. ClaudeBot saw the biggest jump among the top crawlers, rising from 4.33% to 4.62% (+0.29 pp). Google-Extended climbed from 4.06% to 4.36% (+0.30 pp), which tells me more site owners are finding Google's AI training opt-out mechanism.

Here's what bugs me about this data: meta-externalagent sits at only 3.26% despite being the second-largest AI crawler at 15.6% of traffic. That's the biggest gap between traffic share and blocking rate of any crawler on this list. Meta's bot generates nearly as much traffic as GPTBot and ClaudeBot combined, but far fewer domains block it. If you care about AI training crawlers hitting your site, check whether your robots.txt actually includes meta-externalagent.

Robots.txt Mentions Are Creeping Up Again

The flat-to-declining trend from January reversed. Nearly every AI crawler saw increased robots.txt mentions in February:

The +0.30 pp increase for both Google-Extended and Bytespider stands out. Google-Extended adoption suggests growing awareness of Google's opt-out mechanism. Bytespider keeps attracting blocks despite its modest 3.3% traffic share, likely because of ongoing concerns about Chinese AI data collection.

The blocking wave may be entering a second phase, driven by more media coverage of AI crawling practices and more site owners realizing their content feeds model training. But let's be real: more than 94% of domains still allow all AI crawlers unrestricted access. The blocking minority is growing, but it's still a minority.

What's Happening on Cloudflare Workers AI?

Workers AI, the platform that lets developers run AI models at the edge, had a new entry this month worth paying attention to.

Most Popular Models (by Account Share)

Meta dominates here too. Llama 3 8B Instruct powers 40.1% of Workers AI accounts. Add Llama 4 Scout (6.7%), M2M-100 (5.4%), and the AWQ variant (4.9%) and Meta models cover 57.1% of usage by account share. That's down from ~60% in January because a newcomer took some share.

The newcomer: GPT-OSS 120B at 1.6%. OpenAI's open-source 120-billion-parameter model debuted on the leaderboard this month, matching Whisper Large V3 Turbo's share on its first appearance. It's the largest model by parameter count on the platform. Developers are willing to run bigger models at the edge for better inference quality, which tells me something about where edge AI is heading. OpenAI's combined Workers AI footprint (Whisper 8.3% + GPT-OSS 120B 1.6% + Whisper Large V3 Turbo 1.6%) now reaches 11.5%, making OpenAI the second-largest model provider on the platform behind Meta.

Llama 4 Scout dipped from 7.7% to 6.7%. The initial adoption rush is normalizing. It's still growing in absolute terms but losing share to GPT-OSS 120B and other entries competing for developer attention.

Task Distribution

Mostly stable. Text generation held at 62.6% (down slightly from 63.0%), Text-to-Image ticked up to 19.7% (from 19.1%). One new category showed up: Text-to-Speech at 0.4%. Small number. But it's the first time voice synthesis appeared in the Workers AI task distribution, and edge deployment makes a lot of sense for latency-sensitive speech generation. I'll be watching whether this grows in the March data.

What Should Website Owners Do About February's Trends?

Here's what I'd actually prioritize based on this month's data:

Training crawlers are now the majority. If you've been letting all AI crawlers through because you assumed most of that traffic helped your search visibility, that assumption is wrong as of February 2026. Dedicated training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent) now account for more traffic than mixed-purpose bots. The content leaving your servers is more likely going into model weights than into search indexes.

Add meta-externalagent to your robots.txt. Meta's crawler is the #2 AI bot at 15.6% of traffic. Only 3.26% of domains block it. If you've already blocked GPTBot and ClaudeBot but skipped Meta's bot, you're leaving the door wide open to the fastest-growing training crawler on the web.

Try Google-Extended if you want a middle ground. Google-Extended adoption rose to 4.36% of domains (+0.30 pp month-over-month). It lets you opt out of Google's AI training while keeping your search rankings. If you want to reduce AI training access without hurting SEO, this is the simplest first step.

E-commerce operators: check your bandwidth costs. Retail and shopping sites absorb 31.3% of all AI bot traffic. Meta-ExternalAgent alone grew 31% this month, and the Training category hit 45.4%. If you haven't looked at how much bandwidth AI crawlers consume on your infrastructure, now's a good time.

Keep an eye on GPT-OSS 120B. OpenAI's open-source model showing up on Workers AI tells you something about where the open-source LLM space is going. A 120B-parameter model running at the edge opens up inference quality levels that weren't practical outside centralized API calls before.

How WebSearchAPI.ai Fits Into the AI Crawler Ecosystem

Every AI crawler in this report exists because AI companies need fresh, structured web data to power their models and search products. At WebSearchAPI.ai, we sit on the other side of that equation. We give developers and AI agents a clean, fast way to access real-time web data without running their own crawlers.

Why this matters right now:

• Training crawlers now outpace mixed-purpose bots. The web's data is being consumed for model weights more than for search indexing. Instead of building your own crawling infrastructure, WebSearchAPI.ai gives you instant access to structured search results, content extraction, and real-time web intelligence through a single API call.
• AI search crawling grew 15.5% this month. Whether you're building a RAG pipeline, an AI agent, or a search-augmented chatbot, WebSearchAPI.ai handles the retrieval layer so you can focus on your application logic.
• Sub-second latency, 99.9% uptime, and structured responses mean your AI apps get the data they need without the infrastructure overhead of managing crawler fleets.

The volume and complexity of AI web crawling is accelerating, and the balance is shifting toward dedicated training. WebSearchAPI.ai is built for developers who want that web intelligence without becoming a crawling operation. Learn more about what a web search API can do for your stack.

Frequently Asked Questions

What is an AI crawler?

An AI crawler (also called an AI bot or AI spider) is an automated program that visits websites to collect content for training AI models or powering AI-powered search features. Unlike traditional search engine crawlers that index pages for search results, AI crawlers like GPTBot, ClaudeBot, and Meta-ExternalAgent specifically collect data to train large language models (LLMs). Some crawlers like Googlebot serve both purposes, indexing for search and collecting training data at the same time. You can identify AI crawlers by their user-agent strings in your server logs or through tools like Cloudflare Radar.

How often is this AI crawler report updated?

Monthly. Each edition covers a rolling 30-day window and includes month-over-month comparisons so you can track trends over time. Bookmark this page or check back at the beginning of each month for the latest data on AI crawler traffic patterns, market share shifts, and robots.txt blocking trends.

Can I block AI crawlers from my website?

Yes. Add disallow rules to your robots.txt file for specific AI crawler user agents. For example, adding User-agent: GPTBot followed by Disallow: / will request that OpenAI's crawler stop visiting your site. Robots.txt is voluntary though, crawlers aren't technically required to obey it. As of February 2026, only about 5.5% of domains block GPTBot and 4.6% block ClaudeBot. The vast majority of the web remains open to AI crawling. Some CDN providers like Cloudflare also offer dashboard-level controls to block or rate-limit AI bots.

What is the difference between AI training crawlers and AI search crawlers?

AI training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent) collect web content to build and improve AI models. They scrape large volumes from many sites. AI search crawlers (like OAI-SearchBot) fetch specific pages in real time when someone performs a search through an AI tool like ChatGPT. Training crawlers take your content to make the model smarter. Search crawlers fetch your content to answer a specific question and may send traffic back to your site. As of February 2026, training crawling (45.4%) has overtaken mixed-purpose crawling (43.9%) for the first time, while search crawling continues to grow at 8.2%.

Will blocking AI crawlers affect my SEO or search rankings?

Blocking dedicated AI training crawlers like GPTBot, ClaudeBot, or Meta-ExternalAgent won't affect your rankings in Google, Bing, or other traditional search engines. Those crawlers are separate from search indexing bots. But blocking Googlebot will remove your site from Google Search entirely because Google uses the same crawler for both search indexing and AI training. Google-Extended is the middle ground. Blocking it opts you out of AI training while keeping your search presence. Adoption is growing: 4.36% of domains now use this option, up from 4.06% in January.

How does Cloudflare track AI crawler traffic?

Cloudflare's global network spans 330+ cities in 125+ countries and processes over 81 million HTTP requests per second. Through its Radar platform, Cloudflare identifies and classifies AI bot traffic by analyzing user-agent strings, request patterns, and behavioral signatures across all sites on its network. The data in this report comes from Cloudflare Radar's AI Insights endpoints, which aggregate these signals into share-of-traffic percentages by bot, crawl purpose, industry, and region.

Which AI crawler is growing the fastest in 2026?

Meta-ExternalAgent. For a second consecutive month, it had the largest absolute market share gain, jumping from 11.9% to 15.6% (+30.8% relative increase). Over two months, it roughly doubled from 8.5% to 15.6%. In relative terms, Applebot (+17.0%), OAI-SearchBot (+16.8%), and Amazonbot (+15.0%) all showed strong growth from smaller bases.

What percentage of web traffic comes from AI bots?

The percentages in this report are share of identified AI bot requests, not share of total web traffic. Cloudflare Radar tracks AI crawler activity relative to other AI bots, giving a competitive view. The actual percentage of total web traffic from AI bots varies by site, but AI crawlers now account for a growing share of overall internet traffic, particularly for content-heavy sites in retail, technology, and media.

How can I monitor AI crawler activity on my own website?

Check your server access logs for known AI bot user-agent strings (GPTBot, ClaudeBot, meta-externalagent, Bytespider, Amazonbot, etc.). Most web analytics platforms filter out bot traffic by default, so log-level analysis gives the most accurate picture. Cloudflare users can view AI bot activity directly in their dashboard. For a structured approach, consider using a web search API to see how your content appears in AI-powered search results and confirm your most important pages are accessible.

Cloudflare Radar Data Source & Methodology

Where this data comes from and what it can't tell you matters. Here's how Cloudflare Radar collects, classifies, and aggregates the AI crawler data I used in this report.

Network Scale

This scale is what makes the data useful. Two primary sources feed Cloudflare Radar:

1. Cloudflare's global network (real-time traffic data from HTTP requests flowing through their infrastructure)
2. 1.1.1.1 public DNS resolver (aggregated and anonymized DNS query data)

For routing data, Cloudflare also uses RIPE RIS data from RIPE NCC (BGP route collectors).

How AI Bots Are Identified

Cloudflare uses a layered detection system:

1. User-agent string matching identifies bots that transparently announce themselves (GPTBot, ClaudeBot, etc.)
2. Verified Bot Directory is a manual approval process requiring bots to maintain public robots.txt commitments, use verifiable IPs, and honor crawl-delay settings
3. Machine learning is a supervised ML system that assigns a Bot Score (1-99)
4. Heuristics are tailored rulesets for AI bot classification
5. Behavioral analysis does pattern recognition from request sequences
6. AI Labyrinth honeypot plants hidden links to AI-generated decoy pages; bots that follow them get flagged because human visitors never see those links
7. ai.robots.txt list is used as the basis for which AI bots to track

💡 Expert Insight: The layered approach matters because not all AI crawlers identify themselves honestly. User-agent matching catches transparent bots like GPTBot and ClaudeBot. Behavioral analysis and honeypots catch crawlers that disguise themselves as regular browsers.

How Crawl Purpose Is Classified

Bots fall into these categories:

• Training: dedicated training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent)
• Search: AI search bots (OAI-SearchBot)
• User Action: bots fetching pages for real-time user queries (ChatGPT-User)
• Mixed Purpose: bots serving dual roles like search indexing + AI training (Googlebot, Bingbot)
• Undeclared: purpose not identifiable

Data Aggregation Methods

• 7-day trailing average smooths daily fluctuations
• IPv4 addresses aggregated into /20 prefixes for visualization
• HTML traffic is classified into human, AI bot, and non-AI bot categories
• Normalization: data is expressed as percentage of total requests, not absolute counts
• Methodologies are unchanged year-over-year for valid comparisons
• API data is available under CC BY-NC 4.0 license

Caveats & Limitations

⚠️ Warning: Keep these limitations in mind when interpreting the data in this report:

• Countries with insufficient data volume are excluded from trend reporting
• Some metrics are available only at worldwide level, not per-country
• Mobile device categorization relies on User-Agent headers (accuracy limitations)
• Speed test data excludes locations with fewer than 100 tests per week
• The "location" filter corresponds to the billing country of the Cloudflare customer whose site received the traffic, not where the crawler is physically located
• Cloudflare sees traffic only to sites behind its network, not the entire internet, so the data is representative but not exhaustive

Report Parameters

This edition uses data from Cloudflare Radar's AI Insights endpoint (get_ai_data) and robots.txt analysis endpoint (get_robots_txt_data). The data covers February 11 through March 13, 2026, with month-over-month comparisons using a 30-day control window (January 12 - February 11, 2026).

I queried bot traffic breakdowns by user agent, crawl purpose, industry, and vertical. Workers AI data covers model and task distribution by account share. Robots.txt analysis covers domain-level crawler policies for AI user agents. All percentages are share of identified AI bot requests (for crawling data) or share of accounts (for Workers AI data), not share of total web traffic.

Data source: Cloudflare Radar AI Insights, get_ai_data and get_robots_txt_data endpoints (radar.cloudflare.com). Last updated: March 13, 2026.

About the Author: I'm James Bennett, Lead Engineer at WebSearchAPI.ai, where I architect the core retrieval engine enabling LLMs and AI agents to access real-time, structured web data with over 99.9% uptime and sub-second query latency. With a background in distributed systems and search technologies, I've reduced AI hallucination rates by 45% through advanced ranking and content extraction pipelines for RAG systems. My expertise includes AI infrastructure, search technologies, large-scale data integration, and API architecture for real-time AI applications.

Credentials: B.Sc. Computer Science (University of Cambridge), M.Sc. Artificial Intelligence Systems (Imperial College London), Google Cloud Certified Professional Cloud Architect, AWS Certified Solutions Architect, Microsoft Azure AI Engineer, Certified Kubernetes Administrator, TensorFlow Developer Certificate.