And now we have it: ClaudeBot just made 958 requests.

Entry #8 — And now we have it: ClaudeBot just made 958 requests.

Entry details

• Date: March 24, 2026
• Data period: March 17–24, 2026
• Data source: CloudFront access logs for rozz.genymotion.com; Bot classification based on User-Agent strings
• Author: Adrien Schmidt, CEO, ROZZ
• Data period in view: March 17–24, 2026

Key findings

• ClaudeBot: 958 requests (Mar 17–24)
• ClaudeBot requests are up 8x from 123 in the prior week
• ClaudeBot crawled 503 GEO pages and 162 Q&A pages (69% content)
• March 20: 577 ClaudeBot requests in a single day
• ChatGPT-User: 1,154 requests, stable
• PerplexityBot: 58 requests, down from 511 (post-indexing maintenance)
• Claude-SearchBot: still 0 requests. Zero. The open question remains.

The data

All LLM bots (Mar 17–24, 2026)

| Bot | Requests | Category | Change vs. prior week | |---|---:|---|---| | ChatGPT-User | 1,154 | Citation | Stable (~1,200) | | ClaudeBot | 958 | Training | 8x (was 123) | | OpenAI SearchBot | 145 | index | Down from 328 | | ByteSpider | 91 | Training | Up from 69 | | PerplexityBot | 58 | index | Down from 511 | | OpenAI GPTBot | 37 | Training | Down from 171 | | Meta AI | 3 | Training | Down from 52 |

ClaudeBot content breakdown

| Content type | Requests | % | |---|---:|---:| | GEO Pages | 503 | 52.5% | | Q&A Pages | 162 | 16.9% | | Sitemap | 108 | 11.3% | | Robots.txt | 91 | 9.5% | | Other (topics) | 91 | 9.5% | | Homepage | 3 | 0.3% |

Compare this to last week’s ClaudeBot breakdown: 64% robots.txt/sitemap/topics, 0.8% GEO pages, 0% Q&A pages. The shift from structural monitoring to content ingestion happened in a single week.

ClaudeBot daily breakdown

| Date | ClaudeBot requests |

|---|---:|

| Mar 17 | 56 |

| Mar 18 | 13 |

| Mar 19 | 25 |

| Mar 20 | 577 |

| Mar 21 | 170 |

| Mar 22 | 74 |

| Mar 23 | 28 |

| Mar 24 | 15 |

March 20 is ClaudeBot’s version of the events we’ve now seen from every major crawler. GPTBot had January 7 (547 requests). PerplexityBot had March 10–15 (511 requests across several sessions). ClaudeBot had March 20 (577 requests in one day).

March 20, hour by hour

Monitoring loop Content crawl Sitemapindex deployed (15:57 UTC)

What we changed (and when) The ClaudeBot surge didn’t come from nothing. It followed two rounds of infrastructure changes, including one that likely triggered it.

• March 9: four content and navigation changes
• These are the same changes that preceded PerplexityBot’s activation in article 7.
• 1. Topic restructuring: 15 topics to 25.
• The old topic system had a mega-topic problem. “Genymotion Testing Suite” contained 166 of 178 Q&As (93%).
• The brand keyword “Genymotion” appeared on 56% of pages, and the clustering algorithm kept grouping everything around it.
• We filtered brand keywords appearing on more than 25% of pages, increased the topic count from a hardcoded 15 to a dynamic formula based on content volume, and re-ran the pipeline.
• Result: the top topic dropped from 166 Q&As (93%) to 87 (49%). Specific user-intent topics emerged: “Apple Silicon Macs” (65 Q&As), “CI/CD Tools” (36 Q&As), “Licensing & Certificates” (17 Q&As). Topics that match how people actually ask questions.
• 2. Index page revamp.
• The old index listed API endpoints and content counts.
• The new index opens with a product description, then a topic directory with inline summaries for each of the 25 topics.
• Enough context for a crawler to navigate to relevant content or answer a query directly from the index.
• 3. Featured Q&A reranking.
• The old index.html showed 6 “featured” Q&As in arbitrary database order.
• The first featured question was “How many free virtual Android devices does Genymotion offer?”
• We replaced this with retrieval-count-based ranking: the Q&As that ChatGPT-User fetches most often appear first.
• PerplexityBot activated one day after these changes. ClaudeBot didn’t. For 11 days, ClaudeBot continued its monitoring loop: robots.txt + sitemap every 1–2 hours, no content pages.
• March 20: per-topic sitemaps
• On March 20, we deployed a structural change to how the sitemap works.
• The old sitemap was a single monolithic sitemap.xml listing ~700 URLs. No semantic structure, no topic grouping, no signal about which content clusters had been updated.
• ClaudeBot had been reading this sitemap 15 times per week for months without ever committing to a content crawl.
• The new sitemap is a <sitemapindex> pointing to per-topic child sitemaps.
• Each child sitemap (sitemaps/pricing-and-refunds.xml, sitemaps/android-development-stack.xml, etc.) lists only the pages in that topic, with its own <lastmod> date based on the newest page in the cluster.
• Topic names in the URLs provide semantic signal: a crawler can see that sitemaps/pricing-billing.xml was updated yesterday without fetching the contents.
• We also simplified robots.txt from ~140 lines of redundant per-bot sections down to ~12 lines with a single User-agent: * rule.
• The llms.txt and llms-full.txt pointers, previously buried at line 90+ after 15 individual bot sections, moved to the top of the file.
• We made these changes after observing the behavior of ClaudeBot on the AI site.
• ClaudeBot checks robots.txt and sitemap.xml every 1–2 hours. It’s the only crawler we’ve observed that systematically reads the site’s structural layer. A sitemapindex organized by topic is a table of contents, and table-of-contents reading is what ClaudeBot does.

The timeline

| Date | Event | |---|---| | Mar 9 | Content and navigation changes deployed (index, topics, featured Q&As) | | Mar 10 | PerplexityBot content burst (50+ pages in 20 seconds) | | Mar 14–15 | PerplexityBot Q&A deep dive + topic sweeps | | Mar 17 | Article 7 published: “PerplexityBot wakes up, Claude still waiting” | | Mar 20, 15:57 UTC | Per-topic sitemaps + simplified robots.txt deployed | | Mar 20, 22:00 UTC | ClaudeBot mass crawl begins: 523 requests in two hours | | Mar 21 | ClaudeBot follow-up: 170 requests |

We can’t prove the per-topic sitemap caused the crawl. But ClaudeBot is the bot that reads structure. It had been checking the old monolithic sitemap 15 times a week for months without crawling. On the day we replaced it with a topic-organized sitemapindex, six hours after deployment, ClaudeBot ran its largest content crawl since December.

The open question: where is Claude-SearchBot? ClaudeBot made 958 requests. Claude-SearchBot made 0.

This has been true for the entire observation period. Across 60 days of logs, Claude-SearchBot has never visited the AI site. Not once.

For context, OpenAI splits its crawler infrastructure into three bots with distinct roles:

• GPTBot: training data collection
• OAI-SearchBot: builds the retrieval index for ChatGPT’s web search
• ChatGPT-User: real-time retrieval during live conversations

Anthropic has ClaudeBot documented as the training/indexing crawler. Claude-SearchBot is documented as the bot that builds the retrieval index for Claude’s web search feature. We see ClaudeBot. We don’t see Claude-SearchBot.

Three possible explanations:

1. ClaudeBot handles both training and indexing.

• Unlike OpenAI’s split architecture, Anthropic may route all crawl activity through a single bot.
• If so, the March 20 mass crawl feeds both Claude’s training data and its search retrieval index.
• We’d have no way to distinguish from log data alone.

2. Claude’s web uses a partner’s index.

• If Claude’s search feature relies on Bing’s index or another third-party search infrastructure rather than a proprietary crawl, then Claude-SearchBot may not need to visit sites directly.
• ClaudeBot would handle training data only.

3. Claude-SearchBot hasn’t activated for this domain yet.

• The training crawl may be a prerequisite.
• ClaudeBot evaluates the content, and Claude-SearchBot follows later once the domain is flagged as worth indexing for search.

The test is straightforward. If Claude’s citation rate for Genymotion improves in the next 2–3 weeks without any Claude-SearchBot traffic, explanation 1 is likely correct: ClaudeBot serves both purposes. If it doesn’t improve, the missing Claude-SearchBot may be the bottleneck. We’ll know by mid-April.

ChatGPT: still steady

• Period: January (30 days) | ChatGPT-User: 42 | Change: Baseline
• Feb 2–9 | 345 | Change: 8x
• Feb 10–17 | 1,077 | Change: 3x
• Feb 17–24 | 1,329 | Change: +23%
• Mar 3–10 | ~1,200 | Change: Stable
• Mar 10–17 | 1,176 | Change: Stable
• Mar 17–24 | 1,154 | Change: Stable

Four consecutive weeks at ~1,200. ChatGPT is a reliable citation channel producing roughly 44 content sessions per day, with 40% involving multi-turn follow-up questions. The growth story has moved to other platforms.

PerplexityBot: post-indexing quiet

• PerplexityBot dropped from 511 to 58 this week.
• 45 of those 58 requests were topic pages and robots.txt.
• Only 2 Q&A pages.

This is normal post-indexing behavior. GPTBot did the same thing after January 7: a mass crawl followed by weeks of 1–2 requests per day. PerplexityBot indexed the content. Now it’s monitoring for changes. The question is whether Perplexity citations start appearing in the next two weeks.

Where we are

Four platforms, four stages:

• ChatGPT: steady state. 83% citation rate (last tested March 2). ~1,200 requests/week. The AI site approach is proven here.
• Perplexity: indexed, waiting for citations. Deep indexing event completed March 10–15. Citations expected early April.
• Claude: just indexed, waiting for everything. First real content crawl March 20. 503 GEO pages, 162 Q&A pages. No Claude-SearchBot traffic. Whether this feeds the citation pipeline or only training data is the open question. Answer expected mid-April.
• Gemini: different game entirely. Doesn’t crawl the AI site. Answers from training data and Google grounding. Not part of this story.

Three months ago we had one platform crawling. Now every major AI crawler has completed a deep indexing event on the same infrastructure. Same content, same Schema.org markup, same topic taxonomy. The bots arrive on their own schedules. The content was ready when they showed up.

Data source and author

• Data source: CloudFront access logs for rozz.genymotion.com, March 17–24, 2026. Bot classification based on User-Agent strings.
• ClaudeBot daily breakdown from timestamped request logs.
• Author: Adrien Schmidt, CEO, ROZZ
• March 24, 2026 | Data period: Mar 17–24, 2026
• rozz@rozz.site | © 2026 ROZZ. All rights reserved.