Entry #8 · Mar 24, 2026

Source:

Published:

Keywords: ClaudeBot, Claude-SearchBot, GEO pages, Q&A pages, sitemapindex, robots.txt

Document ID: https://rozz.site/insights/claudebot-cliffhanger-resolved.html#split1

And now we have it: ClaudeBot just made 958 requests.

Last week we wrote: “ClaudeBot read the map and closed the book.” This week ClaudeBot opened it. ClaudeBot made 958 requests. ClaudeBot made 503 GEO pages and 162 Q&A pages. On March 20 alone, ClaudeBot made 577 requests to rozz.genymotion.com. Every major AI crawler completed a deep indexing event on the AI site.

ClaudeBot had done a full crawl once before, back in December. ClaudeBot hit 1,267 pages when it first discovered the site. ClaudeBot then went quiet. For three months ClaudeBot ran a monitoring loop. ClaudeBot checked robots.txt and sitemap.xml every 1–2 hours. ClaudeBot occasionally read a topic page or a handful of content pages. ClaudeBot never committed to a real crawl. In article 5, the team described the behavior as “reading the map, not the pages.”

In article 7, last week the team reported 123 requests with 0 Q&A pages and 1 GEO page. ClaudeBot appeared stuck. Six hours after the team deployed a per-topic sitemap, ClaudeBot came back.

Key findings

ClaudeBot requests and crawl scope

Other bots

> Warning: 0 requests

The data

All LLM bots (Mar 17–24, 2026)

| Bot | Requests | Category | Change vs. prior week | | --- | --- | --- | --- | | ChatGPT-User | 1,154 | Citation | Stable (~1,200) | | ClaudeBot | 958 | Training | 8x (was 123) | | OpenAI SearchBot | 145 | index | Down from 328 | | ByteSpider | 91 | Training | Up from 69 | | PerplexityBot | 58 | index | Down from 511 | | OpenAI GPTBot | 37 | Training | Down from 171 | | Meta AI | 3 | Training | Down from 52 |

ClaudeBot content breakdown

| Content type | Requests | % | | --- | --- | --- | | GEO Pages | 503 | 52.5% | | Q&A Pages | 162 | 16.9% | | Sitemap | 108 | 11.3% | | Robots.txt | 91 | 9.5% | | Other (topics) | 91 | 9.5% | | Homepage | 3 | 0.3% |

Compare this to last week’s ClaudeBot breakdown: 64% robots.txt/sitemap/topics, 0.8% GEO pages, 0% Q&A pages. The shift from structural monitoring to content ingestion happened in a single week.

ClaudeBot daily breakdown

| Date | ClaudeBot requests | | --- | --- | | Mar 17 | 56 | | Mar 18 | 13 | | Mar 19 | 25 | | Mar 20 | 577 | | Mar 21 | 170 | | Mar 22 | 74 | | Mar 23 | 28 | | Mar 24 | 15 |

March 20 is ClaudeBot’s version of the events we’ve now seen from every major crawler. GPTBot had January 7 (547 requests). PerplexityBot had March 10–15 (511 requests across several sessions). ClaudeBot had March 20 (577 requests in one day).

March 20, hour by hour

Monitoring loop Content crawl Sitemapindex deployed (15:57 UTC)

What we changed (and when)

The ClaudeBot surge didn’t come from nothing. The ClaudeBot surge followed two rounds of infrastructure changes. The infrastructure changes included one change that likely triggered the surge.

March 9: four content and navigation changes

These are the same changes that preceded PerplexityBot’s activation in article 7.

1. Topic restructuring: 15 topics to 25. The old topic system had a mega-topic problem. “Genymotion Testing Suite” contained 166 of 178 Q&As (93%). The brand keyword “Genymotion” appeared on 56% of pages. The clustering algorithm kept grouping everything around the brand keyword. The team filtered brand keywords appearing on more than 25% of pages. The team increased the topic count from a hardcoded 15 to a dynamic formula based on content volume. The team re-ran the pipeline.

Result: the top topic dropped from 166 Q&As (93%) to 87 (49%). Specific user-intent topics emerged. “Apple Silicon Macs” had 65 Q&As. “CI/CD Tools” had 36 Q&As. “Licensing & Certificates” had 17 Q&As. Topics matched how people actually ask questions.

2. Index page revamp. The old index listed API endpoints and content counts. The new index opens with a product description. The new index then shows a topic directory with inline summaries for each of the 25 topics. The index provides enough context for a crawler to navigate to relevant content. The index also provides enough context for a crawler to answer a query directly from the index.

3. Featured Q&A reranking. The old index.html showed 6 “featured” Q&As in arbitrary database order. The first featured question was “How many free virtual Android devices does Genymotion offer?” The team replaced the arbitrary order with retrieval-count-based ranking. The Q&As that ChatGPT-User fetches most often appear first.

PerplexityBot activated one day after these changes. ClaudeBot didn’t activate one day after these changes. For 11 days ClaudeBot continued its monitoring loop: robots.txt + sitemap every 1–2 hours, and no content pages.

March 20: per-topic sitemaps

On March 20, the team deployed a structural change to how the sitemap works.

The old sitemap was a single monolithic sitemap.xml listing ~700 URLs. The old sitemap had no semantic structure. The old sitemap had no topic grouping. The old sitemap provided no signal about which content clusters had been updated. ClaudeBot had been reading the old sitemap 15 times per week for months. ClaudeBot never committed to a content crawl after reading the old sitemap. The old sitemap told ClaudeBot “here are 700 pages.” The old sitemap gave ClaudeBot no way to assess which pages were fresh. The old sitemap also gave ClaudeBot no way to assess which topics were worth prioritizing.

The new sitemap is a <sitemapindex> pointing to per-topic child sitemaps. Each child sitemap lists only the pages in that topic. Each child sitemap has its own <lastmod> date based on the newest page in the cluster. Topic names in the URLs provide semantic signal. A crawler can see that sitemaps/pricing-billing.xml was updated yesterday. A crawler can see this without fetching the contents.

The team also simplified robots.txt. The team reduced robots.txt from ~140 lines of redundant per-bot sections down to ~12 lines. The simplified robots.txt includes a single User-agent: * rule. The team moved the llms.txt and llms-full.txt pointers. The pointers moved to the top of the file. Previously the pointers were buried at line 90+ after 15 individual bot sections.

The team made these changes after observing ClaudeBot on the AI site. ClaudeBot checks robots.txt and sitemap.xml every 1–2 hours. ClaudeBot is the only crawler the team has observed that systematically reads the site’s structural layer. A sitemapindex organized by topic is a table of contents. Table-of-contents reading is what ClaudeBot does.

The timeline

| Date | Event | | --- | --- | | Mar 9 | Content and navigation changes deployed (index, topics, featured Q&As) | | Mar 10 | PerplexityBot content burst (50+ pages in 20 seconds) | | Mar 14–15 | PerplexityBot Q&A deep dive + topic sweeps | | Mar 17 | Article 7 published: “PerplexityBot wakes up, Claude still waiting” | | Mar 20, 15:57 UTC | Per-topic sitemaps + simplified robots.txt deployed | | Mar 20, 22:00 UTC | ClaudeBot mass crawl begins: 523 requests in two hours | | Mar 21 | ClaudeBot follow-up: 170 requests |

We can’t prove that the per-topic sitemap caused the crawl. ClaudeBot reads structure. ClaudeBot checked the old monolithic sitemap 15 times a week for months without crawling. On the day the team replaced the sitemap with a topic-organized sitemapindex, six hours after deployment, ClaudeBot ran its largest content crawl since December.

Every major crawler has now had its moment

| Bot | Deep indexing event | Requests | What preceded it | | --- | --- | --- | --- | | GPTBot | January 7 | 547 in one day | Sitemap discovery | | PerplexityBot | March 10–15 | 511 in one week | Index page revamp (Mar 9) | | ClaudeBot | March 20–21 | 747 in two days | Per-topic sitemapindex (Mar 20) |

Three bots. Three mass crawl events. Three different structural signals. GPTBot found the site through its sitemap and mass-crawled content. PerplexityBot responded to a redesigned index page with a topic directory. ClaudeBot responded to a sitemapindex organized by topic with per-cluster freshness dates.

ClaudeBot had crawled the site once before, in December (1,267 pages), when ClaudeBot first discovered the domain. ClaudeBot then went into monitoring mode for three months. The March 20 event brought ClaudeBot back.

The open question: where is Claude-SearchBot?

ClaudeBot made 958 requests. Claude-SearchBot made 0 requests.

This has been true for the entire observation period. Across 60 days of logs, Claude-SearchBot never visited the AI site. Claude-SearchBot never visited the AI site. Claude-SearchBot visited the AI site not once.

For context, OpenAI splits its crawler infrastructure into three bots with distinct roles:

Anthropic has ClaudeBot documented as the training/indexing crawler. Claude-SearchBot is documented as the bot that builds the retrieval index for Claude’s web search feature. The team sees ClaudeBot. The team does not see Claude-SearchBot.

Three possible explanations:

1. ClaudeBot handles both training and search indexing. Unlike OpenAI’s split architecture, Anthropic may route all crawl activity through a single bot. If Anthropic routes crawl activity through a single bot, the March 20 mass crawl feeds both Claude’s training data and its search retrieval index. The team would have no way to distinguish from log data alone.

2. Claude’s web search uses a partner’s index. If Claude’s search feature relies on Bing’s index or another third-party search infrastructure rather than a proprietary crawl, then Claude-SearchBot may not need to visit sites directly. ClaudeBot would handle training data only.

3. Claude-SearchBot hasn’t activated for this domain yet. The training crawl may be a prerequisite. ClaudeBot evaluates the content. Claude-SearchBot follows later once the domain is flagged as worth indexing for search.

The test is straightforward. If Claude’s citation rate for Genymotion improves in the next 2–3 weeks without any Claude-SearchBot traffic, explanation 1 is likely correct: ClaudeBot serves both purposes. If Claude’s citation rate for Genymotion does not improve, the missing Claude-SearchBot may be the bottleneck. The team will know by mid-April.

ChatGPT: still steady

ChatGPT-User made 1,154 requests this week. The plateau continues.

| Period | ChatGPT-User | Change | | --- | --- | --- | | January (30 days) | 42 | Baseline | | Feb 2–9 | 345 | 8x | | Feb 10–17 | 1,077 | 3x | | Feb 17–24 | 1,329 | +23% | | Mar 3–10 | ~1,200 | Stable | | Mar 10–17 | 1,176 | Stable | | Mar 17–24 | 1,154 | Stable |

Four consecutive weeks at ~1,200. ChatGPT is a reliable citation channel producing roughly 44 content sessions per day. ChatGPT sessions have 40% involving multi-turn follow-up questions. The growth story has moved to other platforms.

PerplexityBot: post-indexing quiet

PerplexityBot dropped from 511 to 58 this week. 45 of those 58 requests were topic pages and robots.txt. Only 2 Q&A pages.

This is normal post-indexing behavior. GPTBot did the same thing after January 7. GPTBot did a mass crawl followed by weeks of 1–2 requests per day. PerplexityBot indexed the content. PerplexityBot is now monitoring for changes. The question is whether Perplexity citations start appearing in the next two weeks.

Where we are

Four platforms, four stages:

ChatGPT: steady state.

ChatGPT has an 83% citation rate (last tested March 2).

ChatGPT has ~1,200 requests/week.

The AI site approach is proven here.

Perplexity: indexed, waiting for citations.

Perplexity completed a deep indexing event March 10–15.

Citations are expected early April.

Claude: just indexed, waiting for everything.

Claude completed its first real content crawl March 20.

Claude crawled 503 GEO pages and 162 Q&A pages.

Claude has no Claude-SearchBot traffic.

The question is whether this feeds the citation pipeline or only training data.

Answer is expected mid-April.

Gemini: different game entirely.

Gemini doesn’t crawl the AI site.

Gemini answers from training data and Google Search grounding.

Gemini is not part of this story.

Three months ago the team had one platform crawling. Now every major AI crawler completed a deep indexing event on the same infrastructure. The infrastructure has the same content. The infrastructure has the same Schema.org markup. The infrastructure has the same topic taxonomy. The bots arrive on their own schedules. The content was ready when the bots showed up.

Get This for Your Site

ROZZ builds this infrastructure automatically. The AI site includes Q&A pages from your chatbot. The AI site includes Schema.org markup on every page. The AI site includes a topic taxonomy that every major AI crawler has now indexed.

$997 /month | ChatGPT citations at 1,200/week. Claude and Perplexity indexing complete.

Data source and author

Data source: CloudFront access logs for rozz.genymotion.com, March 17–24, 2026. Bot classification based on User-Agent strings. ClaudeBot daily breakdown from timestamped request logs.

Author: Adrien Schmidt, CEO, ROZZ

Serial tech entrepreneur with 10+ years experience building AI systems including Aristotle (conversational AI analytics) and products for eBay and Cartier. Previously founded Squid Solutions and built AI products like Aristotle, the conversational big data analytics chatbot, and an AR jewelry try-on device for Cartier.

March 24, 2026 | Data period: Mar 17–24, 2026