​

Crawler Analytics

Before AI platforms can recommend you, their bots need to visit your website and understand your content. Crawler Analytics shows you exactly which AI bots are visiting, what pages they look at, and how often they come back.

​

Two Very Different Kinds of AI Visit

Most analytics lump every AI bot together. The platform does not, because the two kinds mean opposite things for your business.

​

User-initiated visits: a real person is asking about you right now

These fetches happen because a human asked an AI assistant a question and the assistant went to read your page to answer it. They are real-time demand, so they are weighted 3x in your analytics.

​

Training and index crawlers: a machine is collecting content

These bots are building or refreshing a model or an index. Nobody is waiting on an answer. They still matter, because a model that never reads your site cannot recommend it, but they are not demand.

​

Humans arriving from an AI answer

A third class: a normal browser carrying a referrer from an AI surface. That is a real visitor who clicked a citation in an AI answer and landed on your site.

​

What You Learn

​

Which Pages Bots Visit

See the exact URLs that AI bots request. This tells you what content AI platforms consider important on your site. Common patterns include:

• Homepage and about pages: Bots gathering general brand information
• Room or experience pages: Bots collecting specific product details
• Blog and guide content: Bots indexing your expertise on local topics
• FAQ and help pages: Bots looking for structured question-and-answer content

​

Visit Patterns

Understand how bots navigate your site:

• Do they follow internal links or arrive at specific pages directly?
• Do they visit your site in response to specific search queries?
• Are certain sections of your site getting more bot attention than others?

​

Frequency and Trends

Track how often each bot visits over time:

• Increasing visits often mean AI platforms are finding your content more relevant
• Decreasing visits may signal technical issues (slow load times, blocked crawlers) or content that is becoming stale
• Sudden spikes can indicate your content was recently referenced in AI responses, triggering follow-up crawls

​

How to Use This Data

​

Prioritize High-Traffic Bot Pages

Pages that AI bots visit frequently are your strongest assets. Make sure these pages:

• Have accurate, up-to-date information
• Include structured data (schema markup)
• Load quickly and are free of technical errors
• Contain the kind of specific, factual content that AI models prefer

​

Fix Gaps in Bot Coverage

If important pages on your site are never visited by AI bots, investigate why:

• Check that your robots.txt is not blocking AI crawlers
• Ensure these pages are linked from your main navigation
• Add internal links from pages that bots do visit
• Improve page load speed, as bots have crawl budgets and skip slow pages

​

Monitor Changes After Optimization

When you update content based on the platform’s recommendations, Crawler Analytics lets you see whether AI bots return to re-crawl the updated pages. This is an early signal that your changes are being picked up.

​

Connection to Inferred Prompts

Crawler Analytics feeds directly into the Inferred Search Prompts feature. By analyzing which pages bots visit and when, we can reverse-engineer what questions users are asking AI platforms that lead bots to your content.