Crawler Trends
最終更新日：2026-09-10 12:11:39
Reading time: About 4 minutes
Prerequisites: Trends Overview is available only in the Advanced edition. If you are on the Basic edition, upgrade to Advanced to use this feature.
Navigation: AI Crawler Control > Trends Overview
Feature Overview
Trends Overview provides global visual analysis of AI crawler requests, helping you gain comprehensive insight into AI crawler access patterns across dimensions such as time, hostname, crawler category, and action result.
With Trends Overview, you can quickly identify:
- The overall volume of AI crawler requests and how it is trending
- The proportion of requests from each AI crawler category
- Which AI crawlers are the most active
- Which hostnames are crawled most frequently
- Which business paths are the primary targets of AI crawlers
Typical Analysis Scenarios
Understanding the Overall Scale of AI Crawler Traffic
Use the AI Crawler Request Overview and AI Crawler Request Trends panels to assess the overall scale of AI crawler traffic and how it is changing.
- The AI Crawler Request Overview displays four key metrics as stat cards:
| Metric | Description |
|---|---|
| Total Requests | The total number of AI crawler requests matching the filter conditions. |
| Logged Requests | The number of AI crawler requests matching the filter conditions for which the action is Log. |
| Skipped Requests | The number of AI crawler requests matching the filter conditions for which the action is Skip. |
| Blocked Requests | The number of AI crawler requests matching the filter conditions for which the action is Deny. |
- Each stat card also shows the period-over-period (PoP) change rate, reflecting the trend compared with the previous period.
- The AI Crawler Request Trends chart displays the above four metrics over time as a stacked area line chart.
Analysis tip: If the PoP change rate keeps rising and the trend chart shows growth, AI crawler traffic to your website is increasing. We recommend further analyzing the crawler sources and evaluating whether additional control is needed.
Identifying the Primary Crawler Types
Use the Crawler Category Distribution chart to quickly determine which crawler category accounts for the largest share of traffic.
- If AI Data Scraper accounts for a disproportionately high share, it may indicate that a large amount of content is being collected for model training data. We recommend paying closer attention to crawlers in this category in Crawler Control, or setting them to Deny.
- If AI Search Crawler accounts for a high share and you want your content to be indexed by AI search engines, consider setting the action to Skip.
The legend on the right shows the name, request count, and percentage for each category. Clicking a segment updates the Crawler Category filter and refreshes the page data.
Locating the Most Active Crawlers
Check the Crawler Activity ranking. If one crawler’s request volume far exceeds the others, evaluate closely whether its access pattern is legitimate. Combine this with the time filter to observe its request trend — a sudden short-term spike may indicate abnormal behavior.
Discovering Heavily Crawled Business Content
Use the Top 10 Crawled Hostnames and Top 10 Crawled Targets panels to identify the primary targets of AI crawlers.
- Top 10 Crawled Hostnames: Shows the top 10 hostnames most frequently crawled by AI crawlers. Clicking a bar updates the hostname filter and refreshes the page.
- Top 10 Crawled Targets: Shows the top 10 hostname-and-URL-path combinations most frequently crawled by AI crawlers.
Analysis tip: If core business pages (such as product detail pages or article content pages) appear frequently in the Top lists, this content is a primary target of AI crawlers. We recommend setting the appropriate action in Crawler Control based on your business policy.