Manage AI Bots
最終更新日：2026-08-21 15:56:07
Reading time: About 5 minutes
Prerequisites: The hostname has been onboarded, and the Bot Management master switch is turned on.
Navigation Path: Security Settings > Policies > [Target Hostname] > Bot Management > AI Bots
What Are AI Bots
AI Bots refer to automated crawler programs related to large language models. They mainly include the following categories:
| AI Bot Category | Description | Examples |
|---|---|---|
| AI Assistant | Focuses on human-AI interaction and task execution. Uses natural language processing (NLP) to understand user instructions and perform tasks such as answering questions, setting reminders, and processing files. | ChatGPT-User, DuckAssist, and more |
| AI Data Scraper | Focuses on precisely scraping specific data, and supports content crawling in complex scenarios. | GPTBot, ClaudeBot, Google-Extended, and more |
| AI Search Crawler | Dedicated to indexing internet content and providing data support for search engines. | Amazonbot, Claude-SearchBot, OAI-SearchBot, and more |
| Undocumented AI Agent | AI entities whose design purpose, operating mechanism, or scope of application has not been clearly documented. | anthropic-ai, cohere-ai, and more |
Configuration Steps
- Log in to the console and go to the page for your subscribed security product.
- Go to Security Settings > Policies.
- Find the hostname whose policies you want to configure, and click Edit to open the Policies editing page.
- Select the Bot Management tab. If the Bot Management master switch is off, turn it on.
- Locate the AI Bots policy, and click Configure on the right to open the configuration page.
- Set an action (Not Used, Log, Deny, or Skip) for each AI Bot category.
- After making changes, click Deploy at the bottom of the page to deploy the configuration (allow 1–3 minutes for the changes to take effect).
Action Descriptions
| Action | Description |
|---|---|
| Not Used | This policy is not used for traffic detection; the traffic still flows into other detection modules. |
| Log | Only logs requests of this type; the request is still forwarded normally. |
| Deny | Blocks the request and returns a 403 response. |
| Skip | Logs the request and skips all subsequent Bot policy checks, but the request still goes through other protections, such as WAF and API Security. |
Protection Recommendations
When to use Log: For initial configuration, when you haven’t yet assessed the impact of AI crawlers on your business.
When to use Deny: When your website contains copyrighted content or sensitive data that you don’t want AI crawlers to scrape for training.
When to use Skip: When your website needs exposure and you want to drive traffic through AI channels.
If you need granular control (to deny or skip only specific AI crawlers): Use the Custom Bots policy to deny or skip traffic matching the User-Agent keywords for the corresponding AI Bots.
List of Covered AI Bots
| AI Bot Category | Bot Name | Corresponding User-Agent Keyword |
|---|---|---|
| AI Assistant | Amazon Kendra Crawler | amazon-kendra |
| Bedrock Bot | bedrockbot/ | |
| ChatGPT-User Crawler | ChatGPT-User/ | |
| DuckAssist Crawler | DuckAssistBot/ | |
| Echobox Crawler | EchoboxBot/ | |
| Google NotebookLM Crawler | Google-NotebookLM | |
| iAsk Bot | iaskspider/ | |
| Liner Bot | LinerBot | |
| Meta-ExternalFetcher Crawler | meta-externalfetcher/ | |
| MistralAI-User Crawler | MistralAI-User/ | |
| Qualified Bot | QualifiedBot | |
| Rye Bot | RyeBot/ | |
| WP Bot | wpbot/ | |
| AI Data Scraper | AI2 Crawler | AI2Bot |
| Applebot-Extended Crawler | Applebot-Extended/ | |
| CC Crawler | CCBot/ | |
| Claude Crawler | ClaudeBot/ | |
| Cotoyogi Crawler | Cotoyogi | |
| Google-Extended Crawler | Google-Extended/ | |
| GPTBot Crawler | GPTBot/ | |
| ICC Crawler | ICC-Crawler/ | |
| ImageSift Bot | ImagesiftBot | |
| Meta-External Crawler | meta-externalagent/ | |
| Pangu Crawler | pangubot | |
| TikTok Crawler | TikTokSpider | |
| Timpi Crawler | Timpibot | |
| Velen Crawler | VelenPublicWebCrawler/ | |
| WARD Bot | WARDBot/ | |
| Zanista Bot | ZanistaBot/ | |
| Cohere Crawler | cohere-training-data-crawler | |
| Diff Crawler | Diffbot | |
| Kangaroo Crawler | Kangaroo | |
| Omgili Crawler | omgili | |
| Webzio-Extended Crawler | Webzio-Extended | |
| AI Search Crawler | Amazon Crawler | Amazonbot/ |
| Channel3 Bot | Channel3 | |
| Claude Search Bot | Claude-SearchBot/ | |
| Claude-User Crawler | Claude-User/ | |
| Google Cloud Vertex Bot | Google-CloudVertexBot | |
| Meta Web Indexer Crawler | meta-webindexer/ | |
| OAI-Search Crawler | OAI-SearchBot/ | |
| Perplexity-User Crawler | perplexity-user/ | |
| Perplexity Crawler | PerplexityBot/ | |
| You Crawler | YouBot/ | |
| Undocumented AI Agent | Anthropic-AI Crawler | anthropic-ai |
| Claude-Web Crawler | Claude-Web | |
| Cohere-AI Crawler | cohere-ai |
Special Requests
The AI Bots library is regularly updated to include mainstream AI large-model application tools. If you have a special request, you can contact customer technical support to request that a specific bot be added. When submitting your request, we recommend including the following information: the bot’s name and purpose, a link to its official documentation, its User-Agent, its source IP/ASN range, and your desired handling policy.