AI Bots
Last update：2026-03-25 18:14:05
Overview
This feature helps you manage traffic from AI-related automated programs, including AI Assistant, AI Data Scraper, and AI Search Crawler. It provides one-click blocking and monitoring for known AI Bots, helping prevent sensitive data leakage and unauthorized scraping of copyrighted content caused by AI crawlers.
AI Bots Category
AI bots on the platform are grouped into the following four categories:
| AI Bots Category | Description |
|---|---|
| AI Assistant | Focuses on human-computer interaction and task execution, understands user instructions through natural language processing (NLP), and completes operations such as question and answer, reminders, and file processing. |
| AI Data Scraper | Focuses on accurately capturing specific data and supports crawling content in complex scenarios. |
| AI Search Crawler | Dedicated to indexing Internet content and providing data support for search engines. |
| Undocumented AI Agent | An artificial intelligence entity whose design purpose, operating mechanism or scope of application is not clearly defined. |
Response Actions
| Action | Description |
|---|---|
| Not Used | Do not use this policy for traffic detection; traffic will still flow into other detection modules. |
| Log | Only logs this type of request; the request will be forwarded as usual. |
| Deny | Deny request by a default 403 response. |
Steps
- Log in to the console and go to the subscribed security product page.
- Go to Security Settings–>Policies.
- Select the domain you wish to configure the security policy and click
to enter the Security Policy editing page. - Open the Bot Management tab and enable the master switch if it is turned off.
- Go to AI Bots, click <Configure> button on the right to open the configuration page.
- Set different actions for different AI Bots categories according to your business needs, including Not Used, Log, and Deny.
- Click Publish Changes at the bottom to publish the configuration. Changes take effect within 1–3 minutes.
Note: After the configuration takes effect, verify the protection results promptly to ensure normal business access.
Protection Recommendations
- If your website does not require AI application traffic and you are concerned about the security of sensitive data or copyrighted content, it is recommended to set the AI Bots category to Deny. This will allow you to deny access from the corresponding AI Bots with a single click.
- If you need granular protection, such as blocking only a specific AI large model crawler, please use a Custom Bots policy and deny the corresponding User-Agent keyword.
AI Bots List
| AI Bots Category | Bot Name | Corresponding User-Agent Keyword |
|---|---|---|
| AI Assistant | ChatGPT-User Crawler | ChatGPT-User |
| DuckAssist Crawler | DuckAssistBot | |
| Meta-ExternalFetcher Crawler | Meta-ExternalFetcher | |
| AI Data Scraping Tool | GPTBot Crawler | GPTBot |
| AI2 Crawler | AI2Bot | |
| Applebot-Extended Crawler | Applebot-Extended | |
| CC Crawler | CCBot | |
| Claude Crawler | ClaudeBot | |
| Cohere Crawler | cohere-training-data-crawler | |
| Diff Crawler | Diffbot | |
| Google_Extended Crawler | Google-Extended | |
| Kangaroo Crawler | Kangaroo | |
| Meta_External Crawler | Meta-ExternalAgent | |
| Omgili Crawler | omgili | |
| Pangu Crawler | PanguBot | |
| Timpi Crawler | Timpibot | |
| Webzio_Extended Bot | Webzio-Extended | |
| AI Search Bot | Oai_Search Bot | OAI-SearchBot |
| Amazon Bot | Amazonbot | |
| Perplexity Crawler | PerplexityBot | |
| You Crawler | YouBot | |
| AI Proxy Without ICP Filing | Anthropic-AI Crawler | anthropic-ai |
| Claude-Web Crawler | Claude-Web | |
| Cohere-AI Crawler | cohere-ai |
AI Bot Library Updates and Inclusion Requests
The AI Bots library is continuously updated to include mainstream AI large model applications and tools. If a specific AI bot has not yet been included and you want it added to the library, please contact our technical support to submit a request. To help speed up the review process, please provide the following information:
- Bot name and purpose
- Link to official documentation
- User-Agent
- Source IP/ASN range
- Expected handling action