Public Bots

Overview

Manage automated program traffic such as active search engines, data crawlers, and website monitoring tools on the Internet. Crawlers that benefit your website operations can be skipped with a single click. For websites with sensitive information, you can also deny these crawlers with one click.

Public Bots Category

Public bots on the platform are grouped into the following six categories：

Response Actions

You can choose to Log, Deny, or Skip public Bot traffic of different categories. Please refer to the table below for details:

Steps

1. Log in to the console and go to the subscribed security product page.
2. Go to Security Settings–>Policies.
3. Select the domain you wish to configure the security policy and click to enter the Security Policy editing page.
4. Open the Bot Management tab and enable the master switch if it is turned off.
5. Go to Public Bots, click the <Configure> button on the right to open the configuration page.
6. Set different actions for different Public Bots categories according to your business needs, including Not Used, Skip, Log, and Deny.
7. Click Publish Changes at the bottom to publish the configuration. Changes take effect within 1–3 minutes.

Protection Recommendations

• If your website relies on traffic acquisition or promotion, it is recommended to set public bots to Skip to avoid blocking beneficial crawlers by mistake.
• For websites dealing with sensitive information or limited bandwidth, it is recommended to set public bots to Deny to mitigate crawler-induced strain.
• For more granular protection, if you need to only skip or block specific crawlers, please use a Custom Bots policy and define specifying categories + User-Agent keywords. For details, refer to the Custom Bots section.

Inclusion Criteria

To be included in the Public Bot Library, a bot must meet all of the following criteria:

• Verifiable Identity: The bot operator must be clearly identified, possess a verifiable and hard-to-imitate identity, and demonstrate consistent access behavior and stable network characteristics over time.
• Traffic Volume Threshold: The robot needs to generate stable traffic across the network (e.g., the average daily verified traffic request number is greater than 1,000 in the last month) to ensure the universality of the inclusion.
• Traceable IP Intelligence: IP data used for bot identification must have a traceable source, such as official APIs or RDNS databases.

Policy Violations

A bot will be removed from the Public Bot Library if it no longer meets the criteria above. Examples of policy violations include:

• Abusive Behavior: Failing to comply with reasonable rate limiting, or frequently triggering WAF or high-risk risk control rules.
• Mismatched or Spoofed Identity: The User-Agent claims to be a well-known bot, but its IP address, RDNS, or official source information does not match the declared identity.

Public Bot Library Updates and Inclusion Requests

The Public Bot Library is continuously updated to include widely used bots for search engine indexing, SEO, market intelligence, website monitoring, and other common use cases. Category-based handling is supported.
If a specific Public bot is not yet included, you can contact technical support to request its addition to the library. To help speed up the review process, it is recommended that you provide the following information:

• Bot name and purpose
• Link to official documentation
• User-Agent
• Source IP/ASN range
• Expected handling action

Public Bots List