Public Bots

Overview

Manage automated program traffic such as active search engines, data crawlers, and website monitoring tools on the Internet. Crawlers that benefit your website operations can be skipped with a single click. For websites with sensitive information, you can also deny these crawlers with one click.

Public Bots Category

Public bots on the platform are grouped into the following six categories：

Response Actions

You can choose to Log, Deny, or Skip public Bot traffic of different categories. Please refer to the table below for details:

Steps

1. Log in to the console and go to the subscribed security product page.
2. Go to Security Settings–>Policies.
3. Select the domain you wish to configure the security policy and click to enter the Security Policy editing page.
4. Open the Bot Management tab and enable the master switch if it is turned off.
5. Go to Public Bots, click the <Configure> button on the right to open the configuration page.
6. Set different actions for different Public Bots categories according to your business needs, including Not Used, Skip, Log, and Deny.
7. Click Publish Changes at the bottom to publish the configuration. Changes take effect within 1–3 minutes.

Protection Recommendations

• Recommended for Log: Use this when configuring for the first time and the impact of public bots on your business has not yet been assessed.
• Recommended for Deny: Use this when your website contains sensitive information, or when bandwidth is limited and you do not want crawlers to consume resources.
• Recommended for Skip: Use this when your website needs greater visibility or relies on search engine traffic.
• For fine-grained management：denying or skipping only specific public crawlers — use a custom Bot policy and select the specific public crawlers to deny or skip.

Inclusion Criteria

To be included in the Public Bot Library, a bot must meet all of the following criteria:

• Verifiable Identity: The bot operator must be clearly identified, possess a verifiable and hard-to-imitate identity, and demonstrate consistent access behavior and stable network characteristics over time.
• Traffic Volume Threshold: The robot needs to generate stable traffic across the network (e.g., the average daily verified traffic request number is greater than 1,000 in the last month) to ensure the universality of the inclusion.
• Traceable IP Intelligence: IP data used for bot identification must have a traceable source, such as official APIs or RDNS databases.

Policy Violations

A bot will be removed from the Public Bot Library if it no longer meets the criteria above. Examples of policy violations include:

• Abusive Behavior: Failing to comply with reasonable rate limiting, or frequently triggering WAF or high-risk risk control rules.
• Mismatched or Spoofed Identity: The User-Agent claims to be a well-known bot, but its IP address, RDNS, or official source information does not match the declared identity.

Public Bot Library Updates and Inclusion Requests

The Public Bot Library is continuously updated to include widely used bots for search engine indexing, SEO, market intelligence, website monitoring, and other common use cases. Category-based handling is supported.
If a specific Public bot is not yet included, you can contact technical support to request its addition to the library. To help speed up the review process, it is recommended that you provide the following information:

• Bot name and purpose
• Link to official documentation
• User-Agent
• Source IP/ASN range
• Expected handling action