Manage Custom Bots
Last update：2026-08-21 17:47:21
Reading time: About 4 minutes
Prerequisites: The hostname has been onboarded, and the Bot Management master switch is enabled.
Navigation Path: Protection Settings > Shared Configurations > Custom Bots
Use Cases
If you need more granular management than the platform’s preset Bot policies can provide, use Custom Bots:
- Allow authorized automation tools: For example, automated programs developed in-house or DevOps/monitoring/synchronization tools from third-party vendors, to prevent them from being mistakenly blocked by policies such as Definite Bots or Likely Bots.
- Apply finer-grained control over preset Bots: For example, if the platform has set a category of AI Bots/Public Bots to Log by default, but you want to skip only certain specific bots within that category.
- Deny specific malicious Bots: Target specific malicious crawlers, spoofed Bots, or abnormal automated programs that have fixed characteristics.
- Manage Bots not covered by preset categories: Individually manage new or custom-built Bots that are not yet covered by the platform’s preset categories.
Note: Custom Bots take effect before other Bot policies (AI Bots, Public Bots, Definite Bots, Likely Bots, and so on), but after IP/Geo Firewall, Custom Rules, Whitelist, DDoS Protection, and Rate Limiting. If a request matches a DDoS Protection policy, DDoS Protection still takes effect even if Custom Bots is configured to skip that request.
Configuration Steps
- Log in to the console and go to the subscribed security product page.
- Go to Protection Settings > Shared Configurations.
- Select the Custom Bots tab, then click Add Rule.
- Set the match conditions based on characteristics such as IP/CIDR and User-Agent, and preset the action for the Custom Bot.
- Click the Associated Hostnames icon and select the hostnames this rule should apply to.
- Click OK to submit the policy deployment task (takes 1–3 minutes to take effect).
Action Descriptions
| Action | Description |
|---|---|
| Log | The system only logs this type of request; the request is still forwarded normally. |
| Deny | The system blocks the request and returns a 403 response. |
| Skip | The system logs the request and skips all subsequent Bot policy checks, but the request still goes through other protections, such as WAF and API Security. |
Configuration Examples
Example 1: Allow a commonly used internal testing tool
Scenario: An in-house automated testing tool is being identified and blocked by Definite Bots, and needs to be allowed through.
| Configuration Item | Value |
|---|---|
| Rule Name | Allow internal testing tool |
| Match Conditions | User-Agent regex ^Scrapy/ AND client IP is within 192.168.0.0/16 |
| Action | Skip |
Example 2: Allow a specific public crawler
Scenario: You want to skip only the Google crawler while denying all other Public Bots.
| Configuration Item | Value |
|---|---|
| Rule Name | Allow Google crawler |
| Match Conditions | Public Bots equals Search Engines-Googlebot |
| Action | Skip |
Example 3: Manage a Bot not covered by preset categories
Scenario: A new custom data-scraping tool is not yet recognized by the platform and needs to be managed individually.
| Configuration Item | Value |
|---|---|
| Rule Name | Restrict unknown scraping tool |
| Match Conditions | User-Agent regex ^MyCustomBot/[\d\.]+ |
| Action | Deny |