Manage Public Bots
最終更新日：2026-08-21 17:39:05
Reading time: About 5 minutes
Prerequisites: The hostname has been onboarded, and the Bot Management master switch is turned on.
Navigation Path: Security Settings > Policies > [Target Hostname] > Bot Management > Public Bots
What Are Public Bots
Public Bots refer to automated programs on the internet that publicly declare their identity and have a verifiable operating entity, including search engines, website monitoring tools, and marketing analysis crawlers. This type of Bot is usually beneficial to websites, but it can also cause unnecessary bandwidth consumption for websites with sensitive information.
Public Bots are managed by category, and you can set an action independently for each category:
| Public Bot Category | Description | Examples |
|---|---|---|
| Search Engines | Uses automated programs to crawl web content from the internet and store the information in a search engine’s database, providing web search services to users. | Google, Baidu, Yahoo, Sogou crawlers, and more |
| Site Monitor | Automated programs that periodically visit and monitor a website’s availability, performance, and security. | Pingdom, SiteLockSpider, and more |
| Marketing Analysis | Collects and analyzes web content for marketing analysis purposes, helping customers improve the visibility of their ads, materials, and other content. | Google AdWords, Semrush, LinkedIn crawlers, and more |
| Page Preview | Quickly extracts and presents key information from a target page for further processing or display. | Zoom, Twitter crawlers, and more |
| Feed Fetcher | Helps users track and aggregate various content feeds. Uses automated programs to retrieve the latest content from different sources and automatically update and push it. | Archive.org, Feeder, ipip.net crawlers, and more |
| Practical Tools | Automated programs that provide practical assistance to websites, such as saving website snapshots, and analyzing and optimizing page load speed. | Google-Site-Verification, Google-PageRenderer crawlers, and more |
Configuration Steps
- Log in to the console and go to the page for your subscribed security product.
- Go to Security Settings > Policies.
- Click the target hostname to open the Policies editing page.
- Select the Bot Management tab. If the Bot Management master switch is off, turn it on.
- Locate the Public Bots policy, and click Configure on the right to open the configuration page.
- Set an action — Not Used, Log, Skip, or Deny — for each Public Bots category.
- After making changes, click Deploy at the bottom of the page to deploy the configuration (allow 1–3 minutes for the changes to take effect).
Action Descriptions
| Action | Description |
|---|---|
| Not Used | This policy is not used for traffic detection; the traffic still flows into other detection modules. |
| Log | Only logs requests of this type; the request is still forwarded normally. |
| Deny | Blocks the request and returns a 403 response. |
| Skip | Logs the request and skips all subsequent Bot policy checks, but the request still goes through other protections, such as WAF and API Security. |
Protection Recommendations
- When to use Log: For initial configuration, when you haven’t yet assessed the impact of Public Bots on your business.
- When to use Deny: When your website contains sensitive information, or when bandwidth is limited and you don’t want crawlers consuming resources.
- When to use Skip: When your website needs exposure and relies on search engine traffic.
- If you need granular control (to deny or skip only specific public crawlers): Use the Custom Bots policy to deny or skip specific public crawlers.
Inclusion Criteria
Bots included in the Public Bot library must meet all of the following conditions:
- Verifiable identity: The bot has a clearly defined operating entity, with a verifiable identity marker that is difficult to impersonate, and its access behavior and network attributes remain stable and consistent over time.
- Sufficient traffic volume: The bot must generate stable traffic across the network (for example, an average of more than 1,000 verified daily requests over the past month), to ensure the library reflects widely applicable bots.
- Traceable IP intelligence: The IP data used to identify the bot must come from a traceable source, such as an official API or RDNS database.
Removal Conditions
If a previously included bot no longer meets the conditions above, it will be removed from the Public Bots library. Examples of policy violations include:
- Abusive behavior: Failing to comply with reasonable rate limits, or frequently triggering WAF or high-risk control rules.
- Inconsistent or spoofed identity: The User-Agent impersonates a well-known bot, but its IP, RDNS, or official source information does not match the declared identity.
Special Requests
The Public Bot library is regularly updated to include mainstream bots for search engine SEO, marketing analysis, website monitoring, and more, and supports handling by category.
If you have a special request, you can contact customer technical support to request that a specific bot be added. When submitting your request, we recommend including the following information: the bot’s name and purpose, a link to its official documentation, its User-Agent, its source IP/ASN range, and your desired handling policy.