About Bot Management
最終更新日：2026-03-25 18:14:02
Overview
With the widespread adoption of the Internet and the accelerated advancement of intelligent technologies, Bot traffic has become an essential part of overall network traffic. While automated tools and intelligent agents bring convenience to people’s lives, they also present significant challenges to network security. Recent studies show that Bot traffic now accounts for more than 50% of global Internet traffic.
Types of Bot Traffic
The composition of Bot traffic is quite complex; not all Bot traffic is either desired or unwanted by enterprises. Simply categorizing Bots as good or bad is not sufficient:
- Benign Bots: Primarily designed to facilitate daily life and work, such as search engines and website monitoring tools.
- Malicious Bots: Mainly characterized by undermining the security and stability of networks, such as participating in CC attacks, credential stuffing, malicious scanning, and automated ticket scalping. Attacks by malicious Bots are especially prevalent in sectors like finance, e-commerce, and social media.
Note: We do not directly define whether a Bot is good or bad for you; you need to make this determination based on your website’s business requirements and select an appropriate Bot Management Policy to bypass beneficial Bot traffic and mitigate malicious Bot traffic.
Bot Management Philosophy
We are committed to building a simple and intelligent automated threat protection system, striving to achieve an optimal balance between enhanced security and an outstanding user experience.
The product is built on the WAAP architecture and leverages technologies such as Bot intelligence databases, heuristics, machine learning, and proactive detection to identify and manage different types of automated traffic on the network, ensuring business security, stable operations, and the maintenance of a competitive edge.
Introduction to Bot Management Policy
The main Bot Management protection policies are as follows. For detailed information, please refer to their respective sections.
| Policy Name | Policy Description | Main Application Scenarios |
|---|---|---|
| Custom Bots | Custom Bot features—such as User-Agent, request headers, or fingerprints—can be used to precisely manage specific traffic. | Applicable to cases where automated programs from internal corporate tools or third-party vendors have explicit authorization to access website services, and need to be bypassed to avoid being mistakenly blocked by subsequent Bot policies. |
| AI Bots | Manage automated program traffic related to AI applications, including AI search crawlers, AI assistants, AI data scraping, and AI proxies without ICP filing. | Suitable for websites that wish to block AI large model crawlers with one click to prevent copyright content theft, sensitive data leakage, and other issues. |
| Public Bots | Manage publicly declared bot traffic on the Internet, such as SEO, marketing analytics, website monitoring, and more. | Suitable for websites that wish to bypass publicly accessible crawlers that are beneficial to business operations with one click. |
| Definite Bots | Manage non-public automated program traffic with distinct characteristics, including automation frameworks, development frameworks, HTTP libraries, vulnerability scanners, crawler tools, proxy tools, fake spiders, and more. | Applicable for users who wish to block malicious automation conducted via commonly used crawler development tools, frameworks, and scanners associated with black and gray market activities with a single click. |
| Likely Bots | Manage traffic exhibiting abnormal behavioral patterns or deviating from typical user characteristics. Identify potential covert automated tools through multi-dimensional detection methods to reduce business fraud and data scraping attacks. | Applicable for scenarios where secondary verification is needed for abnormal and suspicious requests, which are highly likely to be initiated by automated tools, in order to identify and block malicious automation. |
| Web Bot Detection | In specific client scenarios, enhance bot mitigation using proactive detection techniques, such as embedding a JS SDK within HTML pages. | Applicable to Web/H5 pages accessed via standard PC browsers or mobile browsers, including web pages loaded online within in-app browsers in both APPs and mini programs (Enhanced Countermeasures). |
| Workflow Detection | Customizes rules based on the access behavior logic of legitimate users to identify Bot traffic that does not conform to expected logic. | Applicable for websites that wish to finely manage Bot traffic based on their own business operations (Enhanced Countermeasures). |
| Advanced Detection (To Be Dismounted) | Dynamically models access behavior through big data analytics and machine learning technologies to quickly detect simple Bots, complex Bots, and Advanced Persistent Bots (APBs). | Applicable for websites experiencing large volumes of complex Bot traffic and advanced persistent threat Bot traffic (AI capabilities have already been integrated to identify suspected Bots). |
Policy Priority
The policies will be matched and processed according to the following priority from top to bottom based on the configured actions:
- Custom Bots
- AI Bots
- Public Bots
- Definite Bots
- Likely Bots
- Web Bot Detection
- Workflow Detection
- Advanced Detection
Explanation of Action Handling
- Deny: If a request matches a strategy set to “Deny,” the system will immediately terminate the request and respond with a 403. All subsequent security strategy checks are skipped.
- Skip: If a request matches a strategy set to “Skip,” the system will only skip all subsequent bot strategy detections, but the request will still proceed through other security strategy checks.
- Log: If a request matches a strategy set to “Log” the system will log the event and continue to perform all subsequent security strategy checks.
- Secondary Verification: If a request triggers a secondary challenge (such as JavaScript challenge or interactive challenge), and the challenge fails, the request will be blocked; if the challenge is successful, the request is allowed to proceed and will continue to be checked by subsequent strategies.
Important Limitations
-
Policy execution conditions:
Policies are executed in descending order of priority. For details, refer to the Policy Priority section.
• If a request matches a policy, the system will immediately execute the action configured on the policy (such as Deny, Skip, or Log).
• If the action blocks the request, all subsequent policies will no longer take effect. -
Scope of Web Bot Detection:
• Pure API services are not supported: For API-only services without a web interface (where JavaScript detection code cannot be embedded), risk detection capabilities that rely on active page-side detection are unavailable.
• Browser version limitations: Outdated or older browser versions may present compatibility risks due to differences in rendering engines and incomplete standards support, which may affect detection accuracy. Users are advised to use more recent browser versions for more stable performance. -
Policy update and migration:
The AI detection capability in the ‘Advanced Detection’ feature has been upgraded and integrated into the ‘Likely Bots’ feature. The original ‘Advanced Detection’ feature is pending dismounting. Please update the relevant configurations in a timely manner.