Manage Definite Bots

Reading time: About 4 minutes
Prerequisites: The hostname has been onboarded, and the Bot Management master switch is turned on.
Navigation Path: Security Settings > Policies > [Target Hostname] > Bot Management > Definite Bots

What Is Definite Bots

Definite Bots refers to traffic with clear automated-tool characteristics, or traffic that spoofs Public Bots. This includes:

• HTTP libraries: Python HTTP libraries such as requests and urllib, commonly used for automation scripts and crawler development.
• Command-line HTTP request tools: Tools such as curl and wget that can send HTTP requests directly from a terminal, commonly used for debugging, scripted calls, simple scraping, or probing.
• Browser automation testing tools: Tools such as Selenium and Puppeteer that simulate real user actions in a browser. They are commonly used for functional testing, but can also be abused for automated order fraud, bulk registration, and similar activity.
• Vulnerability scanners: Tools such as Nmap and Nessus, used for security testing and vulnerability scanning, but also commonly used by attackers.
• Crawler tools: Tools such as Heritrix and Scrapy, used to automatically crawl web content. They are legitimately used for data collection, but high-frequency requests or attempts to bypass anti-crawling measures may be treated as malicious.
• Proxy tools: Tools such as Proxychains and Luminati, used to relay or intercept network traffic. They can hide the real IP address or analyze communication content, and are often used together with other tools to bypass restrictions.
• Fake spiders: Requests that impersonate public crawlers (such as Googlebot) by spoofing the User-Agent or other identifiers to evade anti-crawling measures, typically used for covert scraping or attacks.

These tools are commonly used by black-and-grey market operators to launch malicious attacks. Once the system identifies traffic based on User-Agent characteristics or fake spider behavior, it tags the traffic as Definite Bots, records a Bot Score of 100, and generates the corresponding Bot Tag to support further analysis and traceability. For details, see the Bot Score and Bot Tag sections under Related References.

Configuration Steps

1. Log in to the console and go to the page for your subscribed security product.
2. Go to Security Settings > Policies.
3. Find the hostname whose policies you want to configure, and click Edit to open the Policies editing page.
4. Select the Bot Management tab. If the Bot Management master switch is off, turn it on.
5. Locate the Definite Bots policy, where you can directly set the action: Not Used, Log, or Deny.
6. After making changes, click Deploy at the bottom of the page to submit the configuration (allow 1–3 minutes for the changes to take effect).

Action Descriptions

Protection Recommendations

We recommend enabling Deny directly: These crawlers typically originate from malicious activity by black-and-grey market operators, so we recommend enabling Deny mode directly.
Before enabling Deny, confirm the following: Whether your website receives traffic from authorized automation tools (such as internally developed programs or automation scripts from third-party partners). If it does, we recommend first configuring a Skip rule for these tools under Custom Bots, and then enabling Deny for Definite Bots, to avoid false positives.