Heuristics Detection

Reading time: About 8 minutes

Feature Overview

Heuristic Detection is one of the foundational identification capabilities in Bot Management. It relies on a rule and signature library built up from extensive experience countering Bots, matching known risk signals in requests in real time to quickly identify access behavior with clear automation characteristics — such as basic scanning, malicious crawlers, script-based access, and requests from known automation tools.

Heuristic Detection can identify the following common types of automated threats:

• Simple Bots: Scripts or automation tools with a fixed behavior pattern and obvious characteristics.
• Complex Bots: Automated access with some degree of disguise, but that still exhibits identifiable anomalies.
• Disguised Bots: Automated requests that mimic normal client behavior by forging a User-Agent, request headers, or a crawler identity.

The results of Heuristic Detection serve as a key input into the Bot Score, working together with machine learning analysis results to assess request risk — improving detection efficiency, detection coverage, and the explainability of results.

How It Works

The Heuristic Detection engine runs on edge nodes worldwide, analyzing requests online in real time. When a request reaches a node, the system extracts key characteristics from the request and matches them against threat intelligence, an anomaly signature library, and automation-tool signatures to identify known risk patterns.

Unlike machine learning, which focuses on uncovering unknown behavior patterns, Heuristic Detection is better suited to identifying:

• Requests from known automation tools
• Clearly abnormal client and request header characteristics
• Established high-risk access patterns
• Malicious requests impersonating a public crawler identity

Detection Dimensions

Heuristic Detection identifies automated threats primarily along the following dimensions:

1. IP Intelligence

Identifies risk attributes of the request’s source IP using threat intelligence, including but not limited to:

• Cloud provider and hosting-facility IPs
• Known proxy IPs
• High-risk IPs with recent DDoS, vulnerability-scanning, or Bot attack activity

This dimension helps quickly identify request sources with a known risk background, but is not by itself equivalent to a malicious determination.

2. Client Characteristic Anomalies

Analyzes the completeness and consistency of the client’s User-Agent, as well as outdated browser and OS versions that carry higher security or compatibility risk. Includes but is not limited to:

• Forged User-Agent
• Abnormal or mismatched version declarations
• Missing key fields
• High-risk characteristics associated with outdated browser or OS versions

This dimension helps identify disguised browsers, low-quality automation tools, or clients with clearly abnormal characteristics.

3. HTTP Request Header Validation

Validates the completeness and consistency of browser request headers, for example:

• Severe or partial absence of fields such as Accept-, Sec-CH-UA, or Sec-Fetch-
• Request header characteristics that don’t match the client type declared in the User-Agent
• Browser requests missing typical header fields

This dimension is used to identify disguised browser requests and abnormal automated access.

4. TLS Fingerprint and Protocol Characteristics

Based on low-level encryption characteristics from the client-server handshake (such as the JA4 fingerprint), this dimension determines whether the client’s fingerprint is consistent with its declared identity, and detects disguise characteristics from known Bot tool libraries.

This dimension helps identify disguise behavior that would be difficult to detect from surface-level fields alone.

5. Automation Tool and Signature Library Identification

Identifies signatures of full-chain automation tools commonly used by black-and-grey market operators, including but not limited to:

• HTTP request libraries, such as Python’s requests or urllib
• Browser automation frameworks, such as Selenium or Puppeteer
• Command-line and scanning tools, such as curl, wget, Nmap, or Burp Suite
• Crawler and proxy tools, such as Scrapy or Proxychains
• Requests impersonating a public crawler identity, such as spoofing Googlebot or other public crawlers

This dimension is used to quickly identify automated requests from known tools.

Use Cases

Heuristic Detection is well suited to the following scenarios:

• Identifying requests from known automation tools: For detecting access from basic scripts, automation frameworks, scanning tools, and common crawler tools.
• Detecting disguised clients or spoofed crawler identities: For identifying automated behavior that attempts to bypass detection by forging a User-Agent, request headers, or a public crawler identity.
• Identifying basic scanning and bulk access behavior: For detecting scanning, scraping, or bulk requests with a fixed pattern and obvious automation characteristics.
• Serving as a baseline input for Bot risk assessment: Heuristic Detection provides clear risk tags that support Bot Score calculation and downstream policy actions.

How It Works with Machine Learning

Heuristic Detection and machine learning are the two core technologies behind the Bot Score, and each plays a distinct role:

Heuristic Detection is better suited to identifying:

• Known tools
• Known anomalies
• High-confidence risk characteristics
• Automated behavior that can be quickly matched using rules

Its strengths are fast detection, strong real-time performance, and highly explainable results.

Machine learning is better suited to identifying:

• Automated behavior without obvious characteristics
• Bots with complex behavior patterns
• Attack methods that continuously change or evolve
• Anomalous access patterns that are difficult to cover directly with rules

Its strengths include uncovering unknown threats, complements the coverage gaps left by Heuristic Detection, and is better suited to identifying complex or evolving attacks.

Combined Assessment
The system combines heuristic tags, behavioral characteristics, and machine learning analysis results to jointly assess each request and generate a final Bot Score. Based on the Bot Score and related detection results, you can further configure protective actions such as Log, Challenge, or Deny.

Key Capabilities

• Real-time detection: Performs online identification at the edge, suitable for high-concurrency access scenarios.
• Explainable results: Outputs specific risk tags, making log analysis and risk tracing easier.
• High detection efficiency: Achieves high detection efficiency for known tools, known anomalies, and fixed-pattern threats.
• Easy to combine with other capabilities: Can be used together with machine learning, the Bot Score, and security policies.