Crawler Control

Reading time: About 5 minutes
Prerequisites: The hostname is onboarded, and the AI Crawler Control master switch is turned on
Navigation: AI Crawler Control > Crawler Control

Feature Overview

Crawler Control is the core operational page of AI Crawler Control, providing hostname-level control over AI crawlers. On this page, you can view information and control status for all included AI crawlers, and take action on one or more crawlers.

• Basic edition: View the crawler list and control status, and take action (Not Used/Log/Skip/Deny) on one or more crawlers.
• Advanced edition: Additionally view request statistics and activity trends for each crawler to support your control decisions.

Typical Configuration Scenarios

Taking Targeted Action on a Highly Active Crawler

If a specific crawler (such as GPTBot) shows an unusually high request volume, you can take quick action on that single crawler directly from the crawler list.

Steps:

1. Locate the target crawler (for example, GPTBot) in the crawler list.
2. In the Action column for that row, click the target action button (for example, Deny). A confirmation dialog appears: “Are you sure you want to set GPTBot to Deny? Clicking OK deploys the configuration immediately, and it takes effect in about 1–3 minutes.”
3. Click OK → The configuration is deployed immediately, the action button updates, and the control status summary refreshes accordingly.
4. Click Cancel → The operation is canceled and the original setting remains unchanged.

Batch Action on Crawlers

When you need to apply the same action to multiple crawlers of the same or different categories — for example, denying an unwanted category in bulk (such as AI Data Scraper), or skipping a category you want indexed (such as AI Search Crawler) — you can use batch operations.

Steps:

1. Select the crawlers you want to act on, either by filtering by category or by selecting them directly in the crawler list.
2. Once selected, a batch action bar appears above the list, showing “N crawlers selected.”
3. From the Set Action dropdown, choose the target action (Not Used/Log/Skip/Deny).
4. Click Confirm & Deploy. A confirmation dialog appears: “Are you sure you want to set the batch action for the selected N crawlers to [Action]? Clicking OK deploys the configuration immediately, and it takes effect in about 1–3 minutes.”
5. Click OK → The action is updated in bulk for all selected crawlers, and the control status summary refreshes accordingly.
6. To cancel your selection, click Clear Selection to deselect all crawlers.

Applying a Unified Policy Across Multiple Hostnames

If you have already configured an AI Crawler Control policy on a primary hostname and want other hostnames to use the same policy, go to the Security Policy management page and use the Copy Policy feature to copy the AI Crawler Control configuration from the primary hostname to the target hostname, avoiding the need to configure each hostname individually.

Action Descriptions

AI Crawler Control currently provides the following four actions:

Recommendation: For newly onboarded hostnames, we recommend setting all AI crawlers to Log initially. After an observation period, decide whether to set specific crawlers to Skip or Deny based on actual traffic patterns.

Policy Priority

AI Crawler Control follows the same execution order as the “AI Bots” policy in Bot Management:

1. Custom Bots
2. AI Bots
3. Public Bots
4. Definite Bots
5. Likely Bots
6. Web Bot Detection
7. Workflow Detection