Quick Start

Reading time: About 4 minutes
Prerequisites: None. (The “For Onboarded Hostnames” section requires that the hostname is already onboarded and the AI Crawler Control master switch is turned on — see that section for details.)

This document helps you understand how to onboard a hostname for protection, view the AI crawlers interacting with your hostname, and take action on them.

New Hostname Onboarding: Initial Policy Configuration

During the Confirm Initial Policy step of the hostname onboarding flow, you can configure the initial AI Crawler Control policy for a new hostname.

Method 1: Recommended Default Policies

1. On the Confirm Initial Policy page during hostname onboarding, locate the AI Crawler Control module.
2. Turn on the AI Crawler Control master switch (enabled by default).
3. Select the initial action for AI crawlers: Not Used / Log (default) / Skip / Deny. For guidance on when to use each action, see the action descriptions in the Crawler Control section.
4. Complete the onboarding flow. The selected initial action applies to all AI crawlers uniformly.

Method 2: Duplicate Policies from an Existing Hostname

1. During hostname onboarding, select Duplicate policies from an existing hostname.
2. After selecting the reference hostname, review the displayed status of its AI Crawler Control master switch and the number of crawlers configured under each action to confirm they match your expectations.
3. Once confirmed, the new hostname inherits the complete AI Crawler Control policy from the reference hostname.

For Onboarded Hostnames: Adjusting the AI Crawler Control Policy

Feature Location

AI Crawler Control is displayed as a top-level menu in the product’s side navigation, with two subpages:

Website Protection
├── Dashboard
├── AI Crawler Control
│   ├── Trends Overview (Advanced edition only)
│   └── Crawler Control
├── Analysis & Logs
├── Security Settings
└── ...

Viewing AI Crawler Trends

Go to AI Crawler Control > Trends Overview to get a global view of how AI crawlers access your website, including total request volume and trends, crawler category distribution, rankings of the most active crawlers, and the most frequently crawled hostnames and paths. You can filter and analyze the data by combining dimensions such as time, hostname, crawler category, and action result.

For details, see the Trends Overview (Advanced Edition Only) section.

Controlling AI Crawler Access

Go to AI Crawler Control > Crawler Control to view the control status of all AI crawlers currently included for a hostname, and to take action (Not Used/Log/Skip/Deny) on one or more crawlers. In the Advanced edition, you can also view request statistics and activity trends for each crawler to support your control decisions.

For details, see the Crawler Control section.