ianbot

Cloudflare’s bot protection and security rules sometimes challenge ianbot. To let it through:

1. In the Cloudflare dashboard, open your domain and go to Security rules.
2. Select Create rule, then Custom rules.
3. Set Field to User Agent, Operator to contains and Value to ianbot.
4. Under Choose action, pick Skip and tick the features that blocked ianbot, usually All Super Bot Fight Mode Rules or All managed rules. Browser Integrity Check and Security Level are under More components to skip. The security events log shows which one it was.
5. Select Deploy and run the tool again.

To limit the rule to one hostname, edit the expression:

(http.host eq "www.example.com" and http.user_agent contains "ianbot")

No rule can skip Bot Fight Mode, the bot setting on the Free plan. If it is on, turn it off under Security, then Settings (filter by Bot traffic) while the tool runs, and turn it back on after.

Vercel’s Bot Protection challenges requests that do not come from a browser, and ianbot is not a browser. A custom rule with the Bypass action lets it through:

1. In the Vercel dashboard, open your project and select Firewall in the sidebar.
2. Select ⋯, then Configure, then Add New… and Rule.
3. Name the rule, for example “Allow ianbot”.
4. Set If to User Agent, Contains, ianbot.
5. Set Then to Bypass and select Save Rule.
6. Select Review Changes, then Publish. The rule works straight away, with no redeploy.
7. Run the tool again.

Vercel runs custom rules in order. If you have rules that deny or challenge traffic, move this one above them.

Attack Mode only lets through bots on Vercel’s verified list, and Vercel does not offer a way to add your own. If the rule does not help and Attack Mode is on, turn it off under Firewall, then Bot Management while the tool runs.

AWS WAF’s Bot Control rule group blocks user agents that do not look like a browser, through its SignalNonBrowserUserAgent rule. To let ianbot through:

1. In the AWS WAF console, open the web ACL for your site. The new console calls it a protection pack.
2. In the sampled requests or your WAF logs, find a request whose user agent contains ianbot and note the rule that blocked it.
3. If it was the Bot Control rule group, edit the rule group and add a scope-down statement so it only inspects requests whose User-Agent header does not contain ianbot.
4. Save and run the tool again.

If a different rule blocked ianbot, add the same condition to that rule.

Akamai Bot Manager lets you define your own bots and choose what happens to them. To allow ianbot:

1. In Akamai Control Center, create a custom bot category for trusted bots, or use one you already have.
2. Add a custom-defined bot to that category that matches the User-Agent request header *ianbot*, with wildcards turned on.
3. In your security policy, set that category’s action to Allow or Monitor.
4. Save and activate the security configuration, then run the tool again.

If someone else manages Akamai for you, send them these steps.

Sucuri’s Aggressive bot filter blocks any user agent that does not match a real browser. Sucuri can allow IP addresses and URL paths, but not a user agent, so the filter has to be off while the tool runs:

1. In the Sucuri firewall dashboard, open Settings, then Security.
2. Turn off Aggressive bot filter.
3. Check that Access Control, then Block User Agents does not list ianbot.
4. Run the tool again, then turn the filter back on.

For any other firewall, security plugin or host:

1. Find a request from ianbot in your firewall’s or plugin’s log and note the rule that blocked it.
2. Add an exception to that rule, or a rule that allows requests whose user agent contains ianbot.
3. Run the tool again.

If you cannot change the firewall yourself, send your host the user agent at the top of this page and ask them to allow it.