---
title: "ianbot: the crawler behind the tools on ian.is"
description: "ianbot is the crawler behind the free tools on ian.is. It reads your sitemap and pages for one run, obeys robots.txt, stores nothing, and identifies itself as ianbot/1.0."
canonical: "https://ian.is/ianbot"
language: "en"
---

# ianbot

What the crawler does, what it never does, and how to allow it if your firewall blocks it.

ianbot fetches pages for the free [tools](https://ian.is/tools) on this site, such as the [internal links tool](https://ian.is/tools/internal-links). It only runs when someone enters a site URL into a tool. It never crawls on a schedule and never visits a site nobody asked it to.

It identifies itself with this user agent:

```
ianbot/1.0 (+https://ian.is/ianbot)
```

## What it does

-   Reads `robots.txt` first and follows it in full. Rules for `ianbot` win over `*` rules. `Crawl-delay` is honoured.
-   Finds pages through the sitemaps listed in `robots.txt` or at the usual sitemap paths. If there is none, it follows links from the home page.
-   Fetches at most 500 HTML pages per run, six at a time, with a 15 second timeout each.
-   Keeps everything in memory for the length of the run. When the report reaches the browser, the pages, text and any API key are gone.

## What it never does

-   It never fetches a URL that `robots.txt` disallows, including redirect hops.
-   If `robots.txt` cannot be read because the server errored, it stops. It does not guess.
-   It never stores or reuses page content, and it never sells or shares anything it reads.

## Allow it in Cloudflare

Cloudflare’s bot protection sometimes challenges ianbot. The tool then reports that Cloudflare blocked it. To allow it:

1.  Open **Security**, then **WAF**, and create a custom rule.
2.  Match requests whose **User Agent contains** `ianbot`.
3.  Choose **Skip** and select the features that blocked the request. The security events log shows which one it was.
4.  Save the rule and run the tool again.

An example expression for one hostname:

```
(http.host eq "www.example.com" and http.user_agent contains "ianbot")
```

Keep the rule as narrow as you can. A user agent string can be copied by anyone, so do not use it to bypass sign-in or protect private pages. ianbot runs on Cloudflare Workers and has no fixed IP address to allow.

## Allow it in robots.txt

If a tool reports that `robots.txt` blocks ianbot, add a group for it:

```
User-agent: ianbot
Allow: /
```

Keep your rules for other crawlers as they are. Allowing ianbot changes nothing for Googlebot or anyone else.

## Block it

Add a `Disallow` group for `ianbot` to `robots.txt` and it will not fetch a single page. If you have a question or a complaint, [contact me](https://ian.is/contact).