The crawler

Every request we make identifies itself with a user agent: CorpusBot/1.0 (+https://corpus.blog/crawler). The address in it is this page.

It was called BloggerBot until 22 September 2026. Same crawler, same pace, repository's name rather than the product's. If you have a rule blocking BloggerBot, it stopped matching on that date — block the name above instead, and the block works again.

The version means something. 0.x was a crawler that did not read your disallow list. 1.0 is one that does. That is the only thing the number has ever been spent on, so an operator reading an access log can tell the two apart from the string alone.

In January 2026 we published a proposal for decent crawling: three things a crawler can do with infrastructure that already exists on almost every site. This crawler is meant to be that proposal's reference implementation. It is not one yet. Here is each principle, what we do, and what we do not.

1. Respect robots.txt — actually

We read your robots.txt before we guess at anything. Its Sitemap: directives are the only place you can name a sitemap on another host or under a path nobody would try, so they are read first and taken as given. Only when it declares none do we try seven well-known paths — /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap/sitemap.xml, /sitemap.xml.gz, /sitemap.txt — and every candidate is fetched and verified as a sitemap before it is recorded, because a great many sites answer /sitemap.xml with their ordinary HTML error page under a 200 status.

We read your Disallow lines, and they reach every request we make. The feed fetch, the pages behind it, the page fetch that extracts an article's text and the sitemap read all ask the same question of the same parsed file before they connect. If you disallow it, we do not fetch it.

What that means in detail, because "we respect robots.txt" is a sentence anybody can write:

• We look for the group addressed to CorpusBot, and use the * group only when you have not named us. If you have named us, your rules for us replace the wildcard's rather than stacking on top of them — singling us out should mean what you meant by it.
• Allow and Disallow are resolved the way RFC 9309 says: the longest matching rule wins, and an equal-length tie goes to Allow. The Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php pair that ships with WordPress therefore means what its author meant.
• * and $ work. So does percent-encoding: /café/ and /caf%C3%A9/ are the same path to us, and /a%2Fb is not /a/b.
• The path we match on includes the query string, so a rule aimed at ?paged= reaches the catalogue walk below.
• We follow redirects one hop at a time and ask the rules of the host we land on, not only the one we started at.
• We do not obey Crawl-delay as a maximum. Where you declare one it becomes the floor under our own pause — longer if you asked for longer, never shorter.

We ask each host for /robots.txt at most once a day, and keep what it said. That is deliberate: this crawler visits tens of thousands of sites, and the places it makes more than one request to a site — the catalogue walk below, the page fetches after a feed read — would otherwise double their request count asking permission each time. A site with no robots.txt, or one we could not reach, is treated as allowing us, and that answer is kept for hours rather than a day, because it is a guess rather than something you told us.

If your robots.txt refuses the feed we hold for you, we record that against the feed and stop reading it. We do not delete what we already have and we do not drop your site: your feed keeps its place in the rotation, we re-check it about once a week, and the moment the rule goes the refusal is cleared and reading resumes. Nothing has to be re-added by hand.

There is no broader discovery behind any of this. We have no site crawler — nothing walks a site page by page looking for articles. Feed discovery reads your homepage's <link rel="alternate"> tags first, which is what the tag is for, and falls back — only when the homepage declares none — to a short list of well-known feed paths and to the same tags on your /blog or /news page.

2. Poll the sitemap, not the site

Most sources are polled by feed rather than by sitemap: for a blog the feed is the smaller request and it carries the title, the link and the date already. articles:fetch --limit=300 runs hourly over the 45,695 sources that have a feed, longest-waited first, so nothing is polled continuously and nothing can starve. That is 152 hourly runs to cover all of them: your turn comes round about every six days. One request, page one of your feed, and we stop.

The sitemap poll exists for the sources a feed cannot describe — a blog that has published for a decade and whose feed serves its last ten posts. sitemaps:discover --truncated --limit=200 and sitemaps:fetch --truncated --limit=20 run hourly over exactly that population, 2,571 sources as this is written, against the 3,771 sitemaps we hold. Twenty sites an hour is a pass measured in days, not hours, and it drains rather than rotates: once a sitemap has filled in a source's archive, the source stops qualifying and is not polled that way again.

We do not send conditional requests. There is no If-Modified-Since or If-None-Match on a feed or sitemap read, so a file that has not changed since our last visit is downloaded in full rather than answered with a 304. That is our waste and your bandwidth, and it is open work.

3. Crawl only what changed

New URLs are fetched, once. A feed or sitemap entry we already hold is recognised by its URL and ignored, so re-reading a feed inserts nothing and costs nothing beyond the read itself. About two thirds of the feeds we read carry the full post in the feed, so for those sources no page is fetched at all. Where a page is fetched, it is at most one page per post, only for posts your feed or sitemap already named, and it is paced: at least 250 milliseconds after the previous request to your site — or whatever longer gap your Crawl-delay asks for — one request at a time, never in parallel. The page fetch and the sitemap read share one lock per site, so they queue behind each other rather than stack, and a feed fetch of a site is never started while another is still running.

A modified post is not re-fetched. We parse <lastmod> and use it only to decide which child of a sitemap index to read first; we never store it, so we have nothing to compare a later read against. The proposal asks for exactly this and we do not do it.

A removed URL is not de-indexed. A post that drops out of your sitemap keeps its record and its stored text. The proposal says de-index; our record is a citation graph, and a page that has gone is still a page other posts linked to, so what to do here is a decision rather than an oversight — and it has not been made yet.

The one place we ask for many pages

A feed is a window onto a site, not the site. WordPress serves ten posts by default, and for a blog with ten years behind it that window hides almost everything. So once per feed, and only once, we walk the pages behind page one: ?paged=2, then 3, and so on, following whatever the feed itself declares as its next page where it declares one.

This is deliberately the least polite thing we do, so it is the most tightly bounded. Each page is its own request, dispatched only after the previous one has finished — one request in flight at a time, by construction — with catalogue_page_delay, two seconds, between them — or your Crawl-delay, where that is longer. The walk stops as soon as a page returns nothing we do not already hold, and never runs past catalogue_max_pages, 200 pages. After that the walk is finished for that feed and is not started again, unless it was interrupted before it reached an end.

What we never fetch

Images, scripts and comment feeds. A WordPress-style comment feed is refused by name and by path if it turns up during discovery — it would poll forever and fill the record with remarks on posts rather than posts. We do not follow links out of your posts to fetch what they point at, and a source with no feed and no sitemap is not fetched at all.

What we do with it

The text is held privately to compute the figures on this site, and it is never shown or republished. What is public is metadata: titles, links, dates and counts.

How to stop it

A robots.txt Disallow addressed to CorpusBot, or to *, is read and obeyed — see principle 1 for exactly how it is matched and how long it takes to reach us, which is at most a day for a host we have asked recently and immediately for one we have not. Crawl-delay works too, if you would rather slow us down than stop us.

Matching the user agent above at your server or CDN and refusing it also works, works immediately, and needs no trust in us at all. Once claiming opens, pausing your own source from your claim page will be the direct way to do it.