Record · corpus.blog/posts/01a08760-1544-727d-a777-c61cd424ce6e

Surviving the Crawlers

Chronicae Novis Rebus · published 7 December 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at chronicles.mad-scientist.club Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
Links from this post: 19
Words captured: 2,884

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

You probably shouldn't block AI bots from your website Read at the source ↗

chronicles.mad-scientist.club

https://web.archive.org/web/20251009214719/https://www.anthropic.com/research/small-samples-poison

web.archive.org

https://iocaine.madhouse-project.org/

iocaine.madhouse-project.org

https://kegel.com/c10k.html

kegel.com

https://monitor.madhouse-project.org/dashboard/snapshot/ZNA8k2mGFABofoX6EuqMfbTfqHSVoR2m?orgId=0&from=2025-12-06T00%3A00%3A08.389987178%2B01%3A00&to=2025-12-07T00%3A00%3A08.389987178%2B01%3A00&timezone=browser&var-instance=nam-shub-of-enki&var-hosts=algernon&refresh=1m&viewPanel=panel-15

monitor.madhouse-project.org

https://monitor.madhouse-project.org/dashboard/snapshot/r3ykdoygvkNstYtRAfAknTEpKK0g2nYo?orgId=0&from=2025-12-03T00%3A00%3A10.526776216%2B01%3A00&to=2025-12-04T00%3A00%3A10.526776216%2B01%3A00&timezone=browser&var-instance=nam-shub-of-enki&var-hosts=algernon&refresh=1m&viewPanel=15

monitor.madhouse-project.org

https://github.com/ai-robots-txt/ai.robots.txt

github.com

https://come-from.mad-scientist.club/@iocaine/statuses/01KBK73ZEQTHAVWKH2FX4HCYM4

come-from.mad-scientist.club

https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

developers.google.com

https://gitlab.gnome.org/World/podcasts/-/blob/17adb25301da8fbdcfad85fb8e12d92e5b0f1f1c/podcasts-data/src/lib.rs

gitlab.gnome.org

https://github.com/indieweb/indieauth-client-php/blob/3833f7fcbf5aa26ec2b5caf1a2e4e2336e837c07/src/IndieAuth/Client.php

github.com

https://caddyserver.com/

caddyserver.com

https://caddyserver.com/docs/caddyfile/directives

caddyserver.com

https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/refs/heads/main/Caddyfile

raw.githubusercontent.com

https://caddyserver.com/docs/caddyfile/directives/templates

caddyserver.com

fail2ban Read at the source ↗

github.com

https://ipset.netfilter.org/

ipset.netfilter.org

https://come-from.mad-scientist.club/@algernon/statuses/01KB9Z0N8S6M3NKQDAQEN35R6X

come-from.mad-scientist.club

https://git.madhouse-project.org/algernon/bazmeg

git.madhouse-project.org