corpus.blog
44,124 blogs · 5,475,563 posts Claim your blog
Record · corpus.blog/posts/01a08764-2687-73af-b879-a48083762f5c

Resisting the crawl

cousinthrockmorton.github.io · published 17 April 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at cousinthrockmorton.github.io Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
External links
49
Words captured
3,969

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://www.bbc.com/news/live/cvgq9ejql39t
bbc.com
https://apnews.com/article/trump-transgender-passports-prisons-eggs-sperm-da1d1d280658a8c85c57cfec2f30cefb
apnews.com
https://apnews.com/article/hungary-pride-ban-amendment-orban-gay-rights-lgbtq-155ec12cbbde7cc6be0f96adb323de77
apnews.com
https://apnews.com/article/columbia-university-mahmoud-khalil-ice-440828980a4ee7bf4ddcf3d123e02b3e
apnews.com
https://apnews.com/article/harvard-trump-administration-federal-cuts-antisemitism-0a1fb70a2c1055bda7c4c5a5c476e18d
apnews.com
https://www.youtube.com/watch?v=Gr7T07WfIhM
youtube.com
https://www.greenhills.co.uk/posts/robotstxt-25/
greenhills.co.uk
https://archive.org/
archive.org
https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/
blog.archive.org
https://commoncrawl.org/mission
commoncrawl.org
https://www.forbes.com/sites/virginieberger/2025/03/15/the-ai-copyright-battle-why-openai-and-google-are-pushing-for-fair-use/
forbes.com
https://www.authorsalliance.org/2024/07/30/what-happens-when-your-publisher-licenses-your-work-for-ai-training/
authorsalliance.org
https://en.wikipedia.org/wiki/Neural_scaling_law
en.wikipedia.org
https://arxiv.org/abs/2203.15556
arxiv.org
https://arxiv.org/pdf/2211.04325
arxiv.org
https://www.mdpi.com/2413-4155/6/1/3
mdpi.com
https://arxiv.org/abs/2301.13188
arxiv.org
https://www.zoomsphere.com/blog/zero-click-search-optimization-why-google-is-your-frenemy-and-how-to-beat-it-anyway
zoomsphere.com
https://www.nature.com/articles/s41598-024-61221-0
nature.com
https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/
diff.wikimedia.org
https://pod.geraspora.de/posts/17342163
pod.geraspora.de
https://www.scrye.com/blogs/nirik/posts/2025/03/15/mid-march-infra-bits-2025/
scrye.com
https://www.scrye.com/blogs/nirik/posts/2025/04/20/later-april-infra-bits-2025/
scrye.com
https://social.treehouse.systems/@barthalion/114190930216801561
social.treehouse.systems
https://incidentdatabase.ai/cite/1001/
incidentdatabase.ai
Robots.txt Is Having a Moment: Here's Why We Should Care
techpolicy.press
https://www.ietf.org/slides/slides-aicontrolws-proposal-multi-level-approach-to-managing-ai-crawler-behavior-and-content-protection-00.pdf
ietf.org
https://www.ietf.org/archive/id/draft-jimenez-tbd-robotstxt-update-00.html
ietf.org
https://datatracker.ietf.org/doc/draft-canel-robots-ai-control/
datatracker.ietf.org
https://www.ietf.org/slides/slides-aicontrolws-guardian-news-media-draft-paper-on-an-aitxt-protocol-00.pdf
ietf.org
https://spawning.ai/ai-txt
spawning.ai
https://www.ietf.org/slides/slides-aicontrolws-iptc-and-plus-the-data-mining-embedded-imagevideo-metadata-property-00.pdf
ietf.org
https://github.com/w3c/tdm-reservation-protocol
github.com
https://www.ietf.org/slides/slides-aicontrolws-tdm-ai-making-unit-based-opt-out-declarations-to-providers-of-generative-ai-00.pdf
ietf.org
https://techcrunch.com/2025/03/10/bluesky-is-weighing-a-proposal-that-gives-users-consent-over-how-their-data-is-used-for-ai/
techcrunch.com
https://bsky.app/profile/jay.bsky.team/post/3lkens3n4w223
bsky.app
https://anubis.techaro.lol/
anubis.techaro.lol
https://github.com/TecharoHQ/anubis
github.com
https://kudurru.ai/
kudurru.ai
https://github.com/ai-robots-txt/ai.robots.txt
github.com
https://www.ehow.com/
ehow.com
https://www.web.sp.am/
web.sp.am
https://mailman.nanog.org/pipermail/nanog/2024-April/225407.html
mailman.nanog.org
Nepenthes
zadzmo.org
https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/
arstechnica.com
https://tldr.nettime.org/@tante/113833396846881683
tldr.nettime.org
https://iocaine.madhouse-project.org/
iocaine.madhouse-project.org
Quixotic Read at the source ↗
marcusb.org
https://blog.cloudflare.com/ai-labyrinth/
blog.cloudflare.com