corpus.blog
41,554 blogs · 4,969,865 posts Claim your blog
Record · corpus.blog/posts/01a087a3-0a94-73c5-bf29-86f3d9ebefb7

Defenses against abusive AI scrapers

mayrhofer.eu.org · published 24 January 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at mayrhofer.eu.org Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
External links
15
Words captured
2,472

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://en.wikipedia.org/wiki/Robots.txt
en.wikipedia.org
https://www.rfc-editor.org/rfc/rfc9309.html
rfc-editor.org
https://gohugo.io/templates/robots/
gohugo.io
https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt
github.com
How to block AI Crawler Bots using robots.txt file Read at the source ↗
cyberciti.biz
https://originality.ai/ai-bot-blocking
originality.ai
https://doc.traefik.io/traefik/middlewares/http/ratelimit/
doc.traefik.io
http://nginx.org/en/docs/http/ngx_http_core_module.html
nginx.org
https://github.com/traefik/traefik/issues/3122
github.com
https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-2/
jan.wildeboer.net
Traffic meter per ASN without logs Read at the source ↗
anarc.at
Nepenthes
zadzmo.org
Quixotic Read at the source ↗
marcusb.org
https://github.com/marcus0x62/quixotic/pull/5
github.com
https://idiallo.com/blog/zipbomb-protection
idiallo.com
Elsewhere on mayrhofer.eu.org (5)