41554 blogs · 4,417,066 posts Claim your blog

Record · corpus.blog/posts/01a087cb-d9a6-7337-ac12-6cf32eb095ef

Wrangling LLM Web Crawlers

pve.dev · published 14 April 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at pve.dev Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
Links from this post: 13
Words captured: 730

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/

thelibre.news

https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt

arstechnica.com

https://www.theverge.com/news/634345/cloudflare-ai-labyrinth-web-scraping-bots-training-data

theverge.com

https://iocaine.madhouse-project.org/

iocaine.madhouse-project.org

https://anubis.techaro.lol/

anubis.techaro.lol

https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin

github.com

https://github.com/traefik/yaegi

github.com

https://github.com/rs/zerolog

github.com

https://github.com/ai-robots-txt/ai.robots.txt

github.com

Nepenthes Read at the source ↗

https://plugins.traefik.io/plugins

plugins.traefik.io

https://github.com/holysoles/bot-wrangler-traefik-plugin?tab=readme-ov-file

github.com

https://altcha.org/

altcha.org