Record · corpus.blog/posts/01a087d0-97b6-7006-be44-6da32be50845

Robots.txt Won't Save You

Ryan Bagley · published 12 July 2024

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at rb.ax Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
External links: 29
Words captured: 2,704

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

https://knowyourmeme.com/memes/just-go-on-the-internet-and-tell-lies

knowyourmeme.com

https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/

developers.cloudflare.com

https://www.nytimes.com/robots.txt

nytimes.com

https://www.reddit.com/robots.txt

reddit.com

https://roelant.net/en/2023/im-blocking-ai-crawlers/

roelant.net

Block the Bots that Feed “AI” Models by Scraping Your Website Read at the source ↗

neil-clarke.com

Robots.txt to block AI bots Read at the source ↗

coryd.dev

https://pika.pika.page/

pika.pika.page

https://pika.pika.page/posts/your-pika-robots-txt-file

pika.pika.page

Dark Visitors

darkvisitors.com

https://www.washingtonpost.com/politics/2023/08/08/zooms-privacy-tweaks-stoke-fears-that-its-calls-will-be-used-train-ai/

washingtonpost.com

https://www.theverge.com/2021/3/12/22326975/facebook-training-ai-public-videos-digital-memories

theverge.com

https://www.washingtonpost.com/technology/2023/09/08/gmail-instagram-facebook-trains-ai/

washingtonpost.com

https://techcrunch.com/2023/09/01/xs-privacy-policy-confirms-it-will-use-public-data-to-train-ai-models/

techcrunch.com

https://9to5google.com/2023/07/03/google-privacy-policy-ai-training-data/

9to5google.com

https://arxiv.org/pdf/2001.08361

arxiv.org

https://www.nytimes.com/2024/04/06/technology/tech-giants-harvest-data-artificial-intelligence.html

nytimes.com

https://www.nytimes.com/2023/12/27/business/media/new-york-times-open-ai-microsoft-lawsuit.html

nytimes.com

https://www.theverge.com/2023/8/21/23840705/new-york-times-openai-web-crawler-ai-gpt

theverge.com

https://www.theverge.com/2024/6/25/24185984/reddit-robots-txt-fight-ai-bots-scraping-crawlers

theverge.com

https://www.404media.co/google-is-paying-reddit-60-million-for-fucksmith-to-tell-its-users-to-eat-glue/

404media.co

https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/

rknight.me

https://www.wired.com/story/perplexity-is-a-bullshit-machine/

wired.com

https://www.reuters.com/technology/artificial-intelligence/multiple-ai-companies-bypassing-web-standard-scrape-publisher-sites-licensing-2024-06-21/

reuters.com

https://matthewbutterick.com/chron/ai-scraping-and-publicly-available-web-data.html

matthewbutterick.com

https://blog.cloudflare.com/ai-bots/

blog.cloudflare.com

https://arstechnica.com/information-technology/2023/10/university-of-chicago-researchers-seek-to-poison-ai-art-generators-with-nightshade/

arstechnica.com

https://arxiv.org/pdf/2406.12027

arxiv.org

https://arstechnica.com/tech-policy/2024/07/glaze-a-tool-protecting-artists-from-ai-bypassed-by-attack-as-demand-spikes/

arstechnica.com