Record · corpus.blog/posts/01a0bbef-2698-72a1-86e4-6abdcb347a77

AI Companies Ignoring Robots.txt

Michael Tsai · published 24 June 2024

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at mjtsai.com Wayback capture

1blogs citing

First cited: 5 May 2026
Most recent: 5 May 2026
Rank this month: not ranked
Links from this post: 27
Words captured: 1,070

Cited by

Blog

In the post

Date

gregoryhammond.ca

Wanting to stop, block or confuse AI? Do it slightly differently than everyone else Read at the source ↗

5 May 2026

Links from this post

Link

Host

https://www.fastcompany.com/91144894/perplexity-ai-ceo-aravind-srinivas-on-plagiarism-accusations

fastcompany.com

https://www.wired.com/story/perplexity-is-a-bullshit-machine/

wired.com

https://pxlnv.com/linklog/perplexity-ceo-responds/

pxlnv.com

https://c.im/@nickheer/112671815471138598

c.im

https://news.ycombinator.com/item?id=40779108

news.ycombinator.com

https://www.wired.com/story/perplexity-plagiarized-our-story-about-how-perplexity-is-a-bullshit-machine/

wired.com

https://www.forbes.com/sites/randalllane/2024/06/11/why-perplexitys-cynical-theft-represents-everything-that-could-go-wrong-with-ai/

forbes.com

https://docs.perplexity.ai/docs/perplexitybot

docs.perplexity.ai

https://www.wired.com/tag/chatgpt/

wired.com

https://www.wired.com/story/six-practical-tips-for-using-anthropic-claude-chatbot/

wired.com

https://www.perplexity.ai/search/perplexity-is-a-41uH2h6JT0qazoM87BO.kw

perplexity.ai

https://www.businessinsider.com/openai-anthropic-ai-ignore-rule-scraping-web-contect-robotstxt

businessinsider.com

https://mastodon.macstories.net/@johnvoorhees/112657270550067067

mastodon.macstories.net

https://www.reuters.com/technology/artificial-intelligence/multiple-ai-companies-bypassing-web-standard-scrape-publisher-sites-licensing-2024-06-21/

reuters.com

Apple Intelligence Training Read at the source ↗

mjtsai.com

The New York Times Sues OpenAI Read at the source ↗

mjtsai.com

Perplexity’s grand theft AI Read at the source ↗

theverge.com

594: We Just Found It on the Doorstep Read at the source ↗

atp.fm

https://pxlnv.com/blog/carelessness-of-perplexity/

pxlnv.com

https://www.theverge.com/command-line-newsletter/656599/perplexitys-ceo-on-fighting-google-and-the-coming-ai-browser-war

theverge.com

https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/

blog.cloudflare.com

https://news.ycombinator.com/item?id=44785636

news.ycombinator.com

https://www.cloudflare.com/learning/network-layer/what-is-an-autonomous-system/

cloudflare.com

https://www.cloudflare.com/learning/bots/what-is-robots-txt/

cloudflare.com

https://www.perplexity.ai/hub/blog/agents-or-bots-making-sense-of-ai-on-the-open-web

perplexity.ai

https://www.theregister.com/2025/08/05/perplexity_vexed_by_cloudflares_claims/

theregister.com

https://daringfireball.net/linked/2025/08/05/cloudflare-perplexity

daringfireball.net