56,966 blogs · 4,417,066 posts Claim your blog

Record · corpus.blog/posts/01a087bd-d4b2-708f-8778-9742072edca9

AI Bots and Robots.txt

paulcalvano.com · published 21 August 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at paulcalvano.com Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
Links from this post: 18
Words captured: 3,628

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

https://httparchive.org/

httparchive.org

https://www.robotstxt.org/robotstxt.html

robotstxt.org

https://datatracker.ietf.org/doc/html/rfc9309

datatracker.ietf.org

https://almanac.httparchive.org/

almanac.httparchive.org

https://almanac.httparchive.org/en/2024/seo

almanac.httparchive.org

https://platform.openai.com/docs/bots

platform.openai.com

https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

support.anthropic.com

https://support.apple.com/en-us/119829

support.apple.com

https://github.com/ai-robots-txt/ai.robots.txt

github.com

https://developer.chrome.com/docs/crux/methodology/metrics

developer.chrome.com

https://www.theverge.com/2023/8/7/23823046/openai-data-scrape-block-ai

theverge.com

https://news.ycombinator.com/item?id=37030568

news.ycombinator.com

https://arstechnica.com/information-technology/2023/08/openai-details-how-to-keep-chatgpt-from-gobbling-up-website-data/

arstechnica.com

https://public.tableau.com/views/UserAgentsAppearinginRobots_txtFiles-HTTPArchiveJanuary2023-July2025/Sheet1?:language=en-US&publish=yes&:sid=&:redirect=auth&:display_count=n&:origin=viz_share_link

public.tableau.com

https://support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site

support.squarespace.com

https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/

developers.cloudflare.com

https://public.tableau.com/app/profile/paul.calvano8666/viz/AICrawlerrobots_txtfileDirectivesforPopularSites/Sheet1?publish=yes

public.tableau.com

https://public.tableau.com/views/AICrawlerrobots_txtfileDirectivesforAllSites/Sheet12?:language=en-US&:sid=&:redirect=auth&:display_count=n&:origin=viz_share_link

public.tableau.com