41554 blogs · 4,417,066 posts Claim your blog

Record · corpus.blog/posts/01a0b877-e9bf-7002-94d4-c97083ce4a64

The challenge of responsible web crawling and scraping in the AI era

thegreenwebfoundation.org · published 29 June 2026

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at thegreenwebfoundation.org Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
Links from this post: 15
Words captured: 1,255

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

Adoption, Hype, Refusal, and Resistance: Talking about AI Read at the source ↗

thegreenwebfoundation.org

https://carbontxt.org/

carbontxt.org

AI model cards in carbon.txt and an AI Sustainability Directory Read at the source ↗

thegreenwebfoundation.org

https://carbontxt.org/ai-model-cards

carbontxt.org

https://negroniventurestudios.com/2026/05/27/it-wasnt-built-for-this/

negroniventurestudios.com

Wayback Machine Director Pushes Back on AI Scraping Fears Driving Archive Blocks Read at the source ↗

blog.archive.org

https://www.thegreenwebfoundation.org/tools/green-web-dataset/

thegreenwebfoundation.org

https://blog.cloudflare.com/18-november-2025-outage/

blog.cloudflare.com

New helper added: Green web checker – Check if a website is hosted on a verified green web host.. Read at the source ↗

thegreenwebfoundation.org

https://www.linkedin.com/in/julien-nioche-4b7b453/

linkedin.com

https://stormcrawler.apache.org/

stormcrawler.apache.org

https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent

developer.mozilla.org

https://developers.google.com/search/docs/crawling-indexing/robots/intro?hl=en

developers.google.com

https://www.thegreenwebfoundation.org/green-web-checker-faq/

thegreenwebfoundation.org

https://www.thegreenwebfoundation.org/support-form/

thegreenwebfoundation.org