Record · corpus.blog/posts/01a08783-2996-72ba-a876-ac53f42b5025

Designing Websites for AI Agents - llms.txt, Structured Data, and Agent-Readable Web Practices | hidekazu-konishi.com

hidekazu-konishi.com · published 26 July 2026

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at hidekazu-konishi.com Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
External links: 34
Words captured: 8,312

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

support.claude.com

https://arxiv.org/abs/2311.09735

arxiv.org

https://developers.google.com/search/docs/appearance/ai-features

developers.google.com

https://developers.google.com/search/docs/fundamentals/ai-optimization-guide

developers.google.com

https://www.rfc-editor.org/rfc/rfc9309.html

rfc-editor.org

https://blog.cloudflare.com/content-signals-policy/

blog.cloudflare.com

https://blog.cloudflare.com/introducing-pay-per-crawl/

blog.cloudflare.com

https://datatracker.ietf.org/wg/aipref/about/

datatracker.ietf.org

https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/

w3.org

https://www.sitemaps.org/protocol.html

sitemaps.org

https://developers.openai.com/api/docs/bots

developers.openai.com

https://developers.google.com/crawling/docs/about-crawling

developers.google.com

https://docs.perplexity.ai/docs/resources/perplexity-crawlers

docs.perplexity.ai

https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers

developers.facebook.com

https://developer.amazon.com/amazonbot

developer.amazon.com

https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/

blog.cloudflare.com

https://support.apple.com/en-us/119829

support.apple.com

https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap

developers.google.com

https://blogs.bing.com/webmaster/July-2025/Keeping-Content-Discoverable-with-Sitemaps-in-AI-Powered-Search

blogs.bing.com

https://www.rfc-editor.org/rfc/rfc4287.html

rfc-editor.org

https://www.rssboard.org/rss-specification Read at the source ↗

rssboard.org

https://llmstxt.org/

llmstxt.org

https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data

developers.google.com

https://schema.org/Article

schema.org

https://developers.google.com/search/docs/appearance/structured-data/article

developers.google.com

https://developers.google.com/search/docs/appearance/structured-data/dataset

developers.google.com

https://developers.google.com/search/updates

developers.google.com

https://developers.google.com/crawling/docs/troubleshooting/http-status-codes

developers.google.com

https://developers.google.com/search/docs/crawling-indexing/canonicalization

developers.google.com

https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics

developers.google.com

https://commoncrawl.org/faq

commoncrawl.org

The rise of the AI crawler

vercel.com

https://web.dev/articles/vitals

web.dev

https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

developers.google.com

Elsewhere on hidekazu-konishi.com (8)

Hidekazu Konishi - Profile / Biography | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Amazon Bedrock Converse API Deep Dive - Unified Model Interface, Conversation State, Tool Use, and Streaming | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

AWS WAF AI Traffic Monetization - Returning HTTP 402, and What It Puts on the Request Path | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Tool Use and Agent Protocol History and Timeline - Function Calling, MCP, and A2A | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

MCP Server Ecosystem Reference 2026 | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

HTTP Security Headers Complete Reference - CSP, HSTS, COOP, COEP, Permissions Policy | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Web Performance Checklist for Core Web Vitals - LCP, INP, and CLS | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Self-Hosted Static Site Analytics - A Privacy-First Implementation Guide | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com