Record · corpus.blog/posts/01a0877d-55d9-73ed-9148-856e14c1e1ef

Which AI Fetchers Send Which Headers, Measured on a Live Site

gkoreli.com · published 3 September 2026

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at gkoreli.com Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
Links from this post: 27
Words captured: 6,332

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

Does llms.txt Work? What a Live Implementation Revealed Read at the source ↗

gkoreli.com

https://github.com/gkoreli/blog/tree/main/packages/blog/drafts/research/ai-fetcher-headers

github.com

https://github.com/gkoreli/blog/blob/main/packages/blog/drafts/research/ai-fetcher-headers/data/captures.jsonl

github.com

https://developers.openai.com/api/docs/bots

developers.openai.com

https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

support.claude.com

https://docs.mistral.ai/robots

docs.mistral.ai

https://duckduckgo.com/duckassistbot.html

duckduckgo.com

https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/

datatracker.ietf.org

https://radar.cloudflare.com/bots/directory/duckassistbot

radar.cloudflare.com

https://docs.perplexity.ai/guides/bots

docs.perplexity.ai

https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers

developers.google.com

https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

developers.google.com

https://stackfox.co/research/grok-user-agent

stackfox.co

How I Built First-Party Analytics for a Personal Blog Read at the source ↗

gkoreli.com

https://github.com/omrilotan/isbot

github.com

https://github.com/matomo-org/device-detector

github.com

https://github.com/monperrus/crawler-user-agents

github.com

https://github.com/ai-robots-txt/ai.robots.txt

github.com

https://github.com/arp242/isbot

github.com

Bot Detection Without JavaScript: What My Blog Measured Read at the source ↗

gkoreli.com

https://github.com/cloudflare/web-bot-auth

github.com

https://ppc.land/gemini-ios-app-traffic-revealed-through-undocumented-user-agent/

ppc.land

https://www.seroundtable.com/geminiios-google-user-agent-40333.html

seroundtable.com

https://stat.ripe.net/docs/02.data-api/as-overview.html

stat.ripe.net

https://www.team-cymru.com/ip-asn-mapping

team-cymru.com

https://developers.cloudflare.com/workers/runtime-apis/request/

developers.cloudflare.com

https://github.com/X4BNet/lists_vpn

github.com