Record · corpus.blog/posts/01a07df5-2eb4-7161-ae13-8a9777bc6b4f

Scrapers I block (and allow), with explanations

Seirdy · published 10 December 2024

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at seirdy.one Wayback capture

1blogs citing

First cited: 1 Feb 2025
Most recent: 1 Feb 2025
Rank this month: not ranked
Links from this post: 60
Words captured: 3,745

Cited by

Blog

In the post

Date

Slatians hideout on the Web | slatecave.net

Interesting Resources for building Search Engines Read at the source ↗

1 Feb 2025

Links from this post

Link

Host

https://git.sr.ht/~seirdy/seirdy.one/blob/b9536a6a9d592d644d53a65b952bd7c95a0faa04/static/robots.txt

git.sr.ht

https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

developers.google.com

https://www.deviantart.com/team/journal/UPDATE-All-Deviations-Are-Opted-Out-of-AI-Datasets-934500371

deviantart.com

https://web.archive.org/web/20241207040446/https://cohost.org/staff/post/272195-cohost-now-sets-devi

web.archive.org

https://github.com/misskey-dev/misskey/pull/10833

github.com

https://github.com/rom1504/img2dataset/pull/218

github.com

https://w3c.github.io/tdm-reservation-protocol/

w3c.github.io

https://www.rfc-editor.org/rfc/rfc9309.html

rfc-editor.org

https://www.peer39.com/crawler-notice

peer39.com

https://www.turnitin.com/robot/crawlerinfo.html

turnitin.com

https://bccampus.ca/2024/10/16/beyond-surveillance-the-case-against-ai-proctoring-ai-detection/

bccampus.ca

https://libguides.bcit.ca/prf.php?id=2c1673be-7cd6-11ed-8528-0266e2e2286e

libguides.bcit.ca

https://academicbot.rtu.lv/

academicbot.rtu.lv

http://webmeup-crawler.com/

webmeup-crawler.com

https://web.archive.org/web/20241112175427/http://webmeup-crawler.com/

web.archive.org

https://www.checkmarknetwork.com/spider.html/

checkmarknetwork.com

https://www.brandverity.com/why-is-brandverity-visiting-me

brandverity.com

https://www.brandverity.com/products/paid-search-monitoring/

brandverity.com

https://pipl.com/bot

pipl.com

https://pipl.com/why-pipl

pipl.com

https://platform.openai.com/docs/bots

platform.openai.com

https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

developers.google.com

https://support.apple.com/en-us/119829

support.apple.com

https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

support.anthropic.com

https://developers.facebook.com/docs/sharing/webmasters/web-crawlers

developers.facebook.com

https://developers.facebook.com/docs/sharing/bot/

developers.facebook.com

https://ds.rois.ac.jp/center8/crawler/

ds.rois.ac.jp

https://ds.rois.ac.jp/center8/

ds.rois.ac.jp

https://webz.io/bot.html

webz.io

https://kangaroollm.com.au/kangaroo-bot/

kangaroollm.com.au

https://openwebsearch.eu/owler/

openwebsearch.eu

https://www.semrush.com/bot/

semrush.com

https://velen.io/

velen.io

https://web.archive.org/web/20031008144550/http://nameprotect.com/botinfo.html

web.archive.org

https://web.archive.org/web/20060207051223/http://www.slysearch.com:80/static/home.html

web.archive.org

https://chatglm.cn/

chatglm.cn

https://allenai.org/crawler

allenai.org

https://imho.alex-kunz.com/2024/01/25/an-update-on-friendly-crawler/

imho.alex-kunz.com

https://www.alex-kunz.com/contact/

alex-kunz.com

https://www.perplexity.ai/hub/technical-faq/how-does-perplexity-follow-robots-txt

perplexity.ai

Robb Knight Read at the source ↗

rknight.me

https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/

rknight.me

https://www.wired.com/

wired.com

https://www.wired.com/story/perplexity-is-a-bullshit-machine/

wired.com

https://en.wikipedia.org/wiki/Well-known_URI

en.wikipedia.org

https://iabtechlab.com/ads-txt/

iabtechlab.com

https://grapheneos.org/articles/sitewide-advertising-industry-opt-out

grapheneos.org

http://commoncrawl.org/

commoncrawl.org

https://webdatacommons.org/

webdatacommons.org

https://seirdy.one//posts/2021/03/10/search-engines-with-own-indexes/

seirdy.one

https://openwebsearch.eu/common-goals-with-common-crawl/

openwebsearch.eu

https://seirdy.one//notes/2023/10/01/living-dead-internet-theory/

seirdy.one

https://blog.majestic.com/company/new-15-year-plus-historic-index/

blog.majestic.com

https://majestic.com/reports/majestic-million

majestic.com

https://tranco-list.eu/

tranco-list.eu

Dark Visitors Read at the source ↗

darkvisitors.com

https://github.com/ai-robots-txt/ai.robots.txt

github.com

https://github.com/matomo-org/device-detector

github.com

https://radar.cloudflare.com/traffic/verified-bots

radar.cloudflare.com

https://tilde.green/~jmjl/

tilde.green