Record · corpus.blog/posts/01a087cf-fde6-72ee-bc2f-6d662fc3e198

Building a web crawler in Go

rauluis.com · published 7 June 2026

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at rauluis.com Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
Links from this post: 19
Words captured: 3,318

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

https://www.fastcompany.com/91380448/cloudflare-vs-perplexity-a-web-scraping-war-with-big-implications-for-ai

fastcompany.com

https://www.californialawreview.org/print/great-scrape

californialawreview.org

https://github.com/devraulu/crowlr/blob/main/pkg/crawler/frontier.go

github.com

https://github.com/devraulu/crowlr/blob/main/pkg/crawler/link.go

github.com

https://github.com/devraulu/crowlr/blob/main/pkg/crawler/crawler.go

github.com

https://en.wikipedia.org/wiki/Lexeme

en.wikipedia.org

https://www.postgresql.org/docs/current/gin.html

postgresql.org

https://github.com/golang-migrate/migrate

github.com

https://github.com/lib/pq

github.com

https://pkg.go.dev/database/sql

pkg.go.dev

https://github.com/devraulu/crowlr/blob/main/pkg/crawler/postgres_store.go

github.com

https://www.postgresql.org/docs/current/datatype-textsearch.html

postgresql.org

https://www.postgresql.org/docs/current/textsearch-controls.html

postgresql.org

https://github.com/devraulu/crowlr

github.com

https://github.com/devraulu/crowlr/blob/main/cmd/crawler/web.go

github.com

https://pdos.csail.mit.edu/6.828/2012/xv6.html

pdos.csail.mit.edu

https://developer.mozilla.org/en-US/docs/Glossary/Crawler

developer.mozilla.org

https://en.wikipedia.org/wiki/URI_normalization

en.wikipedia.org

https://datatracker.ietf.org/doc/html/rfc3986

datatracker.ietf.org