corpus.blog
42,501 blogs · 5,361,753 posts Claim your blog
Record · corpus.blog/posts/01a08745-a662-7267-844e-14458de9b63d

Large language model data pipelines and Common Crawl (WARC/WAT/WET)

blog.christianperone.com · published 3 June 2023

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at blog.christianperone.com Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
External links
18
Words captured
2,440

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://arxiv.org/abs/2302.13971v1
arxiv.org
https://arxiv.org/abs/2306.01116
arxiv.org
https://arxiv.org/abs/2101.00027
arxiv.org
https://aclanthology.org/2020.lrec-1.494/
aclanthology.org
https://commoncrawl.org/
commoncrawl.org
https://commoncrawl.org/donate/
commoncrawl.org
https://commoncrawl.org/the-data/get-started/
commoncrawl.org
https://github.com/miso-belica/jusText
github.com
https://huggingface.co/datasets/tiiuae/falcon-refinedweb
huggingface.co
https://trafilatura.readthedocs.io/en/latest/
trafilatura.readthedocs.io
https://arxiv.org/abs/2107.06499
arxiv.org
https://arxiv.org/abs/2304.01373
arxiv.org
https://github.com/facebookresearch/cc_net/blob/main/cc_net/text_normalizer.py
github.com
https://aclanthology.org/2020.lrec-1.494.pdf
aclanthology.org
https://arxiv.org/abs/2112.11446
arxiv.org
https://www.cs.princeton.edu/courses/archive/spring13/cos598C/broder97resemblance.pdf
cs.princeton.edu
https://fasttext.cc/docs/en/language-identification.html
fasttext.cc
https://arxiv.org/abs/1607.01759
arxiv.org
Elsewhere on blog.christianperone.com (1)