corpus.blog
56,966 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a08792-98f5-70a6-af08-8adbc0bdf22b

Optimizing training a GPT style Tokenizer with C++

justinhj · published 30 July 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at justinhj.github.io Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
Links from this post
13
Words captured
4,750

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://arxiv.org/pdf/1508.07909
arxiv.org
https://en.wikipedia.org/wiki/Huffman_coding
en.wikipedia.org
https://www.boost.org/doc/libs/1_88_0/libs/multi_index/doc/index.html
boost.org
https://github.com/rsennrich/subword-nmt/blob/92d6139d07d30e12735a0af9e7f7f925ebe62c54/subword_nmt/learn_bpe.py
github.com
https://www.youtube.com/watch?v=zduSFxRajkE
youtube.com
https://huggingface.co/datasets/Salesforce/wikitext
huggingface.co
https://platform.openai.com/tokenizer
platform.openai.com
https://github.com/justinhj/minbpe-cc
github.com
https://docs.python.org/3/library/functions.html
docs.python.org
https://www.amazon.ca/Tour-C-2nd-Bjarne-Stroustrup/dp/0134997832
amazon.ca
https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines
isocpp.github.io
https://github.com/karpathy/minbpe
github.com
https://github.com/rsennrich/subword-nmt
github.com