corpus.blog
41,554 blogs · 5,358,632 posts Claim your blog
Record · corpus.blog/posts/01a08779-8ca4-7328-91af-0673336d838c

Byte Pair Encoding: building the GPT tokenizer with Karpathy

francescopochetti.com · published 17 March 2024

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at francescopochetti.com Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
External links
37
Words captured
4,073

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://www.youtube.com/watch?v=zduSFxRajkE
youtube.com
https://github.com/karpathy/minbpe
github.com
https://colab.research.google.com/drive/1y0KnCFZvGVf_odSfcNAws6kcDD7HsI0L?usp=sharing
colab.research.google.com
https://i0.wp.com/francescopochetti.com/wp-content/uploads/2024/03/embedding.png?ssl=1
i0.wp.com
https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf
d4mucfpksywv.cloudfront.net
https://tiktokenizer.vercel.app/
tiktokenizer.vercel.app
https://docs.python.org/3/library/stdtypes.html
docs.python.org
https://en.wikipedia.org/wiki/Unicode
en.wikipedia.org
https://en.wikipedia.org/wiki/Text_encoding
en.wikipedia.org
https://en.wikipedia.org/wiki/Unicode_Consortium
en.wikipedia.org
https://en.wikipedia.org/wiki/Writing_system
en.wikipedia.org
https://en.wikipedia.org/wiki/Script_(Unicode
en.wikipedia.org
https://en.wikipedia.org/wiki/Comparison_of_Unicode_encodings
en.wikipedia.org
https://en.wikipedia.org/wiki/UTF-8
en.wikipedia.org
https://en.wikipedia.org/wiki/UTF-16
en.wikipedia.org
https://en.wikipedia.org/wiki/UTF-32
en.wikipedia.org
https://en.wikipedia.org/wiki/ASCII
en.wikipedia.org
https://en.wikipedia.org/wiki/Byte_pair_encoding
en.wikipedia.org
https://i0.wp.com/francescopochetti.com/wp-content/uploads/2024/03/image-3.png?ssl=1
i0.wp.com
https://www.reedbeta.com/blog/programmers-intro-to-unicode/
reedbeta.com
http://www.unicode.org/versions/latest/
unicode.org
http://www.unicode.org/reports/
unicode.org
http://www.unicode.org/notes/
unicode.org
https://i0.wp.com/francescopochetti.com/wp-content/uploads/2024/03/image-4.png?ssl=1
i0.wp.com
https://i0.wp.com/francescopochetti.com/wp-content/uploads/2024/03/image-9.png?ssl=1
i0.wp.com
https://i0.wp.com/francescopochetti.com/wp-content/uploads/2024/03/image-7.png?ssl=1
i0.wp.com
https://i0.wp.com/francescopochetti.com/wp-content/uploads/2024/03/image-8.png?ssl=1
i0.wp.com
https://i0.wp.com/francescopochetti.com/wp-content/uploads/2024/03/image-11.png?ssl=1
i0.wp.com
https://youtu.be/zduSFxRajkE?si=Oym32C9iNY1viZvY&t=4215
youtu.be
https://github.com/openai/gpt-2/blob/9b63575ef42771a015060c964af2c3da4cf7c8ab/src/encoder.py
github.com
https://github.com/openai/tiktoken
github.com
https://github.com/openai/tiktoken/blob/1b9faf2779855124f05174adf1383e53689ed94b/tiktoken_ext/openai_public.py
github.com
https://i0.wp.com/francescopochetti.com/wp-content/uploads/2024/03/image-17.png?ssl=1
i0.wp.com
https://i0.wp.com/francescopochetti.com/wp-content/uploads/2024/03/Screenshot-2024-03-17-at-17.38.54.png?ssl=1
i0.wp.com
https://i0.wp.com/francescopochetti.com/wp-content/uploads/2024/03/Screenshot-2024-03-17-at-17.39.19.png?ssl=1
i0.wp.com
https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation
lesswrong.com
https://i0.wp.com/francescopochetti.com/wp-content/uploads/2024/03/image-18.png?ssl=1
i0.wp.com