corpus.blog
56,966 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a0875f-dd24-728a-8d3b-06047e39b780

The Technical User's Introduction to LLM Tokenization

christophergs.com · published 27 February 2024

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at christophergs.com Wayback capture
3blogs citing
First cited
30 Sept 2024
Most recent
15 Mar 2026
Rank this month
not ranked
Links from this post
10
Words captured
2,957

Cited by

Blog
In the post
Date
Martin Alderson
Why Claude's new 1M context length is a big deal Read at the source ↗
15 Mar 2026
Martin Alderson
Which programming languages are most token-efficient? Read at the source ↗
8 Jan 2026
The Content Technologist | Content strategy newsletter & website, publishing since 2019.
Tokens and vector embeddings: The first steps in calculating semantics for LLMs Read at the source ↗
15 Aug 2025
khromov
How Chain of Thought Prompting Boosts LLM Performance Read at the source ↗
30 Sept 2024

Links from this post

Link
Host
https://www.youtube.com/watch?v=zduSFxRajkE
youtube.com
https://github.com/openai/tiktoken
github.com
https://tiktokenizer.vercel.app/
tiktokenizer.vercel.app
https://en.wikipedia.org/wiki/Softmax_function
en.wikipedia.org
https://en.wikipedia.org/wiki/Unicode
en.wikipedia.org
https://www.reedbeta.com/blog/programmers-intro-to-unicode/
reedbeta.com
https://paperswithcode.com/paper/language-models-are-unsupervised-multitask
paperswithcode.com
https://en.wikipedia.org/wiki/Byte_pair_encoding
en.wikipedia.org
https://blog.devgenius.io/understanding-tokens-and-tokenization-in-large-language-models-1058cd24b944
blog.devgenius.io
https://github.com/karpathy/minbpe
github.com