corpus.blog
41554 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a07d5d-72e7-7112-bd07-c15501fcc034

Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Hugging Face · published 18 December 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at huggingface.co Wayback capture
1blogs citing
First cited
14 Jan 2026
Most recent
14 Jan 2026
Rank this month
not ranked
Links from this post
28
Words captured
3,146

Cited by

Blog
In the post
Date
manthanguptaa.in
How To Train Your Own Language Model - Part 1: Tokenization Read at the source ↗
14 Jan 2026

Links from this post

Link
Host
https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/thumbnail.png
huggingface.co
Transformers v5: Simple model definitions powering the AI ecosystem Read at the source ↗
huggingface.co
https://github.com/huggingface/transformers/pull/40936/files
github.com
https://huggingface.co/spaces/Xenova/the-tokenizer-playground
huggingface.co
https://github.com/huggingface/transformers
github.com
https://github.com/huggingface/tokenizers
github.com
https://x.com/suchenzang/status/1697862650053660721
x.com
https://huggingface.co/docs/tokenizers/en/api/normalizers
huggingface.co
https://huggingface.co/docs/tokenizers/en/api/models
huggingface.co
https://huggingface.co/learn/llm-course/en/chapter6/5
huggingface.co
https://huggingface.co/learn/llm-course/en/chapter6/7
huggingface.co
https://huggingface.co/learn/llm-course/en/chapter6/6
huggingface.co
http://hf.co/HuggingFaceTB/SmolLM3-3B
hf.co
https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/hierarchy.png
huggingface.co
https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/tokenization_utils_base.py
github.com
https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/tokenization_utils_tokenizers.py
github.com
https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/tokenization_python.py
github.com
https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/tokenization_utils_sentencepiece.py
github.com
https://github.com/google/sentencepiece
github.com
https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/models/auto/tokenization_auto.py
github.com
https://huggingface.co/openai-community/gpt2/blob/main/config.json
huggingface.co
https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/v4-llama.png
huggingface.co
https://github.com/huggingface/transformers/blob/0a8465420eecbac1c6d7dd9f45c08dd96b8c5027/src/transformers/models/llama/tokenization_llama.py
github.com
https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/v5-llama.png
huggingface.co
https://youtu.be/zduSFxRajkE?si=ZAfCjZjpyPHsnyfF
youtu.be
https://huggingface.co/blog/qgallouedec/gotchas-in-tokenizer-behavior
huggingface.co
Chat Templates: An End to the Silent Performance Killer Read at the source ↗
huggingface.co
https://x.com/ariG23498/status/1999058214906888237
x.com