Record · corpus.blog/posts/01a087c0-18e0-73a3-bc6b-6b7218ed9bc7

Gigatoken: making tokenisation care about the hardware

peterbhabra.com · published 30 July 2026

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at peterbhabra.com Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
External links: 27
Words captured: 4,154

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

https://github.com/marcelroed/gigatoken/tree/542367a3efed134883fb4f1140b49c04e6fad3a3

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/profiling/campaign_report.md

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/README.md

github.com

https://arxiv.org/abs/1508.07909

arxiv.org

https://github.com/openai/tiktoken/blob/main/tiktoken/_educational.py

github.com

https://github.com/marcelroed/gigatoken/commit/97678e8d1dd2426035909235e03bbafeeefe6cc1

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/src/pretokenize/fast/mask.rs

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/src/pretokenize/fast/r50k.rs

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/src/pretokenize/fast/qwen2.rs

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/src/pretokenize/fast/o200k.rs

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/src/pretokenize/fast/deepseek_v3.rs

github.com

https://www2.cs.sfu.ca/~ashriram/papers/2014_PACT_GREP.pdf

www2.cs.sfu.ca

https://arxiv.org/abs/1902.08318

arxiv.org

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/src/pretokenize/unicode.rs

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/src/bpe/pretoken_cache.rs

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/src/bpe/tiktoken.rs

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/src/pretokenize/mod.rs

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/profiling/zen5_st_profile.md

github.com

https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html

kernel.org

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/src/load_tokenizer/hf.rs

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/src/bpe/mod.rs

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/profiling/x86_port_plan.md

github.com

https://github.com/huggingface/tokenizers/blob/v0.21.4/tokenizers/src/tokenizer/mod.rs

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/src/batch.rs

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/profiling/mt4_analysis/mt_round4_findings.md

github.com

https://github.com/marcelroed/gigatoken/issues/36

github.com

https://github.com/marcelroed/gigatoken/blob/542367a3efed134883fb4f1140b49c04e6fad3a3/benchmarks/compare/measure.py

github.com