corpus.blog
56,966 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a087d7-423d-70f1-bc61-94baef790254

Transformer Design Guide (Part 2: Modern Architecture)

rohitbandaru.github.io · published 11 January 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at rohitbandaru.github.io Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
Links from this post
82
Words captured
8,947

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://arxiv.org/abs/1706.03762
arxiv.org
https://arxiv.org/abs/2303.08774
arxiv.org
https://arxiv.org/abs/2403.05530
arxiv.org
https://arxiv.org/abs/2302.13971
arxiv.org
https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf
www-cdn.anthropic.com
https://arxiv.org/abs/2307.09288
arxiv.org
https://arxiv.org/abs/2407.21783
arxiv.org
https://arxiv.org/abs/2403.08295
arxiv.org
https://arxiv.org/abs/2408.00118
arxiv.org
https://arxiv.org/abs/2310.06825
arxiv.org
https://arxiv.org/abs/2401.04088
arxiv.org
https://arxiv.org/abs/2309.16609
arxiv.org
https://arxiv.org/abs/2404.14219
arxiv.org
https://arxiv.org/abs/2412.08905
arxiv.org
https://arxiv.org/abs/2401.02954
arxiv.org
https://arxiv.org/abs/2405.04434
arxiv.org
https://arxiv.org/abs/2412.19437
arxiv.org
https://arxiv.org/abs/2402.00838
arxiv.org
https://arxiv.org/abs/2501.00656
arxiv.org
https://arxiv.org/abs/2412.13663
arxiv.org
https://arxiv.org/abs/2407.07726
arxiv.org
https://arxiv.org/abs/2412.03555
arxiv.org
https://arxiv.org/abs/2412.10302
arxiv.org
https://arxiv.org/abs/2302.05442
arxiv.org
Transformer Design Guide (Part 1: Vanilla) Read at the source ↗
rohitbandaru.github.io
https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
cdn.openai.com
https://arxiv.org/abs/1910.07467
arxiv.org
https://arxiv.org/pdf/2302.13971
arxiv.org
https://arxiv.org/abs/2002.05202
arxiv.org
https://arxiv.org/abs/1606.08415
arxiv.org
https://github.com/openai/gpt-2/blob/master/src/model.py
github.com
https://arxiv.org/abs/1710.05941v1
arxiv.org
https://arxiv.org/abs/1612.08083
arxiv.org
https://arxiv.org/abs/1803.02155
arxiv.org
https://arxiv.org/abs/1809.04281
arxiv.org
https://arxiv.org/abs/2108.12409
arxiv.org
https://arxiv.org/abs/1910.10683
arxiv.org
https://arxiv.org/abs/2104.09864
arxiv.org
https://arxiv.org/abs/2306.15595
arxiv.org
https://arxiv.org/abs/2309.16039
arxiv.org
https://arxiv.org/abs/2309.00071
arxiv.org
https://arxiv.org/abs/2407.10671v1
arxiv.org
https://www.youtube.com/watch?v=DwaBQbqh5aE
youtube.com
Relative Positional Encoding Read at the source ↗
jaketae.github.io
https://gudgud96.github.io/2020/04/01/annotated-music-transformer/
gudgud96.github.io
Rotary Embeddings: A Relative Revolution Read at the source ↗
blog.eleuther.ai
You could have designed state of the art Positional Encoding Read at the source ↗
fleetwood.dev
https://nn.labml.ai/transformers/rope/index.html
nn.labml.ai
https://github.com/AliHaiderAhmad001/T5-Relative-Position
github.com
Extending the RoPE Read at the source ↗
blog.eleuther.ai
https://www.youtube.com/watch?v=Mn_9W1nCFLo
youtube.com
https://arxiv.org/abs/2007.14062
arxiv.org
https://arxiv.org/abs/1904.10509
arxiv.org
https://arxiv.org/abs/2004.05150
arxiv.org
https://arxiv.org/abs/2205.14135
arxiv.org
https://arxiv.org/abs/2307.08691
arxiv.org
https://arxiv.org/abs/2407.08608
arxiv.org
https://mathworld.wolfram.com/BlockMatrix.html
mathworld.wolfram.com
https://www.youtube.com/watch?v=zy8ChVd_oTM&t=7s
youtube.com
https://www.youtube.com/watch?v=gMOAud7hZg4
youtube.com
https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1244/slides/cs224n-2024-lecture18-deployment-and-efficiency.pdf
web.stanford.edu
https://arxiv.org/abs/1701.06538
arxiv.org
https://arxiv.org/pdf/1701.06538
arxiv.org
https://pytorch.org/blog/training-moes/
pytorch.org
https://arxiv.org/abs/2006.16668
arxiv.org
https://arxiv.org/abs/2101.03961
arxiv.org
https://arxiv.org/pdf/2401.04088.pdf
arxiv.org
https://github.com/huggingface/transformers/blob/main/src/transformers/models/mixtral/modeling_mixtral.py
github.com
https://arxiv.org/abs/2401.06066
arxiv.org
https://arxiv.org/abs/2408.15664
arxiv.org
https://arxiv.org/abs/2106.05974
arxiv.org
https://www.youtube.com/watch?v=RcJ1YXHLv5o&list=PLoROMvodv4rNiJRchCzutFw5ItR_Z27CM&index=29&ab_channel=StanfordOnline
youtube.com
Mixture of Experts Explained Read at the source ↗
huggingface.co
https://www.reddit.com/r/LocalLLaMA/comments/1c7h4wq/why_llama_3_is_not_a_moe/
reddit.com
https://github.com/huggingface/transformers/issues/31464
github.com
https://github.com/mistralai/mistral-inference/blob/main/src/mistral_inference/moe.py
github.com
https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py
github.com
https://arxiv.org/abs/2211.05102
arxiv.org
https://arxiv.org/abs/1911.02150
arxiv.org
https://arxiv.org/abs/2305.13245
arxiv.org
https://arxiv.org/pdf/2305.13245
arxiv.org
https://arxiv.org/abs/2312.00752
arxiv.org