corpus.blog
41554 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a07df3-ecc9-70ed-9df1-2ee532e2f235

A Visual Guide to Attention Variants in Modern LLMs

Ahead of AI · published 22 March 2026

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at magazine.sebastianraschka.com Wayback capture
5blogs citing
First cited
6 Apr 2026
Most recent
22 Sept 2026
Rank this month
not ranked
Links from this post
55
Words captured
5,340

Cited by

Blog
In the post
Date
Sebastian Raschka
MiMo-V2.6 Pro Architecture and Training Notes Read at the source ↗
22 Sept 2026
Machine Learning Mastery
The Roadmap to Mastering LLM Inference Optimization Read at the source ↗
21 Sept 2026
chizkidd.github.io
How Attention Became Efficient & Scalable: KV Caching, MQA, GQA, MLA, and Sparse Attention. Read at the source ↗
5 Aug 2026
@adlrocha Beyond The Code | Substack
@adlrocha - Towards local plug-and-play AI Read at the source ↗
17 May 2026
mz00
The Average of Everything (136) Read at the source ↗
6 Apr 2026

Links from this post

Link
Host
https://substackcdn.com/image/fetch/$s_!wjZ_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F187277b8-e96f-4ef8-8a31-d810076a3057_1494x974.png
substackcdn.com
https://sebastianraschka.com/llm-architecture-gallery/
sebastianraschka.com
https://www.redbubble.com/i/poster/LLM-Architecture-Gallery-by-Ahead-of-AI/179274487/flk2
redbubble.com
https://substackcdn.com/image/fetch/$s_!ceew!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03198b65-472c-495e-92c0-62b6af66d720_1999x1313.jpeg
substackcdn.com
https://substackcdn.com/image/fetch/$s_!GrJT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3ed41e9-86ee-4c1c-80a5-c9c2ed59dd9c_1366x1410.png
substackcdn.com
Understanding and Coding Self-Attention, Multi-Head Attention, Causal Attention, and Cross-Attention in LLMs Read at the source ↗
magazine.sebastianraschka.com
https://substackcdn.com/image/fetch/$s_!jdH6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95e6491c-6457-4af0-83f3-b73310bad668_3566x2495.webp
substackcdn.com
https://github.com/rasbt/LLMs-from-scratch
github.com
https://substackcdn.com/image/fetch/$s_!T2XV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feaafc89d-0765-4e90-8f2b-881032cb2d5b_3415x1914.webp
substackcdn.com
https://arxiv.org/abs/1706.03762
arxiv.org
https://substackcdn.com/image/fetch/$s_!8wZw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6e9affd-687b-4170-b1d5-f8e3d2834d2b_3449x1646.webp
substackcdn.com
https://substackcdn.com/image/fetch/$s_!ZbSh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F635f204a-bd68-4162-ac88-ca5da9ba6be8_1498x1092.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!VJyp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7dfb70d9-5eb6-44df-a880-3a37dfe72d7f_1836x1116.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!MSOX!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f47ffd2-bb3a-4936-9720-a88b8d867337_1766x1154.png
substackcdn.com
https://arxiv.org/abs/2305.13245
arxiv.org
Understanding and Coding the KV Cache in LLMs from Scratch Read at the source ↗
magazine.sebastianraschka.com
https://substackcdn.com/image/fetch/$s_!h6wM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f39923c-8357-487d-9e69-40ee18a902e8_2523x1248.png
substackcdn.com
The Big LLM Architecture Comparison Read at the source ↗
magazine.sebastianraschka.com
https://substackcdn.com/image/fetch/$s_!87ib!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95380983-7afc-4456-9d5b-733216c16a91_1920x1440.png
substackcdn.com
https://github.com/rasbt/LLMs-from-scratch/tree/main/ch04/04_gqa
github.com
https://arxiv.org/abs/2405.04434
arxiv.org
A Dream of Spring for Open-Weight LLMs: 10 Architectures from Jan-Feb 2026 Read at the source ↗
magazine.sebastianraschka.com
https://substackcdn.com/image/fetch/$s_!BMEL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d04f89a-aad5-4af7-8b46-4623d1d53ae9_5860x5682.webp
substackcdn.com
https://substackcdn.com/image/fetch/$s_!FcJB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe29cb535-8854-4412-af8d-9bca8d8d05f2_1550x858.webp
substackcdn.com
https://substackcdn.com/image/fetch/$s_!56rA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F186dfe98-db29-4b12-97ab-39d5b21ab460_1920x1440.png
substackcdn.com
https://github.com/rasbt/LLMs-from-scratch/tree/main/ch04/05_mla
github.com
https://substackcdn.com/image/fetch/$s_!1yW7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff564ec9f-5fcb-49c3-b728-b4c8a769bb6c_952x902.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!cbqs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa425874e-baae-45bb-a199-46648c5db725_5860x5682.webp
substackcdn.com
https://substackcdn.com/image/fetch/$s_!edFZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48a9cd31-24a5-47d9-ae37-506763ebc67d_1292x704.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!vcLN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F500e49e0-efb3-476d-b902-e2ede32fdbb3_1600x477.webp
substackcdn.com
https://arxiv.org/abs/2503.19786
arxiv.org
https://substackcdn.com/image/fetch/$s_!3FBb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38e10000-db6c-43fa-844b-ed7d4dfe644c_3299x3302.webp
substackcdn.com
https://github.com/rasbt/LLMs-from-scratch/tree/main/ch04/06_swa
github.com
https://arxiv.org/abs/2512.02556
arxiv.org
https://sebastianraschka.com/llm-architecture-gallery/mla/
sebastianraschka.com
https://substackcdn.com/image/fetch/$s_!TvQp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed46e1ba-6c43-421a-b98a-973987cff3f7_4065x4327.webp
substackcdn.com
From DeepSeek V3 to V3.2: Architecture, Sparse Attention, and RL Updates Read at the source ↗
magazine.sebastianraschka.com
https://substackcdn.com/image/fetch/$s_!B4GF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad5026e1-1dbd-4004-814f-6cd6d735e572_2787x2731.webp
substackcdn.com
https://substackcdn.com/image/fetch/$s_!Efwx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F859ff78b-f05e-49de-91dc-28ecd8a0e22c_1388x928.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!syqZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d109b38-96a1-4830-b1f0-7274e405317a_3973x5350.webp
substackcdn.com
https://arxiv.org/abs/2505.06708
arxiv.org
https://substackcdn.com/image/fetch/$s_!VVeK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafe94d77-ab5f-4420-88e4-8ba1f91992e0_2484x3278.webp
substackcdn.com
https://arxiv.org/abs/2412.06464
arxiv.org
https://substackcdn.com/image/fetch/$s_!XRZY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F876009cb-5479-4e95-9b7d-faebe1f87e89_3252x2158.webp
substackcdn.com
https://substackcdn.com/image/fetch/$s_!hK9M!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb82c3202-6c49-4cd7-a4eb-ab18edcd908e_2100x1350.png
substackcdn.com
https://github.com/rasbt/LLMs-from-scratch/tree/main/ch04/08_deltanet
github.com
https://substackcdn.com/image/fetch/$s_!L9cU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F11deeb70-5766-41eb-bc93-b7c8f81afcb7_2048x1281.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!Q1wq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38f5b449-f122-4520-b6ea-a2437fd9508e_2376x1572.webp
substackcdn.com
https://substackcdn.com/image/fetch/$s_!xCw3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ee9eb2d-c95a-4471-b9b3-72e53f78d722_2048x1028.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!cinD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7d49b8b-9ebb-460e-9a51-20bc4126bda1_2048x1012.png
substackcdn.com
https://huggingface.co/inclusionAI/Ling-2.5-1T
huggingface.co
https://substackcdn.com/image/fetch/$s_!QcU9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65b809c5-83be-4cf7-b4a6-e45d16ef86d3_1596x1562.webp
substackcdn.com
https://substackcdn.com/image/fetch/$s_!KvOc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75c26a29-a18e-4e14-8989-846b0171b9fa_1862x1656.png
substackcdn.com
https://arxiv.org/abs/2603.15569
arxiv.org
https://arxiv.org/abs/2603.15031
arxiv.org