corpus.blog
41554 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a07df3-ecc9-70ed-9df1-2ee530f50643

Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention

Ahead of AI · published 16 May 2026

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at magazine.sebastianraschka.com Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
Links from this post
46
Words captured
5,620

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://substackcdn.com/image/fetch/$s_!D6nK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd336da4a-46ee-4659-a976-1c42a27d5c64_2048x1614.png
substackcdn.com
https://sebastianraschka.com/llm-architecture-gallery/
sebastianraschka.com
https://substackcdn.com/image/fetch/$s_!W4-4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ecb1329-b25a-457b-9c2f-f908fddb667c_2026x2048.png
substackcdn.com
https://arxiv.org/abs/2405.12981
arxiv.org
https://magazine.sebastianraschka.com/i/140464659/bonus-topic-cross-attention
magazine.sebastianraschka.com
A Visual Guide to Attention Variants in Modern LLMs Read at the source ↗
magazine.sebastianraschka.com
https://substackcdn.com/image/fetch/$s_!SjqJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F466ed20e-ea76-4b19-a50b-c3e4b915fb26_2048x868.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!OYUw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe110f15-a3cf-496e-bfbe-57dc4e038ed9_2048x1661.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!5OyN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61360482-0e84-401b-bec9-1a0c682225a0_2048x1205.png
substackcdn.com
https://github.com/rasbt/LLMs-from-scratch/blob/main/ch02/03_bonus_embedding-vs-matmul/embeddings-and-linear-layers.ipynb
github.com
https://substackcdn.com/image/fetch/$s_!VjPJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42430f1c-e645-4093-ab48-dd5844a30751_2048x1689.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!-WWQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6937af22-3316-4ee5-9425-bbf04c040252_2048x1306.png
substackcdn.com
https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/17_gemma4
github.com
https://substackcdn.com/image/fetch/$s_!dJn9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50528b3d-7a4e-4067-ad69-640c7ef760f1_1594x1072.png
substackcdn.com
https://poolside.ai/
poolside.ai
https://substackcdn.com/image/fetch/$s_!kxUJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd44428e5-854c-472c-b4f9-39622911f7c2_2048x1945.png
substackcdn.com
https://huggingface.co/poolside/Laguna-XS.2/blob/main/config.json
huggingface.co
https://substackcdn.com/image/fetch/$s_!eGqC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb64d0bbf-dbac-4926-a89f-c133f01dbe7e_2048x1858.png
substackcdn.com
https://arxiv.org/abs/2404.14619
arxiv.org
https://www.zyphra.com/post/zaya1-8b
zyphra.com
https://huggingface.co/Zyphra/ZAYA1-8B/blob/main/config.json
huggingface.co
https://substackcdn.com/image/fetch/$s_!yR9D!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5e38f80-cb78-463f-90d9-85590acc0830_2048x2037.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!AXXV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcfe260c7-ecd4-4e61-9360-e940bf56309c_2048x1231.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!F5KU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3eaa5744-9a0f-4377-9012-4370045e95bf_2048x1059.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!Kvry!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad6e9cf3-2427-4aca-8afc-74202ddf93c4_2048x1208.png
substackcdn.com
https://arxiv.org/abs/2605.05365
arxiv.org
https://arxiv.org/abs/2510.04476
arxiv.org
https://substackcdn.com/image/fetch/$s_!xOdf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe32dc053-e3c0-4f74-b616-76a5740d25d4_2048x875.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!Obw7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F517336aa-d63f-4402-a71b-945a6e4b3725_1228x2048.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!fwtt!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65f26fb2-3043-44c1-98b3-84ab24af86cf_2048x1700.png
substackcdn.com
https://arxiv.org/abs/2512.24880
arxiv.org
https://arxiv.org/pdf/2409.19606
arxiv.org
https://www.youtube.com/watch?v=q_IlqYlYhlo
youtube.com
https://substackcdn.com/image/fetch/$s_!O-TC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f2bea55-cf52-4e65-bc83-b2c14b281549_1665x2048.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!ORcC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6deb5f4b-5ca7-4e23-9a81-045fa93d1807_2048x665.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!1CCD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f2de79c-c7ab-4b97-8070-f642fd7b4336_2048x939.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!gpmF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faebe9064-5008-435e-ad70-c2217c1cc032_2048x647.png
substackcdn.com
From DeepSeek V3 to V3.2: Architecture, Sparse Attention, and RL Updates Read at the source ↗
magazine.sebastianraschka.com
https://substackcdn.com/image/fetch/$s_!A-_C!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3412fd9-5f5e-435e-9824-88c3abc27d12_2048x946.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!Eri4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3384e17d-a4fb-4361-878c-11fbe73d591d_2048x940.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!XJd5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9887bd77-474c-4973-b171-6638a27c2e29_2048x1280.png
substackcdn.com
https://substackcdn.com/image/fetch/$s_!2GF3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68d954ed-83f7-40f8-9399-3323ecffd6e2_2128x1308.jpeg
substackcdn.com
https://mng.bz/Nwr7
mng.bz
https://amzn.to/4aAKiFY
amzn.to
https://sebastianraschka.com/llm-architecture-gallery/active-parameter-ratio/
sebastianraschka.com
https://arxiv.org/abs/2409.19606
arxiv.org