corpus.blog
56,966 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a0b84e-9c76-7110-8548-2901bb0c5e45

How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog

siboehm.com · published 31 December 2022

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at siboehm.com Wayback capture
31blogs citing
First cited
22 Feb 2024
Most recent
23 Aug 2026
Rank this month
not ranked
Links from this post
35
Words captured
14,105

Cited by

Blog
In the post
Date
hoanh.space
Writing Fast Attention Backward Kernel for 5090 in CUDA Read at the source ↗
23 Aug 2026
sankalp's blog
Auto-research with codex: How I achieved a 232x Faster Kernel over baseline with Codex in GPU Mode's qr_v2 problem Read at the source ↗
8 Jul 2026
mccloskeybr.com
Matrix Multiplication: A Performance Analysis Read at the source ↗
19 Apr 2026
danielvegamyhre.github.io
MXFP8 GEMM: Up to 99% of cuBLAS performance using CUDA + PTX Read at the source ↗
29 Mar 2026
osmarks.net
On Graphcore Read at the source ↗
23 Feb 2026
ternarysearch.blogspot.com
Triton Language Read at the source ↗
18 Feb 2026
benfattori.com
Implementing a 2:4 Sparse GEMM Kernel with Tensor Cores Read at the source ↗
14 Jan 2026
hamzaelshafie.bearblog.dev
Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) Read at the source ↗
12 Jan 2026
Henry Zhu
Maybe consider putting "cutlass" in your CUDA/Triton kernels Read at the source ↗
15 Dec 2025
aminediro.com
Reimplementing FlashAttention for performance and giggles Read at the source ↗
4 Dec 2025
kapilsh.github.io
Learn CUTLASS the hard way! Read at the source ↗
1 Nov 2025
am17an.bearblog.dev
A gentle introduction to GEMM using MMA tensor cores Read at the source ↗
2 Oct 2025
aleksagordic.com
Inside NVIDIA GPUs: Anatomy of high performance matmul kernels Read at the source ↗
29 Sept 2025
Modular
Modular: Matrix Multiplication on Blackwell: Part 1 - Introduction Read at the source ↗
28 Aug 2025
shubhamg.in
GPUs go brrr with Mojo: Algorithms Read at the source ↗
20 Jul 2025
Hugging Face
Creating custom kernels for the AMD MI300 Read at the source ↗
9 Jul 2025
jackyoustra.com
Road to petaflop Read at the source ↗
27 Jun 2025
Modal
'I paid for the whole GPU, I am going to use the whole GPU': A high-level guide to GPU utilization Read at the source ↗
24 Feb 2025
salykova.github.io
Advanced Matrix Multiplication Optimization on NVIDIA GPUs Read at the source ↗
12 Jan 2025
salykova.github.io
Advanced Matrix Multiplication Optimization on NVIDIA GPUs Read at the source ↗
12 Jan 2025
cudaforfun.substack.com
Outperforming cuBLAS on H100: a Worklog Read at the source ↗
29 Nov 2024
spatters.ca
Implementing a fast Tensor Core matmul on the Ada Architecture Read at the source ↗
15 Nov 2024
alexzhang13.github.io
A Meticulous Guide to Advances in Deep Learning Efficiency over the Years Read at the source ↗
30 Oct 2024
osmarks.net
Deep learning accelerator architecture Read at the source ↗
5 Oct 2024
michaelmoroz.github.io
Writing an optimizing tensor compiler from scratch Read at the source ↗
11 Sept 2024
wkaisertexas.github.io
Great Resources Read at the source ↗
19 Aug 2024
alexarmbr.github.io
How To Write A Fast Matrix Multiplication From Scratch With Tensor Cores Read at the source ↗
10 Aug 2024
kharshit.github.io
Matrix Multiplication in CUDA Read at the source ↗
7 Jun 2024
marknagelberg.com
Roam Research Notes on Dwarkesh Patel Conversation with Sholto Douglas & Trenton Bricken – How to Build & Understand GPT-7’s Mind Read at the source ↗
28 May 2024
khushi-411.github.io
Multidimensional Grid and Data Read at the source ↗
13 May 2024
seanzhang.me
A hitchhiker’s guide to CUDA programming Read at the source ↗
5 May 2024
chsasank.com
PortBLAS: Open Source BLAS that is Performance-Portable Across GPUs Read at the source ↗
22 Feb 2024
Suriya's site - Technical Writing & Life
to learn Read at the source ↗
—

Links from this post

Link
Host
https://github.com/siboehm/SGEMM_CUDA
github.com
https://github.com/wangzyon/NVIDIA_SGEMM_PRACTICE
github.com
https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html
docs.nvidia.com
PyTorch internals Read at the source ↗
blog.ezyang.com
https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html
docs.nvidia.com
https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms
en.wikipedia.org
https://developer.nvidia.com/blog/cuda-refresher-cuda-programming-model/
developer.nvidia.com
https://arxiv.org/abs/1804.06826
arxiv.org
https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html
docs.nvidia.com
https://www2.eecs.berkeley.edu/Pubs/TechRpts/2016/EECS-2016-143.pdf
www2.eecs.berkeley.edu
https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html
docs.nvidia.com
CUDA Pro Tip: Increase Performance with Vectorized Memory Access Read at the source ↗
developer.nvidia.com
https://github.com/NVIDIA/cutlass/blob/master/media/docs/efficient_gemm.md
github.com
https://github.com/openai/triton
github.com
Computers can be understood Read at the source ↗
blog.nelhage.com
https://gist.github.com/Chillee/f86675147366a7a0c6e244eaa78660f7
gist.github.com
https://horace.io/
horace.io
https://developer.nvidia.com/nsight-systems
developer.nvidia.com
https://developer.nvidia.com/blog/optimizing-compute-shaders-for-l2-locality-using-thread-group-id-swizzling/
developer.nvidia.com
https://github.com/siboehm/SGEMM_CUDA/commit/9a4dd77232d416649c13401c2382cdf95ded3760
github.com
https://github.com/NVIDIA/cutlass/blob/master/media/docs/implicit_gemm_convolution.md
github.com
New helper added: ExcaliDraw – Create hand-drawn-like diagrams.. Read at the source ↗
excalidraw.com
https://developer.nvidia.com/blog/cutlass-linear-algebra-cuda/
developer.nvidia.com
https://docs.nvidia.com/cuda/cuda-c-best-practices-guide
docs.nvidia.com
https://www.youtube.com/playlist?list=PL5Q2soXY2Zi-Mnk1PxjEIG32HAGILkTOF
youtube.com
https://www.youtube.com/playlist?list=PL5Q2soXY2Zi_OwkTgEyA6tk3UsoPBH737
youtube.com
https://developer.nvidia.com/blog/using-cuda-warp-level-primitives/
developer.nvidia.com
https://leimao.github.io/tags/CUDA/
leimao.github.io
https://leimao.github.io/blog/Proper-CUDA-Error-Checking/
leimao.github.io
https://docs.nvidia.com/cuda/cuda-binary-utilities/index.html
docs.nvidia.com
https://github.com/daadaada/turingas
github.com
https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/core/providers/cuda
github.com
https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/providers/cuda/math/softmax_warpwise_impl.cuh
github.com
https://github.com/NVIDIA/cutlass
github.com
https://github.dev/NVIDIA/cutlass/blob/master/include/cutlass/gemm/device/gemm.h
github.dev