Record · corpus.blog/posts/01a08793-a866-73bd-89f5-9e5ba1f57a4e

Learn CUTLASS the hard way!

kapilsh.github.io · published 1 November 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at kapilsh.github.io Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
Links from this post: 113
Words captured: 29,809

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

https://www.amazon.com/Programming-Massively-Parallel-Processors-Hands/dp/0323912311

amazon.com

How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog Read at the source ↗

siboehm.com

Outperforming cuBLAS on H100: a Worklog Read at the source ↗

cudaforfun.substack.com

Inside NVIDIA GPUs: Anatomy of high performance matmul kernels Read at the source ↗

aleksagordic.com

https://github.com/gpusgobrr/explore-gemm

github.com

http://www.kapilsharma.dev/assets/4090_sm.png

kapilsharma.dev

https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvidia-ada-gpu-architecture.pdf

images.nvidia.com

https://en.wikipedia.org/wiki/Roofline_model

en.wikipedia.org

http://www.kapilsharma.dev/assets/explore_gemm_naive_only.png

kapilsharma.dev

https://www.nvidia.com/en-us/on-demand/session/gtcfall22-a41101/

nvidia.com

http://www.kapilsharma.dev/assets/explore_gemm_global_coalesced.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_shared_mem.png

kapilsharma.dev

https://modal.com/gpu-glossary/perf/occupancy

modal.com

http://www.kapilsharma.dev/assets/explore_gemms_occupancy_modal.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_shared_mem_occupancy.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_summary_shared_mem_ncu.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_lds_too_much.png

kapilsharma.dev

https://github.com/siboehm/SGEMM_CUDA/blob/master/src/kernels/4_kernel_1D_blocktiling.cuh

github.com

http://www.kapilsharma.dev/assets/explore_gemm_id_blocktiling_performance.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_1d_tiling_ncu_1.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_1d_tiling_ncu_2.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_1d_smem_ncu_memory.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_1d_block_tiling_ncu_memory.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_2d_blocktiling_performance.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_ncu_after_2d_tiling.png

kapilsharma.dev

CUDA Pro Tip: Increase Performance with Vectorized Memory Access Read at the source ↗

developer.nvidia.com

http://www.kapilsharma.dev/assets/explore_gemm_performance_vectorized.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_cutlass_hierarchy.png

kapilsharma.dev

https://developer.nvidia.com/blog/cutlass-linear-algebra-cuda/

developer.nvidia.com

http://www.kapilsharma.dev/assets/explore_gemms_warp_tiling_nvidia_blog_2.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemms_thread_tiling_nvidia_blog.png

kapilsharma.dev

https://docs.nvidia.com/cutlass/media/docs/cpp/efficient_gemm.html

docs.nvidia.com

https://moderngpu.github.io/performance.html

moderngpu.github.io

http://www.kapilsharma.dev/assets/explore_gemm_warptiling_performance.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_fp16_baseline.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_bf16_baseline.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemms_wmma_nvidia_blog.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemms_tensorcore_nvidia_gtc_presentation_1.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemms_tensorcore_instruction_sm80.png

kapilsharma.dev

https://www.nvidia.com/en-us/on-demand/session/gtcsj20-s21745/

nvidia.com

http://www.kapilsharma.dev/assets/explore_gemms_wmma_nvidia_blog_2.png

kapilsharma.dev

https://github.com/gpusgobrr/explore-gemm/blob/main/cuda/09_kernel_tensorcore_naive.cu

github.com

http://www.kapilsharma.dev/assets/explore_gemm_tensorcore_warptiled.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_tensorcore_warptiled_bf16.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_regular_warptiled_ncu.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_tensorcore_warptiled_ncu.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemms_pytorch_matmul_ncu.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemms_after_removing_the_allocation.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemm_torch_empty_experiment.png

kapilsharma.dev

https://forums.developer.nvidia.com/t/what-is-double-buffering/13531

forums.developer.nvidia.com

http://www.kapilsharma.dev/assets/explore_gemms_double_buffer_user_answer.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/software-pipelining.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemms_tensorcore_double_buffered.png.png

kapilsharma.dev

https://nvidia.github.io/cccl/libcudacxx/extended_api/synchronization_primitives/pipeline.html

nvidia.github.io

https://docs.nvidia.com/cutlass/media/docs/cpp/quickstart.html

docs.nvidia.com

https://media1.giphy.com/media/v1.Y2lkPTc5MGI3NjExY3J6a2J6emxpeTMxZnN5N3N5YmI1MGhrNzRrbnRhZWcyYWp4NjhhdyZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/BY8ORoRpnJDXeBNwxg/giphy.gif

media1.giphy.com

https://github.com/gpu-mode

github.com

http://www.kapilsharma.dev/assets/explore_gemms_cutlass_performance_1.png

kapilsharma.dev

https://ipd.graylab.jhu.edu/rfdiffusion2/cutlass-3.5.1/docs/classcutlass_1_1gemm_1_1device_1_1Gemm.html

ipd.graylab.jhu.edu

https://modal.com/gpu-glossary/perf/bank-conflict

modal.com

http://www.kapilsharma.dev/assets/explore_gemms_shared_memory_bank_conflict.png

kapilsharma.dev

Triton Linear Layout: Concept Read at the source ↗

lei.chat

http://www.kapilsharma.dev/assets/explore_gemms_swizzling_linear_layout_lei_chat.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemms_swizzling_alternate_layout_lei_chat.png

kapilsharma.dev

https://docs.nvidia.com/cuda/parallel-thread-execution/index.html

docs.nvidia.com

http://www.kapilsharma.dev/assets/explore_gemms_k_major_swizzling_nvidia_docs.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemms_mn_major_swizzling_nvidia_docs.png

kapilsharma.dev

CUTLASS Tutorial: Fast Matrix-Multiplication with WGMMA on NVIDIA® Hopper™ GPUs Read at the source ↗

research.colfax-intl.com

CUTLASS Tutorial: Efficient GEMM kernel designs with Pipelining Read at the source ↗

research.colfax-intl.com

http://www.kapilsharma.dev/assets/explore_gemms_software_pipeling_colfax.png

kapilsharma.dev

https://www.nvidia.com/en-us/on-demand/session/gtc25-s72876/

nvidia.com

https://www.youtube.com/watch?v=GL7ImGZj-Oc

youtube.com

https://semianalysis.com/wp-content/uploads/2025/03/Blackwell-Programming-for-the-Masses-With-OpenAI-Triton-Phil-Tillet.pdf

semianalysis.com

http://www.kapilsharma.dev/assets/explore_gemms_phil_gtc_1.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemms_phil_gtc_2.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemms_persistent_kernel_phil_gtc.png

kapilsharma.dev

http://www.kapilsharma.dev/assets/explore_gemms_autotuning_results.png

kapilsharma.dev

https://triton-lang.org/main/getting-started/tutorials/09-persistent-matmul.html

triton-lang.org

http://www.kapilsharma.dev/assets/explore_gemms_autotune_results_with_triton.png

kapilsharma.dev

https://github.com/NVIDIA/cutlass/issues/1337

github.com

https://github.com/siboehm/SGEMM_CUDA/tree/master

github.com

https://docs.nvidia.com/cuda/cuda-c-programming-guide/

docs.nvidia.com

https://github.com/NVIDIA/cutlass

github.com

https://triton-lang.org/main/getting-started/tutorials/03-matrix-multiplication.html

triton-lang.org

https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/

docs.nvidia.com

https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html

docs.nvidia.com

https://www.reddit.com/r/CUDA/comments/1hk4410/whats_the_point_of_warplevel_gemm/

reddit.com

https://arxiv.org/pdf/1808.07984

arxiv.org

https://leimao.github.io/blog/CUDA-Shared-Memory-Bank/

leimao.github.io

Advanced Matrix Multiplication Optimization on NVIDIA GPUs Read at the source ↗

salykova.github.io

https://developer.nvidia.com/blog/how-access-global-memory-efficiently-cuda-c-kernels/

developer.nvidia.com

https://modal.com/gpu-glossary/readme

modal.com

https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9593-cutensor-high-performance-tensor-operations-in-cuda-v2.pdf

developer.download.nvidia.com

https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html

docs.nvidia.com

https://www.nvidia.com/en-us/on-demand/session/gtc25-s72683/

nvidia.com

CUTLASS Tutorial: Persistent Kernels and Stream-K Read at the source ↗

research.colfax-intl.com

https://newsletter.semianalysis.com/p/nvidia-tensor-core-evolution-from-volta-to-blackwell

newsletter.semianalysis.com

https://arxiv.org/pdf/2208.11174

arxiv.org

https://developer.nvidia.com/blog/programming-tensor-cores-cuda-9/

developer.nvidia.com

https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html

docs.nvidia.com

https://github.com/Bruce-Lee-LY/cuda_hgemm/tree/master

github.com

https://www.youtube.com/watch?v=hQ9GPnV0-50

youtube.com

Tutorial: Matrix Transpose in CUTLASS Read at the source ↗

research.colfax-intl.com

https://www.nvidia.com/en-us/on-demand/session/gtc24-s62191/

nvidia.com

https://docs.google.com/presentation/d/1cvVpf3ChFFiY4Kf25S4e4sPY6Y5uRUO-X-A4nJ7IhFE/edit?slide=id.g2bd7b45cc89_0_59

docs.google.com

CUTLASS Tutorial: Writing GEMM Kernels Using Tensor Memory For NVIDIA® Blackwell GPUs Read at the source ↗

research.colfax-intl.com

https://leimao.github.io/blog/CUDA-Shared-Memory-Swizzling/

leimao.github.io

https://veitner.bearblog.dev/understanding-cute-swizzling-the-math-behind-32b-64b-and-128b-patterns/

veitner.bearblog.dev

https://www.nvidia.com/en-us/on-demand/session/gtc24-s62192/

nvidia.com

https://feldmann.nyc/blog/smem-microbenchmarks

feldmann.nyc

Boosting CUDA Efficiency with Essential Techniques for New Developers Read at the source ↗

developer.nvidia.com

https://www.nvidia.com/en-us/on-demand/session/gtcspring23-s51413/

nvidia.com

https://www.nvidia.com/en-us/on-demand/session/gtc25-s72685/

nvidia.com