Record · corpus.blog/posts/01a0e418-ecbb-7160-9bcc-3e2ee74a4527

Outperforming cuBLAS on H100: a Worklog

cudaforfun.substack.com · published 29 November 2024

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at cudaforfun.substack.com Wayback capture

14blogs citing

First cited: 24 Feb 2025
Most recent: 23 Aug 2026
Rank this month: not ranked
Links from this post: 60
Words captured: 5,471

Cited by

Blog

In the post

Date

hoanh.space

Writing Fast Attention Backward Kernel for 5090 in CUDA Read at the source ↗

23 Aug 2026

Probably Dance | I can program and like games

If AI Writes All the Code, What Do the Programmers Do? Read at the source ↗

28 Jul 2026

sankalp's blog

Auto-research with codex: How I achieved a 232x Faster Kernel over baseline with Codex in GPU Mode's qr_v2 problem Read at the source ↗

8 Jul 2026

hamzaelshafie.bearblog.dev

Dissecting ThunderKittens: Anatomy of a Compact DSL for High-Performance AI Kernels Read at the source ↗

21 May 2026

danielvegamyhre.github.io

MXFP8 GEMM: Up to 99% of cuBLAS performance using CUDA + PTX Read at the source ↗

29 Mar 2026

Probably Dance | I can program and like games

How LLMs Keep on Getting Better Read at the source ↗

31 Jan 2026

hamzaelshafie.bearblog.dev

Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) Read at the source ↗

12 Jan 2026

kapilsh.github.io

Learn CUTLASS the hard way - part 2! Read at the source ↗

31 Dec 2025

gau-nernst.github.io

tcgen05 for dummies Read at the source ↗

21 Dec 2025

kapilsh.github.io

Learn CUTLASS the hard way! Read at the source ↗

1 Nov 2025

am17an.bearblog.dev

A gentle introduction to GEMM using MMA tensor cores Read at the source ↗

2 Oct 2025

aleksagordic.com

Inside NVIDIA GPUs: Anatomy of high performance matmul kernels Read at the source ↗

29 Sept 2025

Modular

Modular: Matrix Multiplication on Blackwell: Part 1 - Introduction Read at the source ↗

28 Aug 2025

gau-nernst.github.io

Writing Speed-of-Light Flash Attention for 5090 in CUDA C++ Read at the source ↗

23 Aug 2025

Hugging Face

Creating custom kernels for the AMD MI300 Read at the source ↗

9 Jul 2025

Subho's research at your service 🫡

Optimizing 3D Square Convolution for cuDNN-like Performance - A Worklog Read at the source ↗

18 Apr 2025

Modal

'I paid for the whole GPU, I am going to use the whole GPU': A high-level guide to GPU utilization Read at the source ↗

24 Feb 2025

Suriya's site - Technical Writing & Life

to learn Read at the source ↗

—

Links from this post

Link

Host

How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog Read at the source ↗

siboehm.com

https://github.com/pranjalssh/fast.cu

github.com

https://substackcdn.com/image/fetch/$s_!Pgrd!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f7e656f-12ff-488d-8656-4297b51b0340_1832x1274.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!Wx-M!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F533fcdb6-536d-4671-898c-e8386450967e_1350x504.png

substackcdn.com

https://en.wikipedia.org/wiki/Bfloat16_floating-point_format

en.wikipedia.org

https://en.wikipedia.org/wiki/Half-precision_floating-point_format

en.wikipedia.org

Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data! [short] Read at the source ↗

thonking.ai

https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/

developer.nvidia.com

https://substackcdn.com/image/fetch/$s_!Tc0J!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d24e54f-0a25-41dc-a6ad-f9806177af7c_1839x1385.png

substackcdn.com

https://docs.nvidia.com/cuda/cuda-c-programming-guide/

docs.nvidia.com

https://docs.nvidia.com/cuda/parallel-thread-execution/

docs.nvidia.com

https://substackcdn.com/image/fetch/$s_!gYKQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19103715-7af2-4c68-93c8-c16693bd5f07_1782x728.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!JYyy!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F593b0399-e841-418c-9687-83aa563a4809_1565x504.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!avtd!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffefdf009-0b38-4bf8-9b64-028897be9b3e_2425x2184.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!QGz4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac84473d-aec0-403a-a109-6d37d9e21fe0_1280x567.png

substackcdn.com

https://hazyresearch.stanford.edu/blog/2024-05-12-tk

hazyresearch.stanford.edu

https://substackcdn.com/image/fetch/$s_!0R_G!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F635bcf9c-9fad-48e5-a69a-96ccfbe3c62a_1782x751.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!80G_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e763709-1bf9-488b-b581-9ba4127040b8_2497x2940.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!wtV-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf13b61d-babe-4a6a-90b5-806027c97548_2175x2352.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!vBMO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2b0a82a-528a-4d2a-8843-4a46acb3f5e8_3142x1512.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!a2I3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c736df7-62e1-4b9b-9bb3-5063216404c2_1324x702.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!FgzJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3348c876-fe22-415d-831e-4d3fc45b5f7e_1208x672.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!gAPU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0e4a133-5ea9-4f16-9de0-435b21235c1c_1423x672.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!hzAw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3992ff21-dd63-42e8-9137-b489d81c9a14_1459x572.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!u8wy!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faaa96b2c-43c7-4c5e-bf01-c1b7db6d1137_2103x1428.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!gDRT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F300b3895-df2d-4c10-8fce-d37be2aefcc7_3178x1344.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!LW0n!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f67d8fc-122f-48e9-a308-3e890c4a96f3_2928x1848.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!tPyL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe08eb127-9cb5-4331-b499-21c753d77ed9_1654x2996.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!-hnA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F392c1c0b-10a2-4f7b-b5f1-7a55fadb7501_2507x1818.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!xpgU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e0a7e77-cd92-4d68-9fe2-9dec83173ecd_1852x1008.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!le3r!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd89bda4-d80c-4647-9043-5abf9f832796_2140x588.png

substackcdn.com

https://forums.developer.nvidia.com/t/about-register-bank-conflict/47853

forums.developer.nvidia.com

https://substackcdn.com/image/fetch/$s_!90j7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cfb2c4a-3f01-4e14-baf9-27e47c1bc2fa_2591x850.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!l0IW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0311ade-8fda-475a-95fb-f77f73457bc6_1852x420.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!k15u!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c275787-f8ad-4125-9f47-e2397da60dcf_1442x961.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!M2_E!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91eb8f70-54ad-4b33-8626-7f909ce28e2d_1442x961.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!rIbe!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b9b304a-d073-4ddf-9a65-44427d945893_1442x961.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!gtv6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff7e14e76-c45f-4369-91b9-6d9e5984b7c7_2282x1176.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!rqvq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa83a72a8-329a-49dd-8af7-d0c2efc0a950_1024x565.jpeg

substackcdn.com

https://substackcdn.com/image/fetch/$s_!4ZMU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F720e3962-f41c-4438-87b9-6e22e55c0066_2246x1512.png

substackcdn.com

https://github.com/pranjalssh/fast.cu/blob/main/examples/matmul/matmul_11.cuh

github.com

https://substackcdn.com/image/fetch/$s_!PeOC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14a18d14-bf32-4af8-8887-cd6e271acf66_2199x1818.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!Etph!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdfbfa663-d1a0-40fa-8582-4eaad752475a_2031x1848.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!Yepf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb36d349-7418-4fb8-84bc-0cf674df1b6e_2318x252.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!Py8j!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a8670af-d84a-4057-a159-faf5aea1e09a_1028x252.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!Me9Y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff317fada-078c-4e87-98a1-719c72ca974a_2999x1848.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!FgyP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff23ba532-b7e1-4bb6-8602-bc063e0f5651_1442x961.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!u7SN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f2a2d6b-3436-4d15-9806-2d24318f8947_1442x961.png

substackcdn.com

https://en.wikipedia.org/wiki/Hilbert_curve

en.wikipedia.org

https://substackcdn.com/image/fetch/$s_!czMS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb080b4e8-beaa-46c3-88bc-2072494f9672_200x200.png

substackcdn.com

NVIDIA Sets New Generative AI Performance and Scale Records in MLPerf Training v4.0 Read at the source ↗

developer.nvidia.com

https://www.youtube.com/watch?v=4pkbXmE4POc&list=PLRRuQYjFhpmubuwx-w8X964ofVkW1T8O4

youtube.com

https://discord.com/invite/gpumode

discord.com

https://www.advancedclustering.com/wp-content/uploads/2022/03/gtc22-whitepaper-hopper.pdf

advancedclustering.com

https://github.com/NVIDIA/cutlass/blob/main/media/docs/efficient_gemm.md

github.com

https://arxiv.org/pdf/2407.08608

arxiv.org

https://substackcdn.com/image/fetch/$s_!iyuv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff274bf2b-7ed4-403d-965f-c846d38e9b05_2784x924.png

substackcdn.com

https://substackcdn.com/image/fetch/$s_!Ak1j!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e03755d-8ef7-46c5-a0be-c5eb5f52b0b3_2461x1092.png

substackcdn.com

https://resources.nvidia.com/en-us-blackwell-architecture

resources.nvidia.com

https://www.nvidia.com/en-us/on-demand/session/gtcspring21-s33322/

nvidia.com