corpus.blog
56,966 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a0877b-d96e-71dc-b69d-ce8b5beafcff

Writing Speed-of-Light Flash Attention for 5090 in CUDA C++

gau-nernst.github.io · published 23 August 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at gau-nernst.github.io Wayback capture
2blogs citing
First cited
22 Apr 2026
Most recent
23 Aug 2026
Rank this month
not ranked
Links from this post
32
Words captured
12,631

Cited by

Blog
In the post
Date
hoanh.space
Writing Fast Attention Backward Kernel for 5090 in CUDA Read at the source ↗
23 Aug 2026
kyrieblunders.bearblog.dev
Flash Attention 2 in CuteDSL: A Naive Kernel, Three Optimizations, and Where I Got Stuck Read at the source ↗
22 Apr 2026

Links from this post

Link
Host
https://triton-lang.org/main/index.html
triton-lang.org
Outperforming cuBLAS on H100: a Worklog Read at the source ↗
cudaforfun.substack.com
https://github.com/gpu-mode/lectures
github.com
https://www.youtube.com/@GPUMODE
youtube.com
https://github.com/gau-nernst/learn-cuda/tree/e83c256/07_attention
github.com
https://arxiv.org/abs/2307.08691
arxiv.org
https://arxiv.org/abs/2405.04434
arxiv.org
https://github.com/deepseek-ai/FlashMLA
github.com
https://github.com/gau-nernst/learn-cuda/blob/e83c256/07_attention/common.h
github.com
https://arxiv.org/abs/1805.02867
arxiv.org
https://pytorch.org/blog/flash-decoding/
pytorch.org
https://github.com/gau-nernst/learn-cuda/blob/e83c256/07_attention/attention_v1.cu
github.com
https://github.com/gau-nernst/learn-cuda/commit/8fdb3e6a
github.com
https://github.com/gau-nernst/learn-cuda/blob/e83c256/07_attention/attention.cpp
github.com
https://github.com/gau-nernst/learn-cuda/blob/e83c256/07_attention/main.py
github.com
https://github.com/Dao-AILab/flash-attention
github.com
https://docs.nvidia.com/nsight-compute/NsightComputeCli/index.html
docs.nvidia.com
https://developer.nvidia.com/nsight-compute
developer.nvidia.com
https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html
docs.nvidia.com
https://forums.developer.nvidia.com/t/ncu-detects-bank-conflicts-in-matrix-transposition-after-padding/239100/6
forums.developer.nvidia.com
https://github.com/gau-nernst/learn-cuda/blob/e83c256/07_attention/attention_v2.cu
github.com
https://github.com/NVIDIA/cutlass/blob/v4.1.0/media/docs/cpp/efficient_gemm.md
github.com
https://developer.nvidia.com/blog/controlling-data-movement-to-boost-performance-on-ampere-architecture/
developer.nvidia.com
https://triton-lang.org/main/python-api/generated/triton.Config.html
triton-lang.org
https://github.com/mingfeima
github.com
https://github.com/pytorch/pytorch/blob/v2.8.0/aten/src/ATen/native/cpu/int8mm_kernel.cpp
github.com
https://github.com/gau-nernst/learn-cuda/blob/e83c256/07_attention/attention_v3.cu
github.com
https://github.com/gau-nernst/learn-cuda/blob/e83c256/07_attention/attention_v4.cu
github.com
https://github.com/gau-nernst/learn-cuda/blob/e83c256/07_attention/attention_v5.cu
github.com
https://github.com/thu-ml/SageAttention
github.com
https://x.com/pranjalssh
x.com
https://arxiv.org/abs/2309.06180
arxiv.org