corpus.blog
41554 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a07d5d-86e9-710f-b9c2-b454972f4fc6

We reverse-engineered Flash Attention 4

Modal · published 26 September 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at modal.com Wayback capture
1blogs citing
First cited
8 Nov 2025
Most recent
8 Nov 2025
Rank this month
not ranked
Links from this post
75
Words captured
4,135

Cited by

Blog
In the post
Date
lubits.ch
Flash Attention from Scratch: Appendix B - Block Size Configuration Read at the source ↗
8 Nov 2025

Links from this post

Link
Host
https://news.ycombinator.com/item?id=45399637
news.ycombinator.com
https://discord.com/invite/gpumode
discord.com
https://www.youtube.com/watch?v=ZIEq-WTquy4
youtube.com
https://hotchips.org/
hotchips.org
https://github.com/Dao-AILab/flash-attention
github.com
Introducing: B200s and H200s on Modal Read at the source ↗
modal.com
https://modal.com/gpu-glossary/host-software/cudnn
modal.com
https://github.com/Dao-AILab/flash-attention/blob/5c1627a7a1cda9c32cb9b937a053564e663f81bc/flash_attn/cute/flash_fwd_sm100.py
github.com
https://github.com/sgl-project/sglang/pull/9588
github.com
https://assets.bitbashing.io/images/cubedrone-103.png
assets.bitbashing.io
https://modal.com/gpu-glossary/readme
modal.com
https://modal.com/gpu-glossary/device-software/warp
modal.com
https://modal.com/gpu-glossary/device-hardware/warp-scheduler
modal.com
https://float.exposed/b0x3a0a
float.exposed
https://modal.com/gpu-glossary/device-software/global-memory
modal.com
https://modal.com/gpu-glossary/device-hardware/gpu-ram
modal.com
https://modal.com/gpu-glossary/device-software/cooperative-thread-array
modal.com
https://modal.com/gpu-glossary/device-software/kernel
modal.com
https://modal.com/gpu-glossary/device-software/cuda-programming-model
modal.com
https://modal.com/gpu-glossary/host-software/cuda-software-platform
modal.com
https://blog.codingconfessions.com/p/simultaneous-multithreading
blog.codingconfessions.com
https://modal.com/gpu-glossary/perf
modal.com
https://modal.com/gpu-glossary/perf/issue-efficiency
modal.com
https://modal.com/gpu-glossary/perf/warp-execution-state
modal.com
Learning Async Rust With Entirely Too Many Web Servers Read at the source ↗
ibraheem.ca
Async/Await Read at the source ↗
os.phil-opp.com
https://groups.google.com/a/chromium.org/g/blink-dev/c/AK_rwEp61ME
groups.google.com
https://modal.com/gpu-glossary/device-software/memory-hierarchy
modal.com
https://microservices.io/patterns/microservices.html
microservices.io
https://modal.com/gpu-glossary/device-software/shared-memory
modal.com
https://modal.com/gpu-glossary/device-hardware/tensor-core
modal.com
https://modal.com/gpu-glossary/device-hardware/tensor-memory
modal.com
https://modal.com/gpu-glossary/device-hardware/special-function-unit
modal.com
https://modal.com/gpu-glossary/device-hardware/streaming-multiprocessor
modal.com
https://nic.schraudolph.org/pubs/Schraudolph99.pdf
nic.schraudolph.org
https://graphics.stanford.edu/papers/brookgpu/brookgpu.pdf
graphics.stanford.edu
https://modal.com/gpu-glossary/host-software/cuda-c
modal.com
https://www.cs.cmu.edu/afs/cs/academic/class/15869-f11/www/readings/lindholm08_tesla.pdf
cs.cmu.edu
https://www.together.ai/blog/flashattention-3
together.ai
Async Rust Is A Bad Language Read at the source ↗
bitbashing.io
Linklog | What Color is Your Function? Read at the source ↗
journal.stuffwithstuff.com
https://fasterthanli.me/articles/pin-and-suffering
fasterthanli.me
https://modal.com/gpu-glossary/device-software/parallel-thread-execution
modal.com
https://www.youtube.com/watch?v=5e1YKqsP8i8&t=1059s
youtube.com
https://docs.nvidia.com/cutlass/media/docs/pythonDSL/cute_dsl_general/dsl_introduction.html
docs.nvidia.com
https://docs.nvidia.com/cutlass/index.html
docs.nvidia.com
https://www.youtube.com/watch?v=uZTtViomW6w
youtube.com
https://github.com/Dao-AILab/flash-attention/blob/1ceaa984b2f348caea18b39a98458d33b4ea7a09/flash_attn/cute/tile_scheduler.py
github.com
https://arxiv.org/abs/2309.06180
arxiv.org
https://modal.com/gpu-glossary/device-hardware/tensor-memory-accelerator
modal.com
https://modal.com/gpu-glossary/perf/latency-hiding
modal.com
https://github.com/Dao-AILab/flash-attention/blob/1ceaa984b2f348caea18b39a98458d33b4ea7a09/flash_attn/cute/blackwell_helpers.py
github.com
https://modal.com/gpu-glossary/device-software/compute-capability
modal.com
https://modal.com/gpu-glossary/device-hardware/texture-processing-cluster
modal.com
https://docs.nvidia.com/cutlass/media/docs/cpp/blackwell_functionality.html
docs.nvidia.com
https://github.com/HazyResearch/ThunderKittens/blob/2ba96ceedfb1b5c5d6e1eb4a1241a24d16049be4/kernels/attn/b200/b200.cu
github.com
https://modal.com/gpu-glossary/device-software/warpgroup
modal.com
https://transformer-circuits.pub/
transformer-circuits.pub
https://modal.com/gpu-glossary/device-hardware/streaming-multiprocessor-architecture
modal.com
https://en.wikipedia.org/wiki/Blackwell_(microarchitecture
en.wikipedia.org
https://images.nvidia.com/aem-dam/Solutions/geforce/blackwell/nvidia-rtx-blackwell-gpu-architecture.pdf
images.nvidia.com
https://godbolt.org/z/7e5jx9qcr
godbolt.org
https://github.com/Dao-AILab/flash-attention/blob/add175637c5d54b74bc25372e49ce282d6f236fc/flash_attn/cute/flash_fwd_sm100.py
github.com
https://github.com/Dao-AILab/flash-attention/blob/0165c96fff7a7cd2e152aa9659f75c972a702f5d/flash_attn/cute/softmax.py
github.com
https://github.com/Dao-AILab/flash-attention/blob/add175637c5d54b74bc25372e49ce282d6f236fc/flash_attn/cute/utils.py
github.com
https://www.wolframalpha.com/input?i=0.07711909*r%5E3%2B0.22756439*r%5E2%2B0.69514614*r%2B1.0+compared+to+2%5Er
wolframalpha.com
https://people.eecs.berkeley.edu/~pattrsn/252S98/Lec06-vector.pdf
people.eecs.berkeley.edu
https://stackoverflow.com/questions/47025373/fastest-implementation-of-the-natural-exponential-function-using-sse
stackoverflow.com
What Shapes Do Matrix Multiplications Like? [medium] Read at the source ↗
thonking.ai
https://en.wikipedia.org/wiki/Softmax_function
en.wikipedia.org
How Suno shaved 4 months off their launch timeline with Modal Read at the source ↗
modal.com
How Modal powered 250,000 Lovable app creations in a weekend Read at the source ↗
modal.com
How Ramp automated receipt processing with fine-tuned LLMs Read at the source ↗
modal.com
Why Substack moved their AI and ML pipelines to Modal Read at the source ↗
modal.com
https://modal.com/careers
modal.com