corpus.blog
41,554 blogs · 5,217,468 posts Claim your blog
Record · corpus.blog/posts/01a0fcf1-cb77-7268-85ba-f4e4564e1a3a

FlexAttention: The Flexibility of PyTorch with the Performance of FlashAttention

PyTorch Blog · published 7 August 2024

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at pytorch.org Wayback capture
10blogs citing
First cited
8 Jan 2022
Most recent
7 Sept 2026
Rank this month
not ranked
External links
20
Words captured
4,612

Cited by

Blog
In the post
Date
le.qun.ch
How Not to Build Another Megatron (Part 1): How Pipeline Parallelism Contaminates the Entire Training Framework Read at the source ↗
7 Sept 2026
ianbarber.blog
LLMs are complicated now Read at the source ↗
19 Jun 2026
accidentalfactors.com
LLMs are complicated now Read at the source ↗
19 Jun 2026
tylerromero.com
NanoGPT Speedrun Worklog Read at the source ↗
8 Mar 2025
Hugging Face
π0 and π0-FAST: Vision-Language-Action Models for General Robot Control Read at the source ↗
4 Feb 2025
njkumar.com
Implementing Character.AI’s memory optimizations in nanoGPT Read at the source ↗
20 Jan 2025
SemiAnalysis
MI300X vs H100 vs H200 Benchmark Part 1: Training – CUDA Moat Still Alive Read at the source ↗
22 Dec 2024
SemiAnalysis
Amazon’s AI Self Sufficiency | Trainium2 Architecture & Networking Read at the source ↗
3 Dec 2024
blog.ezyang.com
Ways to use torch.compile Read at the source ↗
5 Nov 2024
alexzhang13.github.io
A Meticulous Guide to Advances in Deep Learning Efficiency over the Years Read at the source ↗
30 Oct 2024
bodunhu.com
Machine Learning System Resources Read at the source ↗
8 Jan 2022

Links from this post

Link
Host
https://paperswithcode.com/method/relative-position-encodings
paperswithcode.com
https://paperswithcode.com/method/alibi
paperswithcode.com
Mistral 7B
mistral.ai
https://twitter.com/andersonbcdefg/status/1800907703688339569
twitter.com
https://github.com/pytorch/torchtune/pull/875
github.com
https://twitter.com/LysandreJik/status/1807779471891538199
twitter.com
https://arxiv.org/abs/2309.06180
arxiv.org
https://arxiv.org/abs/2108.12409
arxiv.org
https://twitter.com/jefrankle/status/1804567458092605736
twitter.com
Welcome Gemma 2 - Google’s new open LLM Read at the source ↗
huggingface.co
https://github.com/pytorch-labs/attention-gym/blob/main/attn_gym/mods/softcapping.py
github.com
https://github.com/pytorch/pytorch/blob/e49c0acc396e89baf8c6450e1fa0571d4ce2d4ed/torch/nn/attention/flex_attention.py
github.com
https://arxiv.org/abs/2310.06825
arxiv.org
https://arxiv.org/abs/2407.07726
arxiv.org
https://arxiv.org/abs/1910.10683
arxiv.org
https://github.com/pytorch-labs/attention-gym
github.com
https://github.com/pytorch/pytorch/blob/ee09d066d35d7e17cf7e9479c0b8bfc70cffc264/torch/_inductor/kernel/flex_attention.py
github.com
https://github.com/pytorch/pytorch/blob/main/benchmarks/transformer/score_mod.py
github.com
https://github.com/pytorch/torchtune/pull/1193
github.com
https://lmsys.org/blog/2024-01-17-sglang/
lmsys.org
Elsewhere on pytorch.org (1)