Record · corpus.blog/posts/01a07d5d-72e7-7112-bd07-c154cb87306a

Profiling in PyTorch (Part 3): Attention is all you profile

Hugging Face · published 10 July 2026

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at huggingface.co Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
Links from this post: 28
Words captured: 4,096

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/torch-attention-profile/profile-3-thumbnail.png

huggingface.co

Profiling in PyTorch (Part 1): A Beginner's Guide to torch.profiler Read at the source ↗

huggingface.co

Profiling in PyTorch (Part 2): From nn.Linear to a Fused MLP Read at the source ↗

huggingface.co

https://huggingface.co/datasets/ariG23498/profiling-pytorch/blob/main/04_a_naive_attention.py

huggingface.co

https://huggingface.co/datasets/ariG23498/profiling-pytorch/blob/main/04_b_inplace_ops_attention.py

huggingface.co

https://huggingface.co/datasets/ariG23498/profiling-pytorch/blob/main/04_c_sdpa_attention.py

huggingface.co

https://huggingface.co/datasets/ariG23498/profiling-pytorch/blob/main/04_d_kernels_attention.py

huggingface.co

https://huggingface.co/docs/hub/spaces-dev-mode

huggingface.co

https://huggingface.co/docs/huggingface_hub/en/guides/jobs

huggingface.co

https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/torch-attention-profile/cpu-profile-naive.png

huggingface.co

https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/torch-attention-profile/gpu-profile-naive.png

huggingface.co

https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/torch-attention-profile/each-kernels-naive.png

huggingface.co

https://docs.pytorch.org/docs/2.13/generated/torch.Tensor.masked_fill.html

docs.pytorch.org

https://docs.pytorch.org/tutorials/intermediate/scaled_dot_product_attention_tutorial.html

docs.pytorch.org

https://www.nvidia.com/en-us/data-center/tensor-cores/

nvidia.com

https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/torch-attention-profile/mask-math.png

huggingface.co

https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/torch-attention-profile/safe-softmax-extra-kernels.png

huggingface.co

https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/torch-attention-profile/efficient-backend.png

huggingface.co

https://github.com/facebookresearch/xformers

github.com

https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/torch-attention-profile/flash-backend.png

huggingface.co

https://arxiv.org/abs/2307.08691

arxiv.org

https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/torch-attention-profile/flash-occupancy.png

huggingface.co

Creating custom kernels for the AMD MI300 Read at the source ↗

huggingface.co

https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/torch-attention-profile/flash-reg-count.png

huggingface.co

https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/torch-attention-profile/cudnn-backend.png

huggingface.co

https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/torch-attention-profile/cudnn-launch.png

huggingface.co

https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/torch-attention-profile/cudnn-footprint.png

huggingface.co

https://huggingface.co/NoeFlandre

huggingface.co