corpus.blog
41554 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a08767-697e-716d-9a9d-e28f417f444b

An illustrated deep-dive into how the compute and comms in TP+SP are overlapped using Async TP

danielvegamyhre.github.io · published 26 May 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at danielvegamyhre.github.io Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
Links from this post
12
Words captured
6,306

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://dl.acm.org/doi/pdf/10.1145/3567955.3567959
dl.acm.org
https://arxiv.org/pdf/2205.05198
arxiv.org
https://github.com/pytorch/torchtitan
github.com
https://discuss.pytorch.org/t/distributed-w-torchtitan-introducing-async-tensor-parallelism-in-pytorch/209487
discuss.pytorch.org
https://dev-discuss.pytorch.org/t/pytorch-symmetricmemory-harnessing-nvlink-programmability-with-ease/2798
dev-discuss.pytorch.org
https://github.com/pytorch/pytorch/blob/e89ee1e21770bc281bb53d01bcbb71806134c6db/torch/distributed/_symmetric_memory/__init__.py
github.com
https://dev-discuss.pytorch.org/t/torchinductor-a-pytorch-native-compiler-with-define-by-run-ir-and-symbolic-shapes/747
dev-discuss.pytorch.org
https://github.com/pytorch/pytorch/blob/54827752a4287d6a9ec266d21396e07dc14b0ddc/torch/_inductor/fx_passes/micro_pipeline_tp.py
github.com
https://github.com/pytorch/torchtitan/blob/29a67ecff3c9330175d0154f467e15694df56a2c/torchtitan/models/llama3/parallelize_llama.py
github.com
https://blog.shi-labs.com/distributed-gemm-88be6a481e2b
blog.shi-labs.com
https://www.youtube.com/watch?v=9o2TXexHUh8
youtube.com
https://old.hotchips.org/hc30/2conf/2.01_Nvidia_NVswitch_HotChips2018_DGX2NVS_Final.pdf
old.hotchips.org