Record · corpus.blog/posts/01a0879e-aebc-7209-be67-74f194091886

Visualizing 6D Mesh Parallelism

main-horse.github.io · published 14 December 2024

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at main-horse.github.io Wayback capture

1blogs citing

First cited: 30 Aug 2025
Most recent: 30 Aug 2025
Rank this month: not ranked
Links from this post: 55
Words captured: 4,174

Cited by

Blog

In the post

Date

blog.ezyang.com

The Parallelism Mesh Zoo Read at the source ↗

30 Aug 2025

Links from this post

Link

Host

https://main-horse.github.io/posts/TODO

main-horse.github.io

https://en.wikipedia.org/wiki/Collective_operation

en.wikipedia.org

https://main-horse.github.io/posts/xai.png

main-horse.github.io

https://dev-discuss.pytorch.org/t/fsdp-cudacachingallocator-an-outsider-newb-perspective/1486/1

dev-discuss.pytorch.org

https://peterchng.com/blog/2024/08/19/ring-attention-scaling-attention-across-multiple-devices/

peterchng.com

https://www.determined.ai/blog/tp

determined.ai

How to Train Really Large Models on Many GPUs? Read at the source ↗

lilianweng.github.io

https://arxiv.org/pdf/2006.15704

arxiv.org

https://main-horse.github.io/translations/double_binary_tree/

main-horse.github.io

https://arxiv.org/pdf/2411.00284

arxiv.org

https://github.com/pytorch/pytorch/blob/d3d1a78774e5d5578afb8c2ae8150afd665a7a36/torch/nn/parallel/distributed.py

github.com

https://deepspeed.readthedocs.io/en/latest/zero3.html

deepspeed.readthedocs.io

https://github.com/pytorch/pytorch/blob/539c46b6e82e16d22b745d8b320ae0bf9b1df90b/torch/distributed/_functional_collectives.py

github.com

https://arxiv.org/pdf/2406.04093

arxiv.org

Fully Sharded Data Parallel: faster AI training with fewer GPUs Read at the source ↗

engineering.fb.com

https://arxiv.org/pdf/2411.13055

arxiv.org

https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html

pytorch.org

https://deepspeed.readthedocs.io/en/latest/

deepspeed.readthedocs.io

https://github.com/pytorch/pytorch/tree/main/torch/distributed/fsdp

github.com

https://x.com/main_horse/status/1868521643339087979

x.com

https://www.microsoft.com/en-us/research/blog/ZeRO-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/

microsoft.com

https://github.com/NVIDIA/apex/blob/master/apex/contrib/optimizers/distributed_fused_adam.py

github.com

https://gist.github.com/main-horse/bae3a07643e204a48b926870deeef804

gist.github.com

https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/scaling/JAX/data_parallel_fsdp.html

uvadlc-notebooks.readthedocs.io

https://github.com/yandex/YaFSDP

github.com

https://arxiv.org/abs/2411.00284

arxiv.org

https://github.com/pytorch/torchtitan/blob/main/docs/fsdp.md

github.com

https://github.com/MatX-inc/seqax?tab=readme-ov-file

github.com

https://pytorch.org/docs/stable/fsdp.html

pytorch.org

https://arxiv.org/pdf/2310.07096

arxiv.org

https://robotchinwag.com/posts/demystifying-tensor-parallelism/

robotchinwag.com

https://www.servethehome.com/dual-nvidia-geforce-rtx-3090-nvlink-performance-review-asus-zotac/

servethehome.com

https://arxiv.org/pdf/2205.05198

arxiv.org

https://discuss.pytorch.org/t/distributed-w-torchtitan-introducing-async-tensor-parallelism-in-pytorch/209487

discuss.pytorch.org

https://arxiv.org/pdf/2406.06858

arxiv.org

https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html

pytorch.org

https://arxiv.org/pdf/2308.00951

arxiv.org

https://arxiv.org/pdf/2407.21783

arxiv.org

https://github.com/pytorch/pytorch/blob/2cc01cc6d3ad2aff47e8460667ba654b2e4c9f21/torch/distributed/tensor/experimental/_attention.py

github.com

https://github.com/zhuzilin/ring-flash-attention/blob/main/ring_flash_attn/utils.py

github.com

https://github.com/pytorch/pytorch/blob/539c46b6e82e16d22b745d8b320ae0bf9b1df90b/torch/distributed/tensor/experimental/_attention.py

github.com

https://pytorch.org/docs/stable/distributed.html

pytorch.org

https://github.com/pytorch/torchtitan/blob/03f17e844e0ba36fc5adf1c1e107f094b785d8f8/torchtitan/utils.py

github.com

Mixture of Experts Explained Read at the source ↗

huggingface.co

https://arxiv.org/pdf/2303.06318

arxiv.org

https://youtu.be/1yvBqasHLZs?t=317

youtu.be

https://arxiv.org/abs/2401.10241

arxiv.org

https://pytorch.org/docs/stable/distributed.pipelining.html

pytorch.org

https://developer.nvidia.com/blog/doubling-all2all-performance-with-nvidia-collective-communication-library-2-12/

developer.nvidia.com

100,000 H100 Clusters: Power, Network Topology, Ethernet vs InfiniBand, Reliability, Failures, Checkpointing Read at the source ↗

semianalysis.com

https://muellerzr.github.io/blog/gradient_accumulation.html

muellerzr.github.io

https://github.com/pytorch/pytorch/blob/d745b2b5163e821585d0701ebaf16a38f4d57eab/torch/distributed/pipelining/stage.py

github.com

https://github.com/pytorch/pytorch/blob/d745b2b5163e821585d0701ebaf16a38f4d57eab/torch/distributed/pipelining/schedules.py

github.com

https://main-horse.github.io/parallel-comm-viz/

main-horse.github.io

https://x.com/cHHillee/status/1868870350785871873

x.com