corpus.blog
43,228 blogs · 5,519,570 posts Claim your blog
Record · corpus.blog/posts/01a0c525-d410-7214-990f-b4eb0d8aceb6

Fully Sharded Data Parallel: faster AI training with fewer GPUs

Engineering at Meta · published 15 July 2021

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at engineering.fb.com Wayback capture
6blogs citing
First cited
14 Mar 2022
Most recent
12 Jan 2026
Rank this month
not ranked
External links
16
Words captured
1,985

Cited by

Blog
In the post
Date
afterhoursacademic.com
Parallelism strategies for neural network training Read at the source ↗
12 Jan 2026
PyTorch Blog
Enabling Fully Sharded Data Parallel (FSDP2) in Opacus Read at the source ↗
7 Jul 2025
main-horse.github.io
Visualizing 6D Mesh Parallelism Read at the source ↗
14 Dec 2024
PyTorch Blog
Maximizing training throughput using PyTorch FSDP Read at the source ↗
13 Mar 2024
PyTorch Blog
High-Performance Llama 2 Training and Inference with PyTorch/XLA on Cloud TPUs Read at the source ↗
6 Nov 2023
bengubler.com
Accelerate vs. DeepSpeed vs. FSDP Read at the source ↗
29 Aug 2023
benfattori.com
ZeRO Optimizer Sharding with xmap and pjit Read at the source ↗
31 Mar 2023
PyTorch Blog
Scaling Multimodal Foundation Models in TorchMultimodal with Pytorch Distributed Read at the source ↗
21 Nov 2022
PyTorch Blog
Scaling PyTorch models on Cloud TPUs with FSDP Read at the source ↗
13 Oct 2022
opensource.org
When hackers take on AI: Sci-fi – or the future? Read at the source ↗
30 Aug 2022
PyTorch Blog
Introducing PyTorch Fully Sharded Data Parallel (FSDP) API Read at the source ↗
14 Mar 2022

Links from this post

Link
Host
https://github.com/pytorch/fairseq/blob/master/examples/megatron_11b/README.md
github.com
https://fairscale.readthedocs.io/en/latest/deep_dive/pipeline_parallelism.html
fairscale.readthedocs.io
https://github.com/facebookresearch/fairscale
github.com
https://github.com/facebookresearch/fairscale/blob/master/fairscale/nn/moe/moe_layer.py
github.com
https://github.com/pytorch/fairseq
github.com
https://github.com/facebookresearch/vissl
github.com
https://arxiv.org/pdf/2001.08361.pdf
arxiv.org
https://neurips.cc/virtual/2020/public/poster_1457c0d6bfcb4967418bfb8ac142f64a.html
neurips.cc
https://lambdalabs.com/blog/demystifying-gpt-3/
lambdalabs.com
https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html
docs.nvidia.com
https://github.com/pytorch/fairseq/tree/master/examples/fully_sharded_data_parallel
github.com
https://github.com/facebookresearch/vissl/blob/40441123a6f7098500676ca8800025c1f02e28b3/vissl/config/defaults.yaml
github.com
https://pytorch-lightning.readthedocs.io/en/latest/advanced/advanced_gpu.html
pytorch-lightning.readthedocs.io
https://fairscale.readthedocs.io/en/latest/deep_dive/oss_sdp_fsdp.html
fairscale.readthedocs.io
https://github.com/facebookresearch/fairscale/blob/master/fairscale/nn/wrap/auto_wrap.py
github.com
https://github.com/facebookresearch/fairscale/blob/master/fairscale/nn/checkpoint/checkpoint_activations.py
github.com
Elsewhere on engineering.fb.com (2)