corpus.blog
42,501 blogs · 5,361,753 posts Claim your blog
Record · corpus.blog/posts/01a0fd77-1502-7033-a72b-71408034c41b

Maximizing training throughput using PyTorch FSDP

PyTorch Blog · published 13 March 2024

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at pytorch.org Wayback capture
1blogs citing
First cited
29 Jul 2024
Most recent
29 Jul 2024
Rank this month
not ranked
External links
15
Words captured
2,040

Cited by

Blog
In the post
Date
152334h.github.io
Calculating the Cost of a Google Deepmind Paper Read at the source ↗
29 Jul 2024

Links from this post

Link
Host
https://github.com/foundation-model-stack/fms-fsdp
github.com
Fully Sharded Data Parallel: faster AI training with fewer GPUs Read at the source ↗
engineering.fb.com
https://arxiv.org/pdf/2205.05198.pdf
arxiv.org
https://arxiv.org/abs/2304.11277
arxiv.org
https://github.com/foundation-model-stack/foundation-model-stack/blob/main/fms/models/llama.py
github.com
https://github.com/foundation-model-stack/fms-fsdp/blob/main/fms_fsdp/policies/ac_handler.py
github.com
https://github.com/pytorch/pytorch/blob/2240018c03744ee34ea14ad53481db934c37e384/torch/utils/flop_counter.py
github.com
https://github.com/karpathy/nanoGPT
github.com
https://github.com/stas00/ml-engineering/tree/master/compute/accelerator
github.com
https://github.com/EleutherAI/lm-evaluation-harness
github.com
https://github.com/tatsu-lab/stanford_alpaca
github.com
Fine-tuning Llama 2 70B using PyTorch FSDP Read at the source ↗
huggingface.co
https://github.com/facebookresearch/llama-recipes
github.com
https://www.linkedin.com/in/stasbekman/?originalSubdomain=ca
linkedin.com
https://minjiazhang.github.io/
minjiazhang.github.io
Elsewhere on pytorch.org (4)