corpus.blog
41554 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a0879e-aebc-7209-be67-74f1953401dc

Why reduction precision matters

main-horse.github.io · published 4 December 2024

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at main-horse.github.io Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
Links from this post
16
Words captured
2,446

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://github.com/pytorch/pytorch/blob/main/torch/distributed/fsdp/_fully_shard/_fsdp_api.py
github.com
https://github.com/pytorch/torchtitan
github.com
https://github.com/pytorch/torchtitan/blob/0ed31f55977dd9f6bb476736b432d00750c741c0/torchtitan/config_manager.py
github.com
https://pytorch.org/docs/stable/amp.html
pytorch.org
https://github.com/pytorch/pytorch/pull/92882
github.com
https://pytorch.org/docs/stable/ddp_comm_hooks.html
pytorch.org
https://gist.github.com/main-horse/21b74dc1cf8ed7231f74072c8a3343bc
gist.github.com
https://github.com/pytorch/pytorch/blob/8dd4673ceacad362f7c5c4bacdd52d64e2a9f601/torch/distributed/fsdp/_fully_shard/_fsdp_collectives.py
github.com
https://developer.nvidia.com/blog/massively-scale-deep-learning-training-nccl-2-4/
developer.nvidia.com
https://docs.nvidia.com/deeplearning/nccl/release-notes/rel_2-23-4.html
docs.nvidia.com
https://network.nvidia.com/pdf/solutions/hpc/paperieee_copyright.pdf
network.nvidia.com
https://github.com/NVIDIA/nccl/issues/1497
github.com
https://arxiv.org/pdf/2408.14158
arxiv.org
https://github.com/microsoft/mscclpp
github.com
https://www.reddit.com/r/MachineLearning/comments/1eb4xn4/p_ncclx_mentioned_in_llama3_paper/
reddit.com
https://www.lesswrong.com/tag/semantic-stopsign
lesswrong.com