corpus.blog
41554 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a087da-c19c-71e8-b395-e27e81c8dd92

Scientific AI at Scale: Distributed Training

Sam Foreman · published 2 September 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at samforeman.me Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
Links from this post
51
Words captured
22,161

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://samforeman.me/talks/openskai25/training/slides
samforeman.me
Scientific AI at Scale: Distributed Training Read at the source ↗
samforeman.me
https://huggingface.co/docs/transformers/v4.46.0/performance
huggingface.co
https://huggingface.co/docs/transformers/v4.46.0/perf_train_gpu_one
huggingface.co
https://pytorch.org/tutorials/intermediate/ddp_tutorial.html
pytorch.org
https://github.com/saforem2/ezpz
github.com
https://pytorch.org/docs/stable/notes/ddp.html
pytorch.org
https://huggingface.co/docs/transformers/accelerate
huggingface.co
https://www.deepspeed.ai/
deepspeed.ai
https://mpi4py.readthedocs.io/en/stable/tutorial.html
mpi4py.readthedocs.io
https://pytorch.org/docs/stable/distributed.html
pytorch.org
https://developer.nvidia.com/nccl
developer.nvidia.com
https://www.intel.com/content/www/us/en/developer/tools/oneapi/oneccl.html
intel.com
https://arxiv.org/abs/1708.03888
arxiv.org
https://wiki.lustre.org/Configuring_Lustre_File_Striping
wiki.lustre.org
https://www.deepspeed.ai/tutorials/zero/
deepspeed.ai
https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/
pytorch.org
https://www.deepspeed.ai/tutorials/zero-offload/
deepspeed.ai
https://huggingface.co/docs/text-generation-inference/en/conceptual/tensor_parallelism
huggingface.co
https://pytorch.org/tutorials/intermediate/TP_tutorial.html
pytorch.org
https://pytorch.org/docs/main/distributed.pipelining.html
pytorch.org
https://deepspeed.readthedocs.io/en/latest/pipeline.html
deepspeed.readthedocs.io
https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ulysses/README.md
github.com
https://docs.nvidia.com/megatron-core/developer-guide/latest/api-guide/context_parallel.html
docs.nvidia.com
https://arxiv.org/abs/2405.07719v3
arxiv.org
https://github.com/feifeibear/long-context-attention
github.com
https://github.com/argonne-lcf/Megatron-DeepSpeed
github.com
https://huggingface.co/docs/transformers/v4.15.0/parallelism
huggingface.co
https://github.com/huggingface/nanotron
github.com
https://github.com/huggingface/transformers/issues/10321
github.com
https://github.com/anton-l
github.com
https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/
microsoft.com
https://deepspeed.readthedocs.io/en/latest/zero3.html
deepspeed.readthedocs.io
https://www.deepspeed.ai/tutorials/pipeline/
deepspeed.ai
https://pytorch.org/docs/stable/distributed.tensor.parallel.html
pytorch.org
https://github.com/Hannibal046/Awesome-LLM
github.com
https://colab.research.google.com/github/saforem2/intro-hpc-bootcamp-2025/blob/main/docs/02-llms/08-shakespeare-example-colab/index.out.ipynb
colab.research.google.com
https://github.com/argonne-lcf/ai-science-training-series/tree/main/06_parallel_training
github.com
https://github.com/saforem2/wordplay
github.com
https://github.com/saforem2/ezpz/blob/main/src/ezpz/test_dist.py
github.com
https://asciinema.org/a/668460
asciinema.org
https://github.com/karpathy/nanoGPT
github.com
https://asciinema.org/a/668462
asciinema.org
https://samforeman.me
samforeman.me
https://www.twitter.com/saforem2
twitter.com
https://github.com/emilhvitfeldt/quarto-iframe-examples
github.com
https://arxiv.org/abs/2104.04473
arxiv.org
http://jalammar.github.io/illustrated-transformer/
jalammar.github.io
https://huggingface.co/docs/transformers/main/en/llm_tutorial
huggingface.co
https://arxiv.org/abs/2206.07682
arxiv.org
https://arxiv.org/abs/2305.10601
arxiv.org