corpus.blog
41554 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a087da-c19c-71e8-b395-e27e7f4bfa6f

Training Foundation Models on Supercomputers

Sam Foreman · published 24 September 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at samforeman.me Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
Links from this post
43
Words captured
52,614

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://huggingface.co/docs/transformers/v4.46.0/performance
huggingface.co
Scientific AI at Scale: Distributed Training Read at the source ↗
samforeman.me
https://huggingface.co/docs/transformers/v4.46.0/perf_train_gpu_one
huggingface.co
https://pytorch.org/tutorials/intermediate/ddp_tutorial.html
pytorch.org
https://github.com/saforem2/ezpz
github.com
https://pytorch.org/docs/stable/notes/ddp.html
pytorch.org
https://huggingface.co/docs/transformers/accelerate
huggingface.co
https://www.deepspeed.ai/
deepspeed.ai
https://mpi4py.readthedocs.io/en/stable/tutorial.html
mpi4py.readthedocs.io
https://pytorch.org/docs/stable/distributed.html
pytorch.org
https://developer.nvidia.com/nccl
developer.nvidia.com
https://www.intel.com/content/www/us/en/developer/tools/oneapi/oneccl.html
intel.com
https://github.com/ROCm/rccl
github.com
https://wiki.lustre.org/Configuring_Lustre_File_Striping
wiki.lustre.org
https://www.deepspeed.ai/tutorials/zero/
deepspeed.ai
https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/
pytorch.org
https://www.deepspeed.ai/tutorials/zero-offload/
deepspeed.ai
https://huggingface.co/docs/text-generation-inference/en/conceptual/tensor_parallelism
huggingface.co
https://pytorch.org/tutorials/intermediate/TP_tutorial.html
pytorch.org
https://pytorch.org/docs/main/distributed.pipelining.html
pytorch.org
https://deepspeed.readthedocs.io/en/latest/pipeline.html
deepspeed.readthedocs.io
https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ulysses/README.md
github.com
https://docs.nvidia.com/megatron-core/developer-guide/latest/api-guide/context_parallel.html
docs.nvidia.com
https://arxiv.org/abs/2405.07719v3
arxiv.org
https://github.com/feifeibear/long-context-attention
github.com
https://github.com/argonne-lcf/Megatron-DeepSpeed
github.com
https://huggingface.co/docs/transformers/v4.15.0/parallelism
huggingface.co
https://github.com/huggingface/nanotron
github.com
https://dl.acm.org/doi/10.1109/SC41406.2024.00013
dl.acm.org
https://arxiv.org/abs/2509.13523
arxiv.org
LLMs on Aurora: Hands-On Read at the source ↗
samforeman.me
https://saforem2.github.io/ezpz
saforem2.github.io
https://github.com/saforem2/ezpz/blob/main/bin/utils.sh
github.com
https://github.com/saforem2/ezpz/blob/main/src/ezpz/examples/minimal.py
github.com
https://github.com/ezpz/ezpz/blob/main/ezpz/test.py
github.com
https://github.com/ezpz/ezpz/blob/main/ezpz/generate.py
github.com
https://github.com/ezpz/ezpz/blob/main/ezpz/hf_trainer.py
github.com
https://alcf.anl.gov/aurora
alcf.anl.gov
https://alcf.anl.gov/polaris
alcf.anl.gov
https://doi.org/10.1109/SC41406.2024.00013
doi.org
https://bit.ly/ezpz-utils
bit.ly
https://raw.githubusercontent.com/saforem2/ezpz/main/bin/utils.sh
raw.githubusercontent.com
https://samforeman.me/talks/2025/09/24/slides.html
samforeman.me