corpus.blog
41554 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a087da-c19c-71e8-b395-e27e7ef1abcc

Training Foundation Models on Supercomputers

Sam Foreman · published 15 October 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at samforeman.me Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
Links from this post
47
Words captured
3,337

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://huggingface.co/docs/transformers/v4.46.0/performance
huggingface.co
https://huggingface.co/docs/transformers/v4.46.0/perf_train_gpu_one
huggingface.co
https://pytorch.org/tutorials/intermediate/ddp_tutorial.html
pytorch.org
https://github.com/saforem2/ezpz
github.com
https://github.com/NVIDIA/Megatron-LM
github.com
https://huggingface.co/docs/accelerate/index
huggingface.co
https://docs.pytorch.org/tutorials/intermediate/ddp_tutorial.html
docs.pytorch.org
https://docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html
docs.pytorch.org
https://www.deepspeed.ai/
deepspeed.ai
https://www.deepspeed.ai/tutorials/zero/
deepspeed.ai
https://github.com/argonne-lcf/Megatron-DeepSpeed
github.com
https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/
pytorch.org
https://www.deepspeed.ai/tutorials/zero-offload/
deepspeed.ai
https://huggingface.co/docs/text-generation-inference/en/conceptual/tensor_parallelism
huggingface.co
https://pytorch.org/tutorials/intermediate/TP_tutorial.html
pytorch.org
https://pytorch.org/docs/main/distributed.pipelining.html
pytorch.org
https://deepspeed.readthedocs.io/en/latest/pipeline.html
deepspeed.readthedocs.io
https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ulysses/README.md
github.com
https://docs.nvidia.com/megatron-core/developer-guide/latest/api-guide/context_parallel.html
docs.nvidia.com
https://arxiv.org/abs/2405.07719v3
arxiv.org
https://github.com/feifeibear/long-context-attention
github.com
https://huggingface.co/docs/transformers/v4.15.0/parallelism
huggingface.co
https://github.com/huggingface/nanotron
github.com
https://x.com/tenderizzation/status/1944591320796090606
x.com
https://twitter.com/tenderizzation
twitter.com
https://chatgpt.com/share/688ab77e-9ca0-800a-8ab0-a293e06b3cce
chatgpt.com
https://www.alcf.anl.gov/sites/default/files/2024-07/Aurora_FactSheet_2024.pdf
alcf.anl.gov
https://auroragpt.anl.gov
auroragpt.anl.gov
https://github.com/Hannibal046/Awesome-LLM
github.com
https://github.com/argonne-lcf/inference-endpoints
github.com
https://github.com/argonne-lcf/Megatron-DeepSpeed/blob/e3b0398d2f2d3f8ec543e99373ca14bd18a1e4f8/megatron/arguments.py
github.com
https://sc24.supercomputing.org/2024/10/presenting-the-finalists-for-the-2024-gordon-bell-prize/
sc24.supercomputing.org
https://www.researchgate.net/profile/Carla-Mann-3/publication/387390653_MProt-DPO_Breaking_the_ExaFLOPS_Barrier_for_Multimodal_Protein_Design_Workflows_with_Direct_Preference_Optimization/links/67a0f736645ef274a46243f1/MProt-DPO-Breaking-the-ExaFLOPS-Barrier-for-Multimodal-Protein-Design-Workflows-with-Direct-Preference-Optimization.pdf
researchgate.net
https://dl.acm.org/doi/10.1109/SC41406.2024.00013
dl.acm.org
https://github.com/microsoft/DeepSpeed
github.com
https://samforeman.me/posts/auroragpt/long-sequences/
samforeman.me
https://raw.githubusercontent.com/saforem2/scaling4science/main/assets/25B.svg
raw.githubusercontent.com
https://raw.githubusercontent.com/saforem2/scaling4science/main/assets/33B.svg
raw.githubusercontent.com
https://github.com/saforem2/scaling4science
github.com
https://github.com/saforem2/Megatron-DS-Benchmarking
github.com
https://arxiv.org/abs/2509.13523
arxiv.org
https://www.intel.com/content/www/us/en/newsroom/news/intel-powered-aurora-supercomputer-breaks-exascale-barrier.html
intel.com
https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/global-forcast-system-gfs
ncdc.noaa.gov
https://arxiv.org/html/2312.15796v1
arxiv.org
https://arxiv.org/abs/2312.15796
arxiv.org
https://arxiv.org/abs/2310.04610
arxiv.org
https://samforeman.me/talks/2025/10/15/slides.html
samforeman.me