Record · corpus.blog/posts/01a08749-da3d-72bc-8f23-c78d7dbe1076

A closer look at "training" a trillion-parameter model on Frontier

blog.glennklockwood.com · published 13 January 2024

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at blog.glennklockwood.com Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
External links: 20
Words captured: 5,381

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

https://arxiv.org/abs/2312.12705

arxiv.org

https://www.tomshardware.com/tech-industry/supercomputers/frontier-trained-a-chatgpt-sized-large-language-model-with-only-3000-of-its-37888-radeon-gpus-the-worlds-fastest-supercomputer-blasts-through-one-trillion-parameter-model-with-only-8-percent-of-its-mi250x-gpus

tomshardware.com

https://www.techradar.com/pro/most-formidable-supercomputer-ever-is-warming-up-for-chatgpt-5-thousands-of-old-amd-gpu-accelerators-crunched-1-trillion-parameter-models

techradar.com

https://www.olcf.ornl.gov/frontier/

olcf.ornl.gov

https://github.com/microsoft/Megatron-DeepSpeed

github.com

https://github.com/ROCmSoftwarePlatform

github.com

https://github.com/NVIDIA/apex

github.com

https://github.com/deephyper/deephyper

github.com

https://arxiv.org/abs/2001.08361

arxiv.org

https://arxiv.org/abs/2203.15556

arxiv.org

https://doi.org/10.1145/3581784.3613215

doi.org

https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them

help.openai.com

https://arxiv.org/pdf/2101.00027.pdf

arxiv.org

https://www.osti.gov/biblio/1994640

osti.gov

https://dx.doi.org/10.1109/SC41405.2020.00024

dx.doi.org

https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDXHE8vVgUF0NmEfuhsy8LFBx6PgAT32sDwGQC2f6vA4r0qn7PGbofgtN8_iVVF3X57zW8737h3gkxb9Gx7nEZpiJ5e3Um-pm06Exzl5IHK5E5UdH2kJ9Iuyb2ljKWeGQngo_5sec9sl9EvVUSKLOcFpfrBbCpmsLTzGjkznlC6vNCRg3419qTLAC8nPZR/s2103/transformer.png

blogger.googleusercontent.com

https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9ujJnHZNbPl60m3LO00K8xD8-yd_YvyDg2JVY5AaX4t7s0xeSxlzLgGecHzzd16ofsFCqg5QAwiysIJFAFmN18BeHysomeQxx01-ZGQxyzSnbt5rxNU2jlp1dzm0pLr6wJ3nsyJo8AdP6Ijhf-qClyHKEnil_i5iEXCsGIhqH84aC-S7wtCv-_P7oZnGI/s2126/ffnn.png

blogger.googleusercontent.com

https://www.microsoft.com/en-us/research/uploads/prod/2019/08/pipedream.pdf

microsoft.com

Data-Parallel Distributed Training of Deep Learning Models Read at the source ↗

siboehm.com

https://www.microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training/

microsoft.com