Record · corpus.blog/posts/01a087e7-cfb5-73db-b551-7812fd062023

Everything about Distributed Training and Efficient Finetuning

sumanthrh.com · published 1 October 2023

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at sumanthrh.com Wayback capture

1blogs citing

First cited: 1 Oct 2024
Most recent: 1 Oct 2024
Rank this month: not ranked
External links: 83
Words captured: 9,316

Cited by

Blog

In the post

Date

jackyoustra.com

AI Ramp Read at the source ↗

1 Oct 2024

Links from this post

Link

Host

https://github.com/lm-sys/FastChat/

github.com

https://github.com/OpenAccess-AI-Collective/axolotl

github.com

https://github.com/ggerganov/llama.cpp

github.com

https://github.com/stas00/ml-engineering

github.com

https://www.semianalysis.com/p/google-gemini-eats-the-world-gemini

semianalysis.com

Transformer Math 101 Read at the source ↗

blog.eleuther.ai

https://blog.research.google/2019/03/introducing-gpipe-open-source-library.html

blog.research.google

https://huggingface.co/docs/transformers/perf_train_gpu_many

huggingface.co

https://arxiv.org/abs/1910.02054

arxiv.org

https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/

microsoft.com

https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html

docs.nvidia.com

https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c

towardsdatascience.com

https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/operations.html

docs.nvidia.com

https://github.com/stas00/ml-engineering/tree/master/model-parallelism

github.com

https://docs.it4i.cz/dgx2/introduction/

docs.it4i.cz

https://arxiv.org/abs/2004.13336

arxiv.org

https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html

pytorch.org

https://huggingface.co/docs/transformers/main_classes/trainer

huggingface.co

https://www.deepspeed.ai/tutorials/pipeline/

deepspeed.ai

https://github.com/NVIDIA/Megatron-LM

github.com

https://github.com/bigscience-workshop/Megatron-DeepSpeed

github.com

https://github.com/huggingface/transformers/issues/8771

github.com

https://github.com/huggingface/nanotron

github.com

https://en.wikipedia.org/wiki/Bfloat16_floating-point_format

en.wikipedia.org

https://huggingface.co/docs/peft/conceptual_guides/lora

huggingface.co

https://huggingface.co/docs/peft/conceptual_guides/ia3

huggingface.co

https://arxiv.org/abs/2305.14314

arxiv.org

https://x.com/younesbelkada/status/1705258148045750343

x.com

https://github.com/huggingface/transformers/issues/26350

github.com

https://huggingface.co/docs/transformers/en/perf_infer_gpu_one

huggingface.co

https://arxiv.org/abs/2208.07339

arxiv.org

https://arxiv.org/abs/2210.17323

arxiv.org

https://huggingface.co/docs/transformers/v4.34.0/en/main_classes/quantization

huggingface.co

https://openreview.net/forum?id=H1oyRlYgg

openreview.net

https://www.deepspeed.ai/tutorials/one-cycle/

deepspeed.ai

https://github.com/microsoft/DeepSpeed/issues/2928

github.com

https://github.com/artidoro/qlora/blob/7f4e95a68dc076bea9b3a413d2b512eca6d004e5/qlora.py

github.com

https://github.com/EleutherAI/gpt-neox/pull/725

github.com

https://github.com/imoneoi/multipack_sampler

github.com

https://platform.openai.com/docs/guides/legacy-fine-tuning/hyperparameters

platform.openai.com

https://arxiv.org/abs/2005.14165

arxiv.org

https://arxiv.org/abs/2211.05100

arxiv.org

https://huggingface.co/docs/transformers/v4.18.0/en/performance

huggingface.co

https://aws.amazon.com/ec2/instance-types/p4/

aws.amazon.com

https://x.com/StasBekman/status/1710038192983019547?s=20

x.com

https://aws.amazon.com/blogs/aws/new-amazon-ec2-p5-instances-powered-by-nvidia-h100-tensor-core-gpus-for-accelerating-generative-ai-and-hpc-applications/

aws.amazon.com

https://github.com/stas00/ml-engineering/blob/9a51114f8377350bfbf1764f23feac441e865401/multi-node/all_reduce_bench.py

github.com

https://github.com/huggingface/accelerate/issues/2020

github.com

https://github.com/pytorch/pytorch/issues/102434

github.com

https://deepspeed.readthedocs.io/en/latest/memory.html

deepspeed.readthedocs.io

https://huggingface.co/docs/accelerate/usage_guides/fsdp

huggingface.co

https://github.com/lm-sys/FastChat

github.com

https://lmsys.org/

lmsys.org

https://github.com/lm-sys/FastChat/blob/main/docs/model_support.md

github.com

https://vllm.ai/

vllm.ai

https://github.com/lm-sys/FastChat/blob/e64ee0e9a0d2d1a987a59a1cfe24bf711f3dec09/fastchat/train/train.py

github.com

https://github.com/lm-sys/FastChat/issues/1588

github.com

https://github.com/lm-sys/FastChat/tree/main/playground

github.com

https://github.com/lm-sys/FastChat/blob/e64ee0e9a0d2d1a987a59a1cfe24bf711f3dec09/docs/training.md

github.com

https://arxiv.org/abs/2306.05685

arxiv.org

https://huggingface.co/cerebras/btlm-3b-8k-base

huggingface.co

Mistral 7B

mistral.ai

https://github.com/OpenAccess-AI-Collective/axolotl/blob/90e0d673f76f84478856434deb6024c5c869a5ad/src/axolotl/utils/callbacks.py

github.com

https://www.nvidia.com/en-us/data-center/dgx-a100/

nvidia.com

Fine-tune Falcon 180B with DeepSpeed ZeRO, LoRA and Flash Attention Read at the source ↗

philschmid.de

https://huggingface.co/datasets/databricks/databricks-dolly-15k

huggingface.co

https://github.com/stas00

github.com

How to Train Really Large Models on Many GPUs? Read at the source ↗

lilianweng.github.io

https://www.deepspeed.ai/tutorials/zero-offload/

deepspeed.ai

https://arxiv.org/abs/2101.06840

arxiv.org

https://arxiv.org/abs/2104.07857

arxiv.org

https://www.deepspeed.ai/tutorials/zero/

deepspeed.ai

https://arxiv.org/abs/2306.10209

arxiv.org

https://www.deepspeed.ai/tutorials/zeropp/

deepspeed.ai

https://arxiv.org/abs/1710.03740

arxiv.org

https://arxiv.org/abs/2106.09685

arxiv.org

Efficient Large Language Model training with LoRA and Hugging Face Read at the source ↗

philschmid.de

ELI5: FlashAttention Read at the source ↗

gordicaleksa.medium.com

https://tridao.me/publications/flash2/flash2.pdf

tridao.me

https://medium.com/tensorflow/fitting-larger-networks-into-memory-583e3c758ff9

medium.com

https://www.deepspeed.ai/training/

deepspeed.ai

Fine-tuning Llama 2 70B using PyTorch FSDP Read at the source ↗

huggingface.co

https://medium.com/@sourabmangrulkar/falcon-180b-finetuning-using-peft-and-deepspeed-b92643091d99

medium.com