corpus.blog
43,398 blogs · 5,519,570 posts Claim your blog
Record · corpus.blog/posts/01a0fdb6-a145-722f-bcc0-f14e4ef31f54

High-Performance Llama 2 Training and Inference with PyTorch/XLA on Cloud TPUs

PyTorch Blog · published 6 November 2023

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at pytorch.org Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
External links
34
Words captured
3,512

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://ai.meta.com/llama/
ai.meta.com
https://ai.meta.com/blog/large-language-model-llama-meta-ai/
ai.meta.com
https://about.fb.com/news/2023/07/llama-2/
about.fb.com
https://github.com/pytorch/xla
github.com
https://cloud.google.com/blog/products/compute/announcing-cloud-tpu-v5e-and-a3-gpus-in-ga
cloud.google.com
https://github.com/pytorch-tpu/transformers/blob/llama2-google-next-training/SPMD_USER_GUIDE.md
github.com
https://github.com/pytorch-tpu/llama/blob/llama2-google-next-inference/TORCH_XLA_USER_GUIDE.md
github.com
https://www.youtube.com/watch?v=PSpmRtWuMs8
youtube.com
https://arxiv.org/pdf/2307.09288.pdf
arxiv.org
Fully Sharded Data Parallel: faster AI training with fewer GPUs Read at the source ↗
engineering.fb.com
Llama 2 is here - get it on Hugging Face Read at the source ↗
huggingface.co
https://github.com/pytorch/xla/blob/master/docs/fsdp.md
github.com
https://github.com/pytorch/xla/blob/master/docs/spmd.md
github.com
https://arxiv.org/pdf/2105.04663.pdf
arxiv.org
https://github.com/pytorch-tpu/transformers/blob/llama2-google-next-training/src/transformers/models/llama/modeling_llama.py
github.com
https://github.com/pytorch-tpu/transformers/blob/llama2-google-next-training/examples/pytorch/language-modeling/run_clm.py
github.com
https://arxiv.org/pdf/1909.08053.pdf
arxiv.org
https://github.com/pytorch-tpu/transformers/blob/llama2-google-next-training/src/transformers/trainer.py
github.com
https://github.com/pytorch-tpu/transformers/tree/llama2-google-next-training
github.com
https://cloud.google.com/blog/products/compute/using-cloud-tpu-multislice-to-scale-ai-workloads
cloud.google.com
https://github.com/pytorch/xla/blob/master/torch_xla/experimental/xla_sharding.py
github.com
https://jax.readthedocs.io/en/latest/_autosummary/jax.lax.with_sharding_constraint.html
jax.readthedocs.io
https://arxiv.org/pdf/2305.13245.pdf
arxiv.org
https://github.com/pytorch-tpu/llama/blob/llama2-google-next-inference/reshard_checkpoints.py
github.com
https://github.com/facebookresearch/llama/blob/ea9f33d6d3ea8ed7d560d270986407fd6c2e52b7/llama/generation.py
github.com
https://github.com/pytorch-tpu/llama/blob/b89dd0f2351c42fef367670d9d2c5b65cd0ae932/llama/generation.py
github.com
https://github.com/openxla/stablehlo
github.com
https://www.tensorflow.org/guide/saved_model
tensorflow.org
https://github.com/pytorch/xla/releases
github.com
https://github.com/pytorch/xla/blob/r2.1/docs/stablehlo.md
github.com
https://arxiv.org/abs/2204.02311
arxiv.org
https://github.com/pytorch-tpu/transformers/blob/llama2-google-next-training/src/transformers/models/llama/tokenization_llama.py
github.com
https://arxiv.org/abs/1804.04235
arxiv.org
https://colab.sandbox.google.com/github/pytorch/xla/blob/master/contrib/colab/getting-started.ipynb
colab.sandbox.google.com
Elsewhere on pytorch.org (5)