Record · corpus.blog/posts/01a0fdc8-5107-735a-b197-ff3bde5ba27b

High performance Llama 2 deployments with AWS Inferentia2 using TorchServe

PyTorch Blog · published 4 October 2023

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at pytorch.org Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
External links: 24
Words captured: 2,695

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

https://ai.meta.com/llama/

ai.meta.com

https://aws.amazon.com/ec2/instance-types/inf2/

aws.amazon.com

https://aws.amazon.com/machine-learning/inferentia/

aws.amazon.com

https://aws.amazon.com/machine-learning/neuron/

aws.amazon.com

https://ai.meta.com/llama/responsible-use-guide/

ai.meta.com

https://github.com/aws-neuron/transformers-neuronx

github.com

https://github.com/aws-neuron/transformers-neuronx/blob/main/src/transformers_neuronx/llama/model.py

github.com

https://github.com/aws-neuron/transformers-neuronx/blob/33fa412447a4028edb252fd06aae9ed93086a450/src/transformers_neuronx/llama/model.py

github.com

https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/transformers-neuronx/transformers-neuronx-developer-guide.html

awsdocs-neuron.readthedocs-hosted.com

https://github.com/pytorch/serve/blob/d0ae857abfe6d36813c88e531316149a5a354a93/examples/large_models/inferentia2/llama2/Readme.md?plain=1

github.com

https://huggingface.co/meta-llama/Llama-2-13b-hf

huggingface.co

https://github.com/pytorch/serve/blob/master/examples/large_models/inferentia2/llama2/model-config.yaml

github.com

https://github.com/pytorch/serve/blob/master/examples/large_models/inferentia2/llama2/inf2_handler.py

github.com

https://github.com/pytorch/serve/blob/2bf505bae3046b0f7d0900727ec36e611bb5dca3/docs/configuration.md?plain=1

github.com

https://github.com/pytorch/serve/tree/master/examples/micro_batching

github.com

https://github.com/pytorch/serve/blob/d0ae857abfe6d36813c88e531316149a5a354a93/examples/large_models/inferentia2/llama2/inf2_handler.py

github.com

https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/arch/neuron-features/neuron-caching.html

awsdocs-neuron.readthedocs-hosted.com

https://torchserve.s3.amazonaws.com/mar_files/sm-neuronx/llama-2-13b-neuronx-b1/

torchserve.s3.amazonaws.com

https://github.com/aws/deep-learning-containers/blob/master/available_images.md

github.com

https://github.com/pytorch/serve/blob/master/model-archiver/README.md

github.com

https://docs.aws.amazon.com/sagemaker/latest/dg/large-model-inference-uncompressed.html

docs.aws.amazon.com

https://github.com/aws/amazon-sagemaker-examples-community/blob/main/torchserve/inf2/llama2/llama-2-13b.ipynb

github.com

https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/index.html

awsdocs-neuron.readthedocs-hosted.com

https://github.com/aws-neuron/aws-neuron-samples/blob/master/torch-neuronx/transformers-neuronx/inference/meta-llama-2-13b-sampling.ipynb

github.com