Llama 3.1 405B on Gaudi

[ppatel-eng]

We are trying to run Llama 3.1 405B on Gaudi and are running into memory constraints when following the guide below on 8 Gaudi 2 HPUs. Our end goal is to use vllm-fork to serve llama 3.1 405b, ideally with as little quantization as possible.

https://github.com/HabanaAI/vllm-hpu-extension/blob/main/calibration/README.md

[michalkuligowski]

Hi @ppatel-eng for the time being we don't provide a multinode solution. Did you have any issues with running model calibration procedure?

[vinayK34]

Please use the https://github.com/HabanaAI/vllm-hpu-extension/tree/main/calibration to quantize the model. Make sure you are using the latest vllm and vllm-hpu-extenstion version with gaudi pytorch 1.19 image