vllm on ray claster with asymmetric hardware

[arm2arm]

Dear vllm experts, I am trying to deploy vllm in distributed mode, we have at our research institute 4 nodes each with 1xA100, they are working pretty good with distributed ray cluster. Now we got an another node with 2xL40S, ray can show all 6 gpus, but one node with 2 gpus . how to start vllm to use all gpus?
currently we use:

vllm serve  Qwen/Qwen2.5-72B-Instruct-GPTQ-Int4    --tensor-parallel-size 1  --dtype=half    --pipeline-parallel-size 5  --distributed-executor-backend ray  --max-model-len 8192 --trust-remote-code

[wedobetter]

Either run it on 4 GPU's or buy another 2.
The problem is attention heads and vocab need to be divisible by the number of instances and typically they come in 40, 64, and multiples of 8 sizes. Qwen-72b-Instruct has 64 attention heads.
I had the same problem with 7 GPU's, I ended up buying one more.
Otherwise the other GPU's can be used for other purposes such as:

• TTS / STT
• Model guard (detect malicious prompts)
• Judge model
• Embedding models (RAG)

Given that you have generous GPU specs, another option would be to run 3x replicas with each replica taking 2 GPUs.

[Diffizle]

have u tried pipeline parallelism? can it work with an odd number of gpus?

[wedobetter]

I use pipeline parallelism and it works even with an odd number of GPUs, however models won’t load unless vocab size and number of attention heads can be split evenly across the GPUs.