Monitoring Dashboard for vLLM

[jasonacox]

vLLM Dashboard

What are you using to monitor your vLLM installation?

I wanted a simple dashboard to help trend vLLM and GPU metrics and performance. The Prometheus metrics that vLLM provides gives some good insight into the health and use of the system with data like tokens per second, running requests, waiting requests and more. What indicators do you find most useful?

Grafana for vLLM and GPU Metrics

I put together a Grafana dashbaord using InfluxDB as the datastore, Telegraf to pull the metrics from vLLM and a custom script to grab nvidia-smi data. I uploaded the details here and would be happy to create an example PR if there is interest: https://github.com/jasonacox/TinyLLM/tree/main/monitoring

[simon-mo]

https://github.com/vllm-project/vllm/tree/main/examples/production_monitoring

Here's an exampel dashboard put together by @robertgshaw2-neuralmagic

[jasonacox]

Thanks @simon-mo ! I'm going to use that.

@robertgshaw2-neuralmagic Could you share your grafana.json dashboard file using the "Export for sharing externally" toggle? The one that is there seems to be using hardcoded datasources.

[robertgshaw2-neuralmagic]

@jasonacox https://github.com/vllm-project/vllm/pull/4711/files should fix.

thanks for pointing this out, not sure how this got into this state

[frittentheke]

@robertgshaw2-neuralmagic I am using the Dashboard via automatic import from K8s ConfigMaps
(something along the lines of https://artifacthub.io/packages/helm/kiwigrid/grafana-dashboards)

I changed the datasource to come from a template var in: #6136
to account for these kind of use cases - PTAL.