[Grafana] Add a Cluster variable to the Grafana Dashboard to enable filtering of different RayClusters

[win5923]

Why are these changes needed?

The current Grafana Dashboard aggregates metrics from all Ray clusters and summarizes them. Add a Cluster variable to Grafana dashboard to enable viewing metrics for individual RayClusters.

Ref: #2502 (comment)

TODO

• Modify the Using Prometheus and Grafana documentation to guide users on how to filters different RayClusters in Grafana.

Before:

• Ray core dashboard:
  
• Ray data dashboard:
  

After:

• Ray core dashboard:
  

• Ray data dashboard:
  

Related issue number

Closes #2502

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[win5923]

@kevin85421 PTAL

[win5923]

Ray places the JSON file in /tmp/ray/session_latest/metrics/grafana/dashboards/ inside the head pod. I will first modify the JSON file generated by Ray, and then update here.

[win5923]

According to ray-project/ray#49498. We use the Cluster variable to filter metrics from different Ray Clusters.

cc @kevin85421

• For Default Dashboard:
  
  

• For Serve Dashboard:
  
  

• For Serve Deployment Dashboard:
  
  

• For Data Dashboard:
  

---

Some insights:
Perhaps for the new version, we should remove the Legend options for better observability, because in version 2.9.0, the replica includes all the labels, while in the nightly version, the replica only includes the replica itself.

• version nightly:
  

ray_serve_deployment_processing_latency_ms_bucket{Component="core_worker", NodeAddress="10.244.0.14", SessionName="session_2025-01-10_18-51-24_268110_1", Version="3.0.0.dev0", WorkerId="1e9898a77b555c0efdeed443f6032dbb12b0e38468b9a8e9f1271dd2", application="fruit_app", container="ray-worker", deployment="FruitMarket", endpoint="metrics", instance="10.244.0.14:8080", job="prometheus-system/ray-workers-monitor", le="50.0", namespace="default", pod="rayservice-sample-raycluster-j7k5b-small-group-worker-kmvzj", ray_io_cluster="rayservice-sample-raycluster-j7k5b", replica="az01tht6", route="/fruit"}

• version 2.9.0:
  

ray_serve_deployment_processing_latency_ms_bucket{Component="core_worker", NodeAddress="10.244.0.21", SessionName="session_2025-01-10_19-17-35_723643_11", Version="2.9.0", WorkerId="0baa5ccad22b1139591d9db153edbd39f9f3af45064bcfe4638f9a5a", application="math_app", container="ray-worker", deployment="Router", endpoint="metrics", instance="10.244.0.21:8080", job="prometheus-system/ray-workers-monitor", le="+Inf", namespace="default", pod="rayservice-sample-2-raycluster-2j45h-small-group-worker-cqtk2", ray_io_cluster="rayservice-sample-2-raycluster-2j45h", replica="math_app#Router#VzsBpw", route="/calc/"}

[kevin85421]

I will merge the PR after 2.41.0 is out.

[win5923]

Updated the Grafana Dashboards to ray 2.41.0.