v2: node-exporter on multi node cluster always reporting the same node in instance label

[sbidoul]

Hello,

I'm trying to use v2 (2.0.1) to migrate from a grafana-agent-operator deployment. This sounds very easy and intuitive so far, thanks for this work!

I'm having a little issue with node exporter metrics which only have a single node in the instance label.
Apparently, it is always the node where the grafana-k8s-monitoring-alloy-metrics-0 pod is running.

I'm using a basic values.yaml like this. Am I missing something obvious?

cluster:
  name: ...

destinations:
  ...

clusterMetrics:
  enabled: true
  node-exporter:
    metricsTuning:
      useDefaultAllowList: true
      includeMetrics:
        - node_load1
        - node_load5
        - node_load15
        - node_time_seconds
        - node_boot_time_seconds

prometheusOperatorObjects:
  enabled: true

clusterEvents:
  enabled: true

podLogs:
  enabled: true

alloy-logs:
  enabled: true

alloy-metrics:
  enabled: true

alloy-singleton:
  enabled: true

kube-state-metrics:
  enabled: true

[sbidoul]

I found the cause. The node exporter pods deployed by this chart run on the host network, so I had to allow port 9100 from nodes running the grafana-k8s-monitoring-alloy-metrics-0 pod. This was not the case with grafana-agent-operator which deployed grafana-agent-integrations pods on cluster IPs.

[petewall]

Can you share the updated values that made things work?

[sbidoul]

@petewall actually I did not find how to change the values to expose the node exporters on cluster IPs.

So I ended up adapting my firewall rules.

But if there is a way to avoid that it would be preferable.