Monitor with actionsMetricsServer

[totogtr]

Hi,

I'm using ARC v0.27.1 deployed with the helm chart via ArgoCD, RunnerSets and HRA with

 scaleUpTriggers:
   - githubEvent:
       workflowJob: {}

I have an Ingress exposing the github-webhook service and succesfully connected it to the github org, receiving events and running runners.
However I'm having some troubles making the monitoring work with the Prometheus Operator in cluster. I guess I'm missing a bit of documentation.
What is the expected behavior of actionsMetricsServer ? Is it supposed to be exposed as another webhook (and another app in github side) to be able to receive events and in turn expose the metrics ? Is it supposed to work together with the 2 other possible ServiceMonitor from the helm chart ?

When taking a look at the code, I see the serviceMonitor in actionsmetrics.servicemonitor.yaml.yml

  selector:
    matchLabels:
      {{- include "actions-runner-controller-actions-metrics-server.selectorLabels" . | nindent 6 }}

and in the actions metrics helper :

{{- define "actions-runner-controller-actions-metrics-server.selectorLabels" -}}
app.kubernetes.io/name: {{ include "actions-runner-controller-actions-metrics-server.name" . }}
app.kubernetes.io/instance: {{ include "actions-runner-controller-actions-metrics-server.instance" . }}

instance is :

{{- define "actions-runner-controller-actions-metrics-server.instance" -}}
{{- printf "%s-%s" .Release.Name "actions-metrics-server" }}
{{- end }}

Until here it looks fine but then, if I look at the service from actionsmetrics.service.yaml (which I suppose is the Service that the ServiceMonitor aims to monitor :

  labels:
    {{- include "actions-runner-controller.labels" . | nindent 4 }}

which is defined in the main helper (not the one dedicated to actions metrics !) :

{{- define "actions-runner-controller.labels" -}}
[...]
{{ include "actions-runner-controller.selectorLabels" . }}

{{- define "actions-runner-controller.selectorLabels" -}}
app.kubernetes.io/name: {{ include "actions-runner-controller.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

So it looks like the ServiceMonitor is filtering on an instance label expecting something like "app.kubernetes.io/instance: gh-actions-runner-system-actions-metrics-server" and the actual service has only the "app.kubernetes.io/instance: gh-actions-runner-system" label so it won't pick it. But I haven't seen any issue regarding this so I guess I'm missing something. If someone has an idea it would be great !

---

Anyway, if using a custom ServiceMonitor from outside of the chart, I can select the service and the prometheus operator see some endpoints but I get a 401 unauthorized. Same issue when I port-forward & curl the pod directly. Any ideas ?
Is anyone able to share a working actionsMetricsServer and ServiceMonitor configuration ?

Thanks for reading !

[totogtr]

Well after some investigation, the 401 comes from the missing CR & CRB described here.
I still believe there are some issues with the service labels, I'll submit a PR.

[neilharris123]

Hi @totogtr ,

Could you share your working configuration, because I too am not receiving any of the expected metrics. Once you enabled the action-metrics-server, servicemonitor etc, did it just work, or was there further config required? I am already using the latest chart, so the CR and CRB are already deployed.

[andy-townsend]

@neilharris123 @totogtr - did you ever get this working? Trying to figure out the necessary steps after enabling actions-metrics-server as well