Custom metrics in http check don't work in kubernetesResource

[bjan-sag]

Hi!

Now being able to test kubernetesResource, I ran into a different issue :) When trying to use kubernetesResource + http checks, it works fine. But when I try to extract custom metrics from the http check, they don't appear in metrics at all.

Example config (stitched together from https://github.com/flanksource/canary-checker/blob/master/fixtures/k8s/kubernetes_resource_service_pass.yaml and https://github.com/flanksource/canary-checker/blob/master/fixtures/minimal/metrics.yaml):

canary.yaml

---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: pod-svc-test
  namespace: default
  labels:
    "Expected-Fail": "false"
spec:
  schedule: "@every 2m"
  kubernetesResource:
    - name: service accessibility test
      namespace: default
      description: "deploy httpbin & check that it's accessible via its service"
      waitFor:
        expr: 'dyn(resources).all(r, k8s.isReady(r))'
        interval: 2s
        timeout: 2m
      resources:
        - apiVersion: v1
          kind: Pod
          metadata:
            name: httpbin-pod-1
            namespace: default
            labels:
              app: httpbin-pod-1
          spec:
            containers:
              - name: httpbin
                image: "kennethreitz/httpbin:latest"
                ports:
                  - containerPort: 80
        - apiVersion: v1
          kind: Service
          metadata:
            name: httpbin-svc
            namespace: default
          spec:
            selector:
              app: httpbin-pod-1
            ports:
              - port: 80
                targetPort: 80
      checks:
        - http:
            - name: Call httpbin service
              url: "http://httpbin-svc.default.svc"
              metrics:
                - name: httpbin_count
                  type: counter
                  value: "1"
                  labels:
                    - name: check_name
                      valueExpr: check.name
                    - name: code
                      valueExpr: code
                - name: httpbin_2xx_duration
                  type: counter
                  value: elapsed.getMilliseconds()
                  labels:
                    - name: check_name
                      valueExpr: check.name
      checkRetries:
        delay: 2s
        interval: 3s
        timeout: 2m

It shows green in the UI, but when I query metrics the ones I defined are not there:

# kubectl -n canary-checker exec canary-checker-c659c5578-mjt5v -- curl -s localhost:8080/metrics | grep httpbin_count
#

With a simple (standalone) http check it works as expected:

# kubectl -n canary-checker exec canary-checker-c659c5578-mjt5v -- curl -s localhost:8080/metrics | grep httpbin_count
# HELP httpbin_count
# TYPE httpbin_count counter
httpbin_count{check_name="http-minimal-check",code="200"} 2

It's very unfortunate that it doesn't work, as it'd be our main use case for Canary Checker (running custom pods with checks and getting metrics from them).

[miry]

Is it related to #2227?

[yashmehrotra]

@miry No, there is a different flow for health checks which are executed via the kubernetes resource field.

I have raised a PR to fix that