helm chart: ingress definitions ignored when using TUNNEL_TOKEN

[joaocc]

This behaviour was observed when trying to deploy to kubernetes by using the TUNNEL_TOKEN.
Chart configuration is similar to the "semi-official" one (https://github.com/kit-ar/cloudflare--cloudflare-tunnel--helm/tree/kit-ar/main/helm/cloudflare-tunnel)

The main difference is that instead of authenticating with credentials.json (with AccountTag, TunnelID, TunnelName, TunnelSecret) we are using TUNNEL_TOKEN env var.

In this case, the content of ingress: in config.yaml seems to be ignored, and cloudflared always fetches remotely managed configurations from Cloudflare API.

Is there any way to force cloudflared to obtain them from ingress: in config.yaml?

Expected behavior
Local configurations overriding remote configurations.

Environment and versions

• Version: 2022.4.1

Logs and errors

# tunnel config via ENV based token

# General purpose TCP routing for the network
warp-routing:
  enabled: false
# Serves the metrics server under /metrics and the readiness server under /ready
metrics: 0.0.0.0:2000
# Autoupdates applied in a k8s pod will be lost when the pod is removed or restarted, so
# autoupdate doesn't make sense in Kubernetes. However, outside of Kubernetes, we strongly
# recommend using autoupdate.
no-autoupdate: true
# The `ingress` block tells cloudflared which local service to route incoming
# requests to. For more about ingress rules, see
# https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ingress
ingress:
  - hostname: web-00.something.com
    service: http://web-00.web.svc.cluster.local:8080
  # This rule matches any traffic which didn't match a previous rule, and responds with HTTP 404.
  - service: http_status:404

loglevel: info
# loglevel: debug

transport-loglevel: warn
# transport-loglevel: info
protocol: http2    # auto, http2, h2mux, and quic

2022-05-01T08:19:28Z INF Starting tunnel tunnelID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
2022-05-01T08:19:28Z INF Version 2022.4.1
2022-05-01T08:19:28Z INF GOOS: linux, GOVersion: go1.17.1, GoArch: amd64
2022-05-01T08:19:28Z INF Settings: map[config:/etc/cloudflared/config/config.yaml loglevel:info metrics:0.0.0.0:2000 no-autoupdate:true p:http2 proto-loglevel:warn protocol:http2 transport-loglevel:warn]
2022-05-01T08:19:28Z INF Environmental variables map[TUNNEL_TOKEN:*****]
2022-05-01T08:19:28Z INF Generated Connector ID: yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy
2022-05-01T08:19:28Z INF Will be fetching remotely managed configuration from Cloudflare API. Defaulting to protocol: quic
2022-05-01T08:19:28Z INF Initial protocol http2
2022-05-01T08:19:28Z INF Starting metrics server on [::]:2000/metrics
2022-05-01T08:19:28Z INF Connection zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzz0001 registered connIndex=0 location=AMS
2022-05-01T08:19:28Z INF Updated to new configuration config="{\"ingress\":[{\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}}" version=2
2022-05-01T08:19:29Z INF Connection zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzz0002 registered connIndex=1 location=CDG
2022-05-01T08:19:30Z INF Connection zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzz0003 registered connIndex=2 location=AMS
2022-05-01T08:19:31Z INF Connection zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzz0004 registered connIndex=3 location=CDG

[joliveirinha]

Hi @joaocc ,

What you explained is done like that by design. Once a Tunnel is created as remotely managed (via the UI), then the tunnel will always pick the configuration that is sent from the edge.

If you want to create a tunnel that is locally managed but still run with the token, then you can create the tunnel via the "cloudflared create" command, which will be locally managed, and then you can get the token for that tunnel by running:
`cloudflared tunnel token $TUNNEL" .

That said, is there a specific reason for you to prefer your configuration to be locally managed?

[joaocc]

Hi,
Thanks for the clarification.
At this moment we are still in the exploring phase of scalable use of cloudflared, as an alternative to the traditional (LoadBalancer + nginx-ingress-controller + cert-man) + ingress resources.
By scalable I mean devops-scale (not performance), where the ease of use with as few configuration points is critical, especially when managing 10s of endpoints in kubernetes clusters via fluxcd/argocd/...

In our baseline scenario, after setting up the (LoadBalancer + nginx-ingress-controller + cert-man), each "user" only has to add some annotations to the ingress resource, and everything works. The closest we found seems to be adyanth/cloudflare-operator#47 (unofficial) or the apparently abandoned cloudflare/cloudflare-ingress-controller#176.

Configuring locally will keep configurations local and version controlled (as when used via fluxcd or argocd) while creating them manually will introduce a manual step and a source of error.

Hope this helps.
Thx

[joliveirinha]

For kubernetes envs what we purpose is having cloudflared in front of the ingress controller. (https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/routing-to-tunnel/kubernetes/#deploy-in-front-of-an-ingress-controller)

You can then have cloudflared with a single wildcard ingress rule pointing to the ingress controller which is responsible for managing the actual ingresses.
Would this work in your scenario?

[joaocc]

Hi,
It might cover some scenarios (where we can/want to keep the internal ingress controller as "rule master"), but they would all require that all workloads in a single cluster are all part of a same wildcard which isn't always the case. Since we are already using external-dns to set records in CF, it might work.
But were also exploring the case of using cloudflared tunnel rules to replace the internal ingress controller while keeping the use of ingress resources in different workloads, and in that case we currently don't have such a smooth story.
Thx