enabling HTTPS

[gid-fieldcode]

As we get into EnvoyGateway, we have followed the tutorial at https://tetrate.io/blog/envoy-gateway-with-aws-nlb/ and got a demo service to work with HTTP.

We, however, have problems getting it to work with HTTPS. What makes things worse for us is that we can't find any logs that will help us figure out what we're doing wrong. The "envoy-gateway" pod only logs the routes it sees, and the "envoy-test-apps" pod (deployed on namespace=test and with name=apps) logs nothing when our curl commands fail. Can we somehow make logging richer? Currently we see this repeated hundreds of times:

envoy {"start_time":"2024-08-16T10:41:22.255Z","method":"-","x-envoy-origin-path":"-","protocol":"HTTP/1.1","response_code":"400","response_flags":"DPE","response_code_details":"http1.codec_error","connection_termination_details":" ││ -","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"11","duration":"0","x-envoy-upstream-service-time":"-","x-forwarded-for":"-","user-agent":"-","x-request-id":"-",":authority":"-","upstream_host":"-","up ││ stream_cluster":"-","upstream_local_address":"-","downstream_local_address":"10.65.61.127:10080","downstream_remote_address":"10.65.9.30:13535","requested_server_name":"-","route_name":"-"}

Perhaps you can find what we are doing wrong, or share with us a configuration that is working for you. As the tutorial indicates, we are using the AWS LB controller as a NLB provisioner. The certificate you see below matches the hostname for the HTTProute.

$ curl -v  https://httpbin.ourcompany.com/httpbin/headers
....
*  SSL certificate verify ok.
> GET /httpbin/headers HTTP/1.1
> Host: httpbin.ourcompany.com
> User-Agent: curl/7.68.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< content-length: 11
< content-type: text/plain
< date: Fri, 16 Aug 2024 10:36:32 GMT
< connection: close
<
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):
Bad Request

Our setup is as follows:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    namespace: envoy-gateway
    name: custom-proxy-config

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-gateway
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        annotations:          
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          # service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-ssl-cert: 'arn:aws:acm:eu-west-1:361789914539:certificate/6a3b4415-ad5e-4037-96f9-e46bd894acc3'
          service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
          service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: 'ELBSecurityPolicy-TLS13-1-2-2021-06'          
          service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*' 
          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: TCP

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: apps
  namespace: test
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
    - name: https
      protocol: HTTP
      port: 443

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
  namespace: test
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: apps
  hostnames:
    - "httpbin.ourcompany.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /httpbin/
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /
      backendRefs:
        - group: ""
          kind: Service
          name: httpbin
          port: 8000
          weight: 1

[liorokman]

I think that you've enabled the PROXY protocol on the loadbalancer, using the service.beta.kubernetes.io/aws-load-balancer-proxy-protocol annotation, but did not enable it on the EnvoyGateway side using a ClientTrafficPolicy or a BackendTrafficPolicy if your service is actually expecting to be connected to with the PROXY protocol enabled.

This matches the response_code_details":"http1.codec_error error that you're getting.

You should try to either disable the PROXY protocol on the loadbalancer or configure it in Envoy Gateway.

[gid-fieldcode]

Thanks a lot for your quick reply!

Indeed, disabling the proxy protocol solved our problem. However we do need the proxy protocol for our backend, so we enabled it in the NLB (uncommenting the line you saw above) and enabled it also on the Gateway with the BackendTrafficPolicy below, but now our curl is no longer working, and we again have the error:

envoy {"start_time":"2024-08-16T13:03:39.878Z","method":"-","x-envoy-origin-path":"-","protocol":"HTTP/1.1","response_code":"400","response_flags":"DPE","response_code_details":"http1.codec_error","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"11","duration":"0","x-envoy-upstream-service-time":"-","x-forwarded-for":"-","user-agent":"-","x-request-id":"-",":authority":"-","upstream_host" :"-","upstream_cluster":"-","upstream_local_address":"-","downstream_local_address":"10.65.52.120:10080","downstream_remote_address":"10.65.9.30:19980","requested_server_name":"-","route_name":"-"}

Have we missed something?

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: enable-proxy-protocol-policy
  namespace: test
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: apps
    # namespace: default
  proxyProtocol:
    version: V2

[liorokman]

If you enabled the PROXY protocol on the NLB, then it's being used between the NLB and Envoy Proxy. You need to enable the PROXY protocol on the Envoy Proxy listener. To do this, you need a ClientTrafficPolicy.

Once you've enabled the ClientTrafficPolicy, Envoy Proxy will expect the PROXY protocol data at the beginning of each TCP connection, and correctly read it and translate it into a X-Forwarded-For header that will be sent to the backend you configured.

You're getting the codec error because Envoy Proxy is not receiving HTTP right now - but rather the bytes associated with the PROXY protocol.

Your backend doesn't need to support the PROXY protocol if it is configured to use the X-Forwarded-For header.

By enabling the PROXY protocol in a BackendTrafficPolicy you are configuring Envoy Proxy to use the PROXY protocol when it connects to the configured backend. Do this if your backend expects the PROXY protocol to be used and knows how to parse it. But if the NLB is using the PROXY protocol and there is no ClientTrafficPolicy configured, traffic isn't being sent to the backend since Envoy Proxy isn't receiving valid HTTP.

[gid-fieldcode]

Thanks again, your help is much appreciated!