Provide multi-platform default backend image

[stevehipwell]

I'd like the Ingress NGINX project to provide a default backend image as the current image used is almost 5 years old, is only available in single platform versions and is no longer published.

I couldn't find an existing issue for this.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[nlamirault]

/remove-lifecycle stale

[stafot]

We need this too so if you guide me on what image you suggest to use as base and where to deploy it I would me more than happy to work on this issue
cc @strongjz

[strongjz]

We've been trying to consolidate the images we maintain, as it's a burden for us to maintain them all and keep them updated A solution to this issue would be to use the custom backend; we already produce these images as part of the release.

https://kubernetes.github.io/ingress-nginx/examples/customization/custom-errors/#customized-default-backend

https://github.com/kubernetes/ingress-nginx/blob/main/docs/examples/customization/custom-errors/custom-default-backend.yaml

[stafot]

Is this already multiarch?. If yes , for sure this can work for our use case.

[stafot]

@strongjz I tested the following tag

image: ingress-nginx/nginx-errors
tag: "v20230505"

And is not working on arm instances. So even customised backend doesn't solve the arm compatibility problem.

[stafot]

Based on https://github.com/kubernetes/ingress-nginx/blob/e00b45beb567e6d65bb5aae6941c351e17345982/images/custom-error-pages/rootfs/Dockerfile seems that the dockerfile doesn't need any change to become multiarch. So probably just in the buildprocess need to be added the arm7, arm64 support. I didn't find where this is built though.

[stafot]

following up the discussion here I opened a draft PR to update alpine version on custom-error-pages image and also to add arm64 builds. Do you think this approach is to the correct direction or do you think that we should take another route to solve the above problem?

[Jabbl]

I just encountered the same issue trying to deploy nginx-errors on an arm64-based device.
Your draft PR is exactly what I was looking for, as I've recently performed similar updates on my own docker images to get it working in that environment.

If nginx-errors is the image that's going to be maintained as the default-backend, having an arm64-version would be really helpful.

[strongjz]

So cloudbuild should be building them on all the arch defined in the makefile and be pulled from registry.k8s.io the github action updated here are for pushing to docker hub, something were testing out.

So we need to figure out why registry.k8s.io doesn't have the arch requested.

And we need to talk as maintainers if these should go to dockerhub or not.

[strongjz]

The latest nginx-errors support arm, arm64, amd64, and s390x; what is the issue?

crane manifest registry.k8s.io/ingress-nginx/nginx-errors@sha256:c2ca20775f41c0c89906dc74a31239d94ac48e84e7c6164affed41fe669cdaa5
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:894abf3b896d8d935d8259ba345286755da4b9bff760dbbfe7cf23cb3f3853da",
      "size": 2650,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:ba5bc88ea8ef8a4fb514ba1966ac1271ba5645b3bd61abafeb247adc24ee7d58",
      "size": 2650,
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:a344178256cfc39929a4f9d58eaf260d4d3007c3dc195764c25bfe8e2a5dcc6f",
      "size": 2650,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:4d78dc83d5938dcf9432b3110d9c2337f736ea390d27b31b5045da551acbf93d",
      "size": 2650,
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      }
    }
  ]
}

[stafot]

@strongjz Ok let me test this

[stafot]

@strongjz The

    registry: registry.k8s.io
    image: ingress-nginx/nginx-errors
    tag: "v20231208-4c39e6acc"

which is the same with the sha you shared (more recent) worked as expected, so my PR on the build part is not relevant anymore.
Just as side note, it is not very easy for anyone non core contributor of the project to find this information easily. So a good idea would be on the release notes to add as block the more recent images so anyone easily can test them, and not bugging you for problems that are already solved.

Thanks again for your support, really appreciated.

[Jabbl]

Okay, I may have thought those versions were some kind of "beta", since the version I used previously was 1.3.0.
Listing all versions from registry.k8s.io nets me this collection:

    "0.48.1",
    "0.49.0",
    "1.2.0",
    "1.3.0",
    "sha256-0116499ff83f02360faee7b4f3842f63df7eea40be2db17cc41a467fdf991336.sig",
    "sha256-06cd2dbe317505e940d0521cc69d237752800ef0b191cb4265eb0d3d2c1080e7.sig",
    "sha256-09c421ac743bace19ab77979b82186941c5125c95e62cdb40bdf41293b5c275c.sig",
    "sha256-1c31c80828e7800c4eb556e07fdc90c451482767659eb26310e0ad208d28c9cf.sig",
    "sha256-332be6ff8c4e93e8845963932f98839dfd52ae49829c29e06475368a3e4fbd9e.sig",
    "sha256-3600dcd1bbd0d05959bb01af4b272714e94d22d24a64e91838e7183c80e53f7f.sig",
    "sha256-37257a3cbc1aba523cad8f0cf5b946df3aff9e74483e7fa9ce8df2d965e71ec7.sig",
    "sha256-59bd93fede2559c2b61d3baa5e9066e5b3ba71fa4faa06e368629ed287469b1d.sig",
    "sha256-8506829ac6ef6ee74240e7ae62e8ad8085851d3281cc6b9d61a48cf3822daed7.sig",
    "sha256-aabd7a001f6a0a07ed6ea8f6da87e928bfa8f971eba2bef708f3e8504fc5cc9b.sig",
    "sha256-c2ca20775f41c0c89906dc74a31239d94ac48e84e7c6164affed41fe669cdaa5.sig",
    "v20220916-gd32f8c343",
    "v20221219-controller-v1.5.1-49-ge3e0d9c1f",
    "v20230312-helm-chart-4.5.2-28-g66a760794",
    "v20230404-helm-chart-4.6.0-11-gc76179c04",
    "v20230505",
    "v20230626",
    "v20230721-40f94ef1b",
    "v20231208-4c39e6acc"

Of these, only the first four seemed like proper "versions".

I'll test pulling v20231208-4c39e6acc on my setup to verify that it works.

[strongjz]

We switched all the images outside the controller image to use the date+sha since they can change at different paces than that controller itself.

[strongjz]

And you cant overwrite tags in registry.k8s.io, so if we needed to patch an image we cant bump them like we do for the controller.

[stafot]

These are reasonable choices. But still I think that is needed to have a place to easily find/track the latest version for each image.

[stevehipwell]

@strongjz a SemVer version would be preferential if possible; could the image not be independently versioned?