Enable multi-architecture support, specifically for s390x, in CNAO and the associated CNIs.

[ashokpariya0]

Is your feature request related to a problem? Please describe:
The origin-kube-rbac-proxy image(quay.io/openshift/origin-kube-rbac-proxy@sha256:e2def4213ec0657e72eb790ae8a115511d5b8f164a62d3568d2f1bff189917e8) used by CNAO operator is not compatible with the s390x architecture.

Describe the solution you'd like:
use multi arch supported image for kube-rbac-proxy.
origin-kube-rbac-proxy image should be compatible with s390x architecture as well.

Describe alternatives you've considered:
We can use quay.io/brancz/kube-rbac-proxy@sha256:e6a323504999b2a4d2a6bf94f8580a050378eba0900fd31335cf9df5787d9a9b
for kube-rbac-proxy image as this image is compatible with amd64, arm, arm64, ppc64le and s390x architecture.
Please check :
(https://quay.io/repository/brancz/kube-rbac-proxy/manifest/sha256:e6a323504999b2a4d2a6bf94f8580a050378eba0900fd31335cf9df5787d9a9b)

[ormergi]

@ashokpariya0 could you please elaborate on the motivation for supporting s390x arch?
Is there some tracking issue or enhancement proposal for supporting s390x?

[ashokpariya0]

We are enabling s390x (IBM Z system) support in all(most) kubevirt ecosystem components. Already kubevirt (virt-operator), supports s390x and now we are working on additional components like cnao, cdi(almost done), ssp, etc.

[oshoval]

Checked again, wrt auto bumper, i think in this specific case we dont need to update bumper scripts as this parameter is part of CNAO and not part of the components.
So your PR is fine.

What about all the other images, some of them arent multiarch right ?
We won't able to maintain it right now, but will prioritize it as a task for all the images once we can.
If you plan to contribute and able to give support if it breaks we might able to take your PR, thanks.
Maybe we first need to map all the required changes over all repos and just then start by fixing one by one.

Also please consider that kube-rbac-proxy need to be changed on https://github.com/k8snetworkplumbingwg/kubemacpool so if KMP is installed standalone or via CNAO it will be the same

WRT the proposed image, it seems safe to use it as https://github.com/brancz/kube-rbac-proxy is the one
https://github.com/openshift/kube-rbac-proxy is forked from

Wdyt about fixing it / opening an issue on https://github.com/openshift/kube-rbac-proxy please ?

@phoracek agree ? anything i missed ?
Thanks

[ashokpariya0]

Checked again, wrt auto bumper, i think in this specific case we dont need to update bumper scripts as this parameter is part of CNAO and not part of the components. So your PR is fine.

@oshoval Thank you for your feedback.

What about all the other images, some of them arent multiarch right ? We won't able to maintain it right now, but will prioritize it as a task for all the images once we can. If you plan to contribute and able to give support if it breaks we might able to take your PR, thanks. Maybe we first need to map all the required changes over all repos and just then start by fixing one by one.

Yes, you are right. We have noticed that most of the CNI images do not support multiple architectures, and we are in the process of identifying the necessary changes. Our initial focus is to get the CNAO operator running on the s390x architecture(for this we need to change kube-rbac-proxy image), and then we will address the other CNI images one by one, as you suggested. We are committed to actively contributing to this effort.

Also please consider that kube-rbac-proxy need to be changed on https://github.com/k8snetworkplumbingwg/kubemacpool so if KMP is installed standalone or via CNAO it will be the same

Okay, Noted.

WRT the proposed image, it seems safe to use it as https://github.com/brancz/kube-rbac-proxy is the one https://github.com/openshift/kube-rbac-proxy is forked from

Yes, That is correct.

Wdyt about fixing it / opening an issue on https://github.com/openshift/kube-rbac-proxy please ?

Sure, I will create the issue and follow up on it.

[oshoval]

Thank you

Yes, you are right. We have noticed that most of the CNI images do not support multiple architectures, and we are in the process of identifying the necessary changes. Our initial focus is to get the CNAO operator running on the s390x architecture(for this we need to change kube-rbac-proxy image), and then we will address the other CNI images one by one, as you suggested. We are committed to actively contributing to this effort.

Just to make sure, i meant other components that CNAO deploys, do we talk about the same?
(i believe we do)

kubeMacPool
kubeSecondaryDNS
kubevirtIpamController
linuxBridge
macvtap
multus
multusDynamicNetworks
ovs

[ashokpariya0]

Thank you

Yes, you are right. We have noticed that most of the CNI images do not support multiple architectures, and we are in the process of identifying the necessary changes. Our initial focus is to get the CNAO operator running on the s390x architecture(for this we need to change kube-rbac-proxy image), and then we will address the other CNI images one by one, as you suggested. We are committed to actively contributing to this effort.

Just to make sure, i meant other components that CNAO deploys, do we talk about the same?

kubeMacPool
kubeSecondaryDNS
kubevirtIpamController
linuxBridge
macvtap
multus
multusDynamicNetworks
ovs

Yes.
I tried all of the above CNIs, Below is my findings.
Multus : Able to deploy with latest image tag.
Multus Dynamic Networks Controller: We don't have mutli arch supported image for this CNI plugin. We need to build one.
linuxBridge- Image is multi arch supported but we see runtime issue(Marker binary is unavailable for s390x) with images.
macvtap: Image is multi arch supported but cp binray present inside image is not s390x compatible
ovs: We don't have mutli arch supported images for this CNI plugin.
KubeSecondaryDNS : We don't have multi arch image for this CNI plugin.

[oshoval]

Amazing thank you
what about kubeMacPool / kubevirtIpamController if possible as well please ?

[oshoval]

Please keep in mind for changes that are done, that we will also need ARM, so best if changes can be done in a robust manner (or at least strives to)
no need to test for ARM of course, just if some new builds are made, will be nice if the scripts can be robust please

[ashokpariya0]

Amazing thank you what about kubeMacPool / kubevirtIpamController if possible as well please ?

Regarding kubemacpool:- Image is multi arch supported However I see runtime issue with image as manager binary present inside kubemacpool-cert-manager init container is not compatible with s390x.
kubevirtIpamController: looks fine.

[oshoval]

Thanks a lot, we are here whatever needed

[oshoval]

Hi @zhlhahaha
Might interest you about ARM, as some of the changes might affect it as well

[oshoval]

Might worth please to consider a gist about how to emulate s390x so whoever need to sanity check / develop it can
https://www.qemu.org/docs/master/system/target-s390x.html

[ashokpariya0]

Might worth please to consider a gist about how to emulate s390x so whoever need to sanity check / develop it can https://www.qemu.org/docs/master/system/target-s390x.html

Sure, Thanks!

[zhlhahaha]

Hi @zhlhahaha Might interest you about ARM, as some of the changes might affect it as well

Let me take a look, thanks for reminding @oshoval

[ashokpariya0]

@oshoval Please check PR: #1917

[ashokpariya0]

@oshoval I created an issue on GitHub to enable multi-architecture image support for quay.io/openshift/origin-kube-rbac-proxy. You can find the issue here: Issue #113

[ashokpariya0]

@oshoval @RamLavi The last pushed upstream CNAO image (link) was 13 days ago. Since then, a couple of pull requests have been merged in the CNAO repository (specifically the last two). However, I haven't seen a new operator image published. Could you please help clarify the reason for this?

[oshoval]

Hi, we dont release on each PR, just once there is a need, if you need it we can release

[ashokpariya0]

@phoracek @oshoval @RamLavi
Please review the PR at #1923 when you get a chance.