SRIOV Operator e2e test environment

[martinkennelly]

@adrianchiris @zshi-redhat

During the additional K8s Network & Resource Management meeting we discussed adding e2e tests to each "SRIOV" related repositories. See Jan 26 Notes: https://docs.google.com/document/d/1bEyEh_K9lGwQWHgUHUH-vqodYd1W-LCWRGlW0-ZVR5A/edit

Our next steps is to deploy k8 with kind and attach a device and run e2e tests from this repo. We can document our progress/issues here.

DoD: e2e tests from this repo is executed successfully on our test environment with injected HW and document setup flow here to allow future automation.

[ipatrykx]

@martinkennelly @zshi-redhat @adrianchiris

I was working on this recently. Here are some findings that I think it would be nice to share.

I was able to run kind cluster, and pass PF into the container by moving the PF into container's netns with

`ip link set netns <container_netns>.

It turned out that there might be a bug in config-daemon. The code is checking the env variable CLUSTER_TYPE, which should be equal to kubernetes in case of kind cluster, but it seems it is hardcoded to openshift in the Dockerfile. This makes the config daemon's pod crash. After I've changed that to kubernetes and rebuilt the image it started just fine. If it will be necessary I will try to provide more info in separate issue report. But it might be that I have misconfigured something.

After PF was moved to the container's netns, the tests that use vfio-pci driver passed. There is however a problem with tests that use netdevices (with default kernel-driver). For some reason (I don't know yet) the VF's are created with vfio-pci driver by default, so they are not seen in the kernel-space. I was able to manually bind the driver to iavf, but the VFs are then present in the host's netns. After moving those VF's to kind container's netns I was able to finish the test successfully.

It seems that for automation it will be necessary to create some mechanism to

a) bind VFs to proper driver (or make it bind to kernel driver like iavf by default)
b) watch the network interfaces on host and move newly created VF's to kind's netns.

[zshi-redhat]

@ipatrykx thanks for working on this and sharing the findings!

It turned out that there might be a bug in config-daemon. The code is checking the env variable CLUSTER_TYPE, which should be equal to kubernetes in case of kind cluster, but it seems it is hardcoded to openshift in the Dockerfile. This makes the config daemon's pod crash. After I've changed that to kubernetes and rebuilt the image it started just fine. If it will be necessary I will try to provide more info in separate issue report. But it might be that I have misconfigured something.

Yes, it is not hard coded to openshift cluster type in Dockerfile.rhel7. If you build the image from Dockerfile, it would not set to openshift. There is open PR to enable k8s CLUSTER_TYPE: #62

After PF was moved to the container's netns, the tests that use vfio-pci driver passed. There is however a problem with tests that use netdevices (with default kernel-driver). For some reason (I don't know yet) the VF's are created with vfio-pci driver by default, so they are not seen in the kernel-space. I was able to manually bind the driver to iavf, but the VFs are then present in the host's netns. After moving those VF's to kind container's netns I was able to finish the test successfully.

I suspect this might be a bug or mis-configuration.
The default deviceType is netdevice, could you share the sriovnetworknodepolicy when that issue happens?

It seems that for automation it will be necessary to create some mechanism to

a) bind VFs to proper driver (or make it bind to kernel driver like iavf by default)
b) watch the network interfaces on host and move newly created VF's to kind's netns.

So this means when PF is moved to separate netns, newly created VFs appear in host namespace by default?

[ipatrykx]

Hi @fedepaol, thank you for your input. I will check the script you've pointed to. :)

As for the second point, I think you are right. This will need to be continuous process running on host I believe. I also forgot to mention that I was working on single node kind configuration as I thought it would be easier to start with.

[fedepaol]

@ipatrykx thanks for working on this and sharing the findings!

It turned out that there might be a bug in config-daemon. The code is checking the env variable CLUSTER_TYPE, which should be equal to kubernetes in case of kind cluster, but it seems it is hardcoded to openshift in the Dockerfile. This makes the config daemon's pod crash. After I've changed that to kubernetes and rebuilt the image it started just fine. If it will be necessary I will try to provide more info in separate issue report. But it might be that I have misconfigured something.

Yes, it is not hard coded to openshift cluster type in Dockerfile.rhel7. If you build the image from Dockerfile, it would not set to openshift. There is open PR to enable k8s CLUSTER_TYPE: #62

After PF was moved to the container's netns, the tests that use vfio-pci driver passed. There is however a problem with tests that use netdevices (with default kernel-driver). For some reason (I don't know yet) the VF's are created with vfio-pci driver by default, so they are not seen in the kernel-space. I was able to manually bind the driver to iavf, but the VFs are then present in the host's netns. After moving those VF's to kind container's netns I was able to finish the test successfully.

I suspect this might be a bug or mis-configuration.
The default deviceType is netdevice, could you share the sriovnetworknodepolicy when that issue happens?

It seems that for automation it will be necessary to create some mechanism to
a) bind VFs to proper driver (or make it bind to kernel driver like iavf by default)
b) watch the network interfaces on host and move newly created VF's to kind's netns.

So this means when PF is moved to separate netns, newly created VFs appear in host namespace by default?

Not only that, it also means that they'll get back to the host ns as soon as a pod dies.

[ipatrykx]

@zshi-redhat

Sure, here is the policy that test creates:

Name:         policy-2
Namespace:    sriov-network-operator
Labels:       <none>
Annotations:  <none>
API Version:  sriovnetwork.openshift.io/v1
Kind:         SriovNetworkNodePolicy
Metadata:
  Creation Timestamp:  2021-02-19T10:34:06Z
  Generation:          1
  Managed Fields:
    API Version:  sriovnetwork.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:mtu:
        f:nicSelector:
          .:
          f:pfNames:
          f:rootDevices:
        f:nodeSelector:
          .:
          f:feature.node.kubernetes.io/network-sriov.capable:
        f:numVfs:
        f:priority:
        f:resourceName:
      f:status:
    Manager:         e2e.test
    Operation:       Update
    Time:            2021-02-19T10:34:06Z
  Resource Version:  256710
  UID:               360a4653-914f-4cfe-96ff-923e30c7ba31
Spec:
  Mtu:  9000
  Nic Selector:
    Pf Names:
      ens785f3
    Root Devices:
      0000:02:00.3
  Node Selector:
    feature.node.kubernetes.io/network-sriov.capable:  true
  Num Vfs:                                             6
  Priority:                                            99

Yes, the new VFs are created in host's netns.

@fedepaol I would add, that it's not only when the pod/container is deleted, but whenever the VF is deleted and recreated (which is done by test multiple times). Can I maybe ask what kernel did you use in your test, because I am using CentOS7 with 3.10.0-1160.15.2.el7.x86_64 but I am wondering if maybe that was changed in something more recent.

[fedepaol]

@martinkennelly no you are right, that's what happens also when vfs are created.
I don't remember the kernel version, I worked on those tests almost two years ago and tried to revamp them one year ago.

[ipatrykx]

Just a quick update - I've worked on this last week, and been able to create the VF passing script and run all the tests properly using KIND. I should be able to create PR within a few days.

[abdallahyas]

@ipatrykx I tried to follow your instruction for configuring kind #54 (comment), but with a mellanox interface, and i seem to have issues creating vfs from inside the kind nodes after adding them to the node netns.
I tried to add the vf and got:

echo 4 > /sys/class/net/enp3s0f0/device/sriov_numvfs
bash: /sys/class/net/enp3s0f0/device/sriov_numvfs: Read-only file system

Could you please provide us the steps you took to have a successful e2e tests?
thanks.

[ipatrykx]

Hi @AbdYsn, the kind's container's sysfs is by default read-only https://github.com/kubernetes-sigs/kind/blob/92d8408f6211e11eb2c13d6a68f37b1991800d69/images/base/files/usr/local/bin/entrypoint#L53

You can make it writeable by remounting it with mount -o remount,rw /sys

[martinkennelly]

PR posted #100 which implements e2e testing using KinD w/ one master and one worker node. Alsom golang script to ensure VFs remain in test netns.

[Dcjones36]

@adrianchiris @zshi-redhat

During the additional K8s Network & Resource Management meeting we discussed adding e2e tests to each "SRIOV" related repositories. See Jan 26 Notes: https://docs.google.com/document/d/1bEyEh_K9lGwQWHgUHUH-vqodYd1W-LCWRGlW0-ZVR5A/edit

Our next steps is to deploy k8 with kind and attach a device and run e2e tests from this repo. We can document our progress/issues here.

DoD: e2e tests from this repo is executed successfully on our test environment with injected HW and document setup flow here to allow future automation.