Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rootful mode - node to node networking does not work #365

Open
vsoch opened this issue Feb 21, 2025 · 10 comments
Open

Rootful mode - node to node networking does not work #365

vsoch opened this issue Feb 21, 2025 · 10 comments
Labels
bug Something isn't working

Comments

@vsoch
Copy link
Contributor

vsoch commented Feb 21, 2025

Hi @AkihiroSuda - I was testing GPU with usernetes, and got fairly far but hit two erroneous cases:

  • I can't access the GPU devices with rootless mode (see this issue)
  • When I run docker with rootfull (confirmed by a startup message) none of the usernetes networking works. Meaning I can shell into a node and I can't contact any ip address on another node. Pod to pod if they are on the same node works.
  • When I remove the need for the GPU and run the same workflow with CPU and rootless, everything works again.

I don't think it's in scope for you to help with getting GPUs working with rootless, but I'm hoping you might have insight for why networking stops working when it's rootful. From the readme it sounds like it should work?

Thanks!

@AkihiroSuda AkihiroSuda added the bug Something isn't working label Feb 21, 2025
@AkihiroSuda
Copy link
Member

Is this issue reproducible on GHA?

@vsoch
Copy link
Contributor Author

vsoch commented Feb 21, 2025

Probably not, unless you are able and willing to pump out money for custom runners with GPUs! 😆

I can barely get V100s on the clouds... can't imagine what regular use in CI would require.

@AkihiroSuda
Copy link
Member

"Rootful mode - node to node networking does not work" does not seem relevant to GPUs?

@vsoch
Copy link
Contributor Author

vsoch commented Feb 21, 2025

Good idea! Is there a way to do an equivalent of a restart with lima? I can install docker (rootful, skipping the dockerd-rootless installer tool), and add the user to the docker group, and normally then I need a login / out to get it working. In this case I'm getting a permissions error on the socket (because I haven't).

https://github.com/researchapps/usernetes/actions/runs/13460718730/job/37615121293

@vsoch
Copy link
Contributor Author

vsoch commented Feb 21, 2025

Just found this - testing now!

@vsoch
Copy link
Contributor Author

vsoch commented Feb 21, 2025

Yes! It works with that base! (This is the confirmation of rootful, as you know):

Image

And the test fails:

Image

I'll open a PR with this added test.

@vsoch
Copy link
Contributor Author

vsoch commented Feb 21, 2025

All set: #366

Thanks for the help @AkihiroSuda 🙏

@vsoch
Copy link
Contributor Author

vsoch commented Feb 22, 2025

To give you an update @AkihiroSuda - I've spent about 48 hours on the rootless case, and I've actually gotten it working several times with a strategy that uses cdi from the host. The problem is consistency, and all the manual tweaks / customizations that are required. For example, tonight I've brought up a few clusters per hour, and I'll get it working, try to harden the setup, but then when I bring up (what I deem to be) "the same" again, I get a slightly different error. I'll even see cases where it runs once, and then there is a containerd error about permissions. I'm not sure if you have experience about what might be causing that? That's the error that seems to be unsolvable in the sense that once I see it, there is no way to fix it and go back to a working state.

I'm probably not going to work on this over the weekend because I'm a bit behind on everything else that I should have been doing for the last few days. But if you are interested, the VM build, customizations, and setup branch is here. For safe keeping, this is the error that ultimately happens:

  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               116s  default-scheduler  Successfully assigned gpu-operator/nvidia-device-plugin-daemonset-hg6kz to u7s-flux-001
  Warning  FailedCreatePodSandBox  116s  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox "c3594993a3cfcfcee487887ac9b62f601a4da29985385691aa19f8c81c86c661": failed to create containerd task: failed to create shim task: failed to mount rootfs component: operation not permitted

I tried mounting a tmpfs there, and various attempts to cleanup and restart, no fix. What seems to be working OK is getting the container built from the host and being able to run nvidia-smi to see devices, but what isn't clear to me is what commands I should run with nvidia-ctk in the container to properly configure it with the gpu-operator and then have GPUs that are seen by the container. I've had this work a few times (and then fail) so there is something happening w.r.t state. I think the million dollar questions are:

  1. What is going on with containerd permissions (and why)
  2. What is the ideal way to expose the devices to containerd inside the usernetes node

@AkihiroSuda
Copy link
Member

Found a workaround: execute ethtool --offload eth0 tx-checksum-ip-generic off in usernetes-node-1 container

Originally posted by @AkihiroSuda in #366 (comment)


The nvidia stuff is irrelevant to "Rootful mode - node to node networking does not work", and should be discussed in a separate issue.
I saw you already managed to get it work, though https://vsoch.github.io/2025/rootless-usernetes-gpu/

@vsoch
Copy link
Contributor Author

vsoch commented Feb 24, 2025

Yes! Apologies for that. It was only relevant in that the NVIDIA GPUs / device plugins install easily with rootful mode, and helped me discover the bug here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants