kubeadm-reset: add notes about more manual cleanup steps #49448

neolit123 · 2025-01-15T13:53:48Z

currently the kubeadm reset command when finished prints the following:


The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

instead of printing that information from the CLI we should keep it document at the website and link to the reset page from the output of the command.

xref

neolit123 · 2025-01-15T13:55:08Z

/cc @SataQiu @pacoxu @carlory
/cc @danwinship (for the kubeproxy cleanup part)

neolit123 · 2025-01-15T13:58:19Z

content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md

+### Network traffic assets cleanup
+
+The `kubeadm reset` command does not clean any iptables, nftables or IPVS assets applied


changing 'traffic assets' to 'traffic rules', despite IPVS using 'tables', afaik.

netlify · 2025-01-15T14:03:46Z

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`e0905eb`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-io-main-staging/deploys/6787bdef495b810009ac5345
😎 Deploy Preview	https://deploy-preview-49448--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify · 2025-01-15T14:12:46Z

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`fc22259`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-io-main-staging/deploys/678a84eab3a5a5000930b3a8
😎 Deploy Preview	https://deploy-preview-49448--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md

pacoxu · 2025-01-16T01:58:44Z

https://deploy-preview-49448--kubernetes-io-main-staging.netlify.app/docs/reference/setup-tools/kubeadm/kubeadm-reset/#cni-configuration-cleanup

content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md

carlory · 2025-01-16T02:55:21Z

content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md

+```
+
+The output of the above command should print `DONE` at the end.
+Instead of `docker` you can use an alternative tool such as `nerdctl` or `crictl`.


nodes may not contain the docker command since kubernetes removed the dockershim support. nerdctl may be a good example than docker command. crictl is incompatible with docker command, it is hard for many users to start and run a container to do cleanup work without an example.

Compared to containerd and docker, crictl is not friendly to users in this task. I prefer to remove it.

I'm not sure whether the nerdctl binary is installed by default on nodes. Maybe ctr is more common. For example,

(base) ➜ ~ kind create cluster -n demo Creating cluster "demo" ... ✓ Ensuring node image (kindest/node:v1.32.0) 🖼 ✓ Preparing nodes 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 Set kubectl context to "kind-demo" You can now use your cluster with: kubectl cluster-info --context kind-demo Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂 (base) ➜ ~ docker exec -it demo-control-plane bash root@demo-control-plane:/# nerdctl bash: nerdctl: command not found root@demo-control-plane:/# kubeadm reset root@demo-control-plane:/# ctr i ls REF TYPE DIGEST SIZE PLATFORMS LABELS root@demo-control-plane:/# ctr i pull registry.k8s.io/kube-proxy:v1.32.0 registry.k8s.io/kube-proxy:v1.32.0: resolved |++++++++++++++++++++++++++++++++++++++| index-sha256:6aee00d0c7f4869144d1bdbbed7572cd55fd1a4d58fef5a21f53836054cb39b4: exists |++++++++++++++++++++++++++++++++++++++| manifest-sha256:49a3f84e8bce619ff28cc9158971b0e52c46c250b134f0c480724737dcc28730: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:4170bfc2cfa28b288cfcd60bb36f514f1c9205c022d23ecbdfc12da09ec16836: done |++++++++++++++++++++++++++++++++++++++| config-sha256:2f50386e20bfdb3f3b38672c585959554196426c66cc1905e7e7115c47cc2e67: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:3a0d728ec69d783850f9c318b97adfc9c6f726b7d075e0a22b8c2725ba0607e1: done |++++++++++++++++++++++++++++++++++++++| elapsed: 4.9 s total: 25.3 M (5.2 MiB/s) unpacking linux/arm64/v8 sha256:6aee00d0c7f4869144d1bdbbed7572cd55fd1a4d58fef5a21f53836054cb39b4... done: 384.1175ms root@demo-control-plane:/# ctr run --rm --privileged registry.k8s.io/kube-proxy:v1.32.0 kube-proxy sh -c "kube-proxy --cleanup && echo DONE" I0116 03:33:31.409626 7 server_linux.go:66] "Using iptables proxy" time="2025-01-16T03:33:31Z" level=warning msg="Running modprobe ip_vs failed with message: `modprobe: WARNING: Module ip_vs not found in directory /lib/modules/6.10.14-linuxkit`, error: exit status 1" DONE

root@demo-control-plane:/# iptables -S INPUT -P INPUT ACCEPT -A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes load balancer firewall" -j KUBE-PROXY-FIREWALL -A INPUT -m comment --comment "kubernetes health check service ports" -j KUBE-NODEPORTS -A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES -A INPUT -j KUBE-FIREWALL root@demo-control-plane:/# iptables -S OUTPUT -P OUTPUT ACCEPT -A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes load balancer firewall" -j KUBE-PROXY-FIREWALL -A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES -A OUTPUT -j KUBE-FIREWALL

The above chains are not removed. Should we document this?

nodes may not contain the docker command since kubernetes removed the dockershim support. nerdctl may be a good example than docker command. crictl is incompatible with docker command, it is hard for many users to start and run a container to do cleanup work without an example.

Compared to containerd and docker, crictl is not friendly to users in this task. I prefer to remove it.

it just boils down to preference. the users can just use what they want. crictl is indeed complicated but k8s owns a deb/rpm package for it and it's probably the most available on nodes. docker and nerdctl are probably missing but easy to use.
ctr is containerd specific and not available for cri-o users.

The above chains are not removed. Should we document this?

kubernetes/kubeadm#3133 (comment)
kubernetes/kubernetes#129639

sig network plan to fix it. i prefer that we don't have a temporary message about it in the docs.
but if you think it's better i can add it.

sig network plan to fix it. i prefer that we don't have a temporary message about it in the docs.

Agreed.

Is it possible to add podman as an alternative tool for cri-o? https://www.redhat.com/en/blog/crictl-vs-podman
Podman’s CLI is based on the Docker CLI giving users a familiar CLI experience.

i don't want us to be listing all possible tools and/or including examples for all of them, so i just updated this to:

Instead of Docker, you can use your preferred container runtime to start the container.

we assume that our users would know how to run a container on their setup, one way or another.

carlory · 2025-01-16T09:39:41Z

/lgtm
/assign @tengqm
for approval

k8s-ci-robot · 2025-01-16T09:39:48Z

LGTM label has been added.

Git tree hash: 627936f273e37d8f82416f748e9d6ff74867ba7b

content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md

SataQiu

/lgtm

neolit123 · 2025-01-17T16:27:58Z

@tengqm i applied the clarifications you requested.

content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md

tengqm · 2025-01-17T23:10:21Z

/approve
Thanks.

k8s-ci-robot · 2025-01-17T23:10:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: SataQiu, tengqm

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~content/en/docs/OWNERS~~ [tengqm]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

carlory · 2025-01-18T13:07:33Z

/lgtm

k8s-ci-robot · 2025-01-18T13:07:38Z

LGTM label has been added.

Git tree hash: 9567ddf529dc76e07471baa01ba117fad68b3d7f

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 15, 2025

k8s-ci-robot requested review from jbeda and sftim January 15, 2025 13:54

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. language/en Issues or PRs related to English language labels Jan 15, 2025

k8s-ci-robot requested review from carlory, pacoxu and SataQiu January 15, 2025 13:55

neolit123 mentioned this pull request Jan 15, 2025

improve "kubeadm reset" kube-proxy cleanup kubernetes/kubeadm#3133

Closed

neolit123 commented Jan 15, 2025

View reviewed changes

neolit123 force-pushed the 1.33-add-kubeadm-reset-note-about-kubeproxy-cleanup branch from e0905eb to d0e97a5 Compare January 15, 2025 13:59

neolit123 force-pushed the 1.33-add-kubeadm-reset-note-about-kubeproxy-cleanup branch from d0e97a5 to 79af26c Compare January 15, 2025 15:11

neolit123 mentioned this pull request Jan 15, 2025

kubeadm: improve message in 'reset' kubernetes/kubernetes#129644

Merged

pacoxu reviewed Jan 16, 2025

View reviewed changes

content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md Outdated Show resolved Hide resolved

carlory reviewed Jan 16, 2025

View reviewed changes

content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md Outdated Show resolved Hide resolved

carlory reviewed Jan 16, 2025

View reviewed changes

neolit123 force-pushed the 1.33-add-kubeadm-reset-note-about-kubeproxy-cleanup branch 2 times, most recently from dbeb24d to a09240a Compare January 16, 2025 09:32

k8s-ci-robot assigned tengqm and carlory Jan 16, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 16, 2025

tengqm reviewed Jan 16, 2025

View reviewed changes

content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md Outdated Show resolved Hide resolved

content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md Show resolved Hide resolved

content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md Outdated Show resolved Hide resolved

SataQiu approved these changes Jan 17, 2025

View reviewed changes

k8s-ci-robot assigned SataQiu Jan 17, 2025

kubeadm-reset: add notes about more manual cleanup steps

fc22259

neolit123 force-pushed the 1.33-add-kubeadm-reset-note-about-kubeproxy-cleanup branch from a09240a to fc22259 Compare January 17, 2025 16:27

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 17, 2025

k8s-ci-robot requested review from carlory, SataQiu and tengqm January 17, 2025 16:27

tengqm reviewed Jan 17, 2025

View reviewed changes

content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md Show resolved Hide resolved

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 17, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 18, 2025

k8s-ci-robot merged commit 9fc63c9 into kubernetes:main Jan 18, 2025
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm-reset: add notes about more manual cleanup steps #49448

kubeadm-reset: add notes about more manual cleanup steps #49448

neolit123 commented Jan 15, 2025

neolit123 commented Jan 15, 2025

neolit123 Jan 15, 2025

netlify bot commented Jan 15, 2025

netlify bot commented Jan 15, 2025 •

edited

Loading

pacoxu commented Jan 16, 2025

carlory Jan 16, 2025

carlory Jan 16, 2025

carlory Jan 16, 2025

carlory Jan 16, 2025

neolit123 Jan 16, 2025 •

edited

Loading

neolit123 Jan 16, 2025

carlory Jan 16, 2025

carlory Jan 16, 2025 •

edited

Loading

neolit123 Jan 16, 2025

carlory commented Jan 16, 2025

k8s-ci-robot commented Jan 16, 2025

SataQiu left a comment

neolit123 commented Jan 17, 2025

tengqm commented Jan 17, 2025

k8s-ci-robot commented Jan 17, 2025

carlory commented Jan 18, 2025

k8s-ci-robot commented Jan 18, 2025

		### Network traffic assets cleanup

		The `kubeadm reset` command does not clean any iptables, nftables or IPVS assets applied

kubeadm-reset: add notes about more manual cleanup steps #49448

kubeadm-reset: add notes about more manual cleanup steps #49448

Conversation

neolit123 commented Jan 15, 2025

neolit123 commented Jan 15, 2025

Choose a reason for hiding this comment

netlify bot commented Jan 15, 2025

✅ Pull request preview available for checking

netlify bot commented Jan 15, 2025 • edited Loading

✅ Pull request preview available for checking

pacoxu commented Jan 16, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neolit123 Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlory Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlory commented Jan 16, 2025

k8s-ci-robot commented Jan 16, 2025

SataQiu left a comment

Choose a reason for hiding this comment

neolit123 commented Jan 17, 2025

tengqm commented Jan 17, 2025

k8s-ci-robot commented Jan 17, 2025

carlory commented Jan 18, 2025

k8s-ci-robot commented Jan 18, 2025

netlify bot commented Jan 15, 2025 •

edited

Loading

neolit123 Jan 16, 2025 •

edited

Loading

carlory Jan 16, 2025 •

edited

Loading