Stop using system-level certificates directories (i.e. /etc/pki, /etc/ssl/certs, and /etc/ca-certificates directories)

[rcythr]

I posted in slack the following suggestion; @neolit123 suggested we log an issue to track/discuss it here instead:

Issues #279 , #1367, and #1665 all share a common theme regarding the troublesome /etc/pki, /etc/ssl/certs, and /etc/ca-certificates directories.

I suggested in the sig-cluster-lifecycle slack of moving the ca certificates directory which kubeadm uses to something which kubeadm controls (e.g. /etc/kubernetes/ca-certificates/ or similar) instead of using the system controlled directory. Doing this would immediately allow all of these issues to have straightforward resolutions:

• The technical aspects of Document kubeadm usage with SELinux #279 can be resolved via chcon, starting with container_sandbox_t and eventually moving to nicely constrained custom types.
• Support non-root control-plane services #1367 has hostPath entries which can be properly mounted after chown/chmod for a kubernetes user/group.
• caCertsExtraVolumePaths should point to the correct distro paths #1665 becomes a no-op to resolve as we no longer need to know where the particular distro in use puts the certificate files, nor whether they're stored alongside private keys.

/assign @randomvariable @neolit123
/area security
/kind design

[sftim]

It sounds good. I don't know enough about this topic to really get my head around the suggestion though.

At the moment you can have a single root CA, even managed outside the cluster, with the root CA published to the system CA certificate store on machines that care about such things.

It sounds like this change still fits in OK with that single root CA approach. I want to check this is currently feasible and will stay feasible:

• the private key for the root CA is locked away, offline
• the root CA's certificate is in the system-level CA directory, and kubeadm is not managing this root CA
• the CA certificates for the etcd CA and the API server CA are inside /etc/kubernetes/pki, pretty much everywhere connected with the cluster.
• On the API server(s), the private keys for the intermediate API server CAs are also inside /etc/kubernetes/pki/.

The model I have is that if Kubernetes is using a internal CA, and this proposal happens, it's almost the same layout as with an offline root and intermediate CAs, only now the Kubernetes-specific root CA lives in (eg) /etc/kubernetes/ca-certificates/, and kubeadm does manage the root CA.

[rcythr]

• the root CA's certificate is in the system-level CA directory, and kubeadm is not managing this root CA

This use case would be impacted as part of this proposed change. With the change, any necessary external CA root certificates would need to be placed in the kubeadm-managed directory (e.g. /etc/kubernetes/ca-certificates/). Using the system-level directory isn't feasible unless kubeadm intends to get into the "distro mess". This is the problem being discussed on #1665 .

[randomvariable]

If we do this, we'll start being in the business of managing root certificates for the controller-manager cloud provider client code. Is that something kubeadm or the kubernetes project wants to be responsible for?

In addition, organisations that are using proxies may know how to run the distro appropriate commands to include their organisation root CA, but this would break again for kubernetes.

Let's disambiguate a few things:

At the moment you can have a single root CA, even managed outside the cluster, with the root CA published to the system CA certificate store on machines that care about such things.

#1665 was only about the root CAs for clients of external services. Right now, this is controller manager talking to cloud providers. The out-of-tree cloud providers will eventually need this.

In addition, single CA roots for the entire control plane are not desirable. kubeadm does the right thing by producing independent CAs for etcd, API server, front-proxy etc...

The technical aspects of #279 can be resolved via chcon, starting with container_sandbox_t and eventually moving to nicely constrained custom types.

Distro container-selinux packages should be responsible for allowing containers to mount the relevant root CA cert directories.

[neolit123]

If we do this, we'll start being in the business of managing root certificates for the controller-manager cloud provider client code. Is that something kubeadm or the kubernetes project wants to be responsible for?

i think we shouldn't as this falls outside of the "minimal viable cluster" case. kubeadm already manages quite a big number of certificates and we should avoid including more.

In addition, organisations that are using proxies may know how to run the distro appropriate commands to include their organisation root CA, but this would break again for kubernetes.

this can be a problem, yes.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/lifecycle frozen

[neolit123]

@rcythr

i revisited these two issues:
#1692 #1665

currently we are mounting "some" folders in the CP component containers so they can use them.
these components are built with golang and we have no control over how it constructs the bundle of system root CAs.

https://golang.org/src/crypto/x509/root_linux.go
https://github.com/golang/go/blob/2ebe77a2fda1ee9ff6fd9a3e08933ad1ebaea039/src/crypto/x509/root_unix.go

currently golang does not expose a way to return the folders / files it found (no public methods for that).
but if we had these files / folders we could have made kubeadm mount them in the CP static pods.

I suggested in the sig-cluster-lifecycle slack of moving the ca certificates directory which kubeadm uses to something which kubeadm controls (e.g. /etc/kubernetes/ca-certificates/ or similar) instead of using the system controlled directory. Doing this would immediately allow all of these issues to have straightforward resolutions:

so the question here is how to find the paths that will be used for the construction of the root CA bundle and move it to a common location, while avoiding the distro mess and forking golang?

instead of moving these path contents to a common kubeadm maintained location, we might as well just mount them directly.
#1367 is not an issue since non-root components can still mount root owned paths.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[neolit123]

closing as per the discussion above and given nothing actionable was done in the past 5 years.
#1692 (comment)