[WIP] KEP 5105: kubectl top pvc

[tsmetana]

• One-line PR description: Add a new kubectl top persistentvolumeclaim subcommand that would allow to display volume usage of PVCs

• Issue link: kubectl top persistentvolumeclaim #5105

• Other comments:

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tsmetana
Once this PR has been reviewed and has the lgtm label, please assign eddiezane for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/sig-cli/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tsmetana]

/cc @ardaguclu
/cc @gmeghnag

[k8s-ci-robot]

@tsmetana: GitHub didn't allow me to request PR reviews from the following users: gmeghnag.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @ardaguclu
/cc @gmeghnag

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[googs1025]

/cc

[googs1025]

IIUC, kubectl top pod and kubectl top node are obtained by the metrics server definition API, not directly from prometheus. If we want to support kubectl top pvc, what we want is to get it directly from prometheus? 🤔

[googs1025]

FYI: https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/resource-metrics-api.md

[tsmetana]

IIUC, kubectl top pod and kubectl top node are obtained by the metrics server definition API, not directly from prometheus. If we want to support kubectl top pvc, what we want is to get it directly from prometheus? 🤔

By using Prometheus query we can get the "most interesting" value (usage percentage) with just one call -- it is simple. Though I understand the Prometheus dependency is a disadvantage.

[ardaguclu]

As @googs1025 mentioned, the implementation needs to be same with top node and top pod. Direct Prometheus query would be very non-standard (and does not work for some cases like exec credentials plugins, etc.).

Good part is following the same patterns of these 2 commands would make the implementation easier.

[googs1025]

@tsmetana We have discussed similar issues in other issues, you can also refer to: 🤔
FYI: kubernetes/kubectl#1696
FYI: kubernetes-sigs/metrics-server#1623

[tsmetana]

OK. I removed the Prometheus references. I believe the same constraints apply for the metrics-server, so the change is not that big in the end.

[tsmetana]

After going through the documentation, I wonder what would be the benefit of using the metrics API. The metrics-server README states:

Metrics Server is meant only for autoscaling purposes. For example, don't use it to forward metrics to monitoring solutions, or as a source of monitoring solution metrics. In such cases please collect metrics from Kubelet /metrics/resource endpoint directly.

And what we're trying to achieve sounds more like the "for example": just monitoring. Reading the metrics directly from kubelet would remove the necessity of running the metrics-server altogether. Yes, it's lighter dependency than Prometheus, but still an additional dependency that doesn't look necessary. In the light of the above quote I even wonder if the existing kubectl top implementation is doing the right thing by depending on the metrics API for simple resource usage querying and whether adding APIs for volume usage metrics would not be too heavyweight solution -- autoscalers have no use for them at all.

[toVersus]

A similar KEP had been proposed in the past at #497, but it was closed without being implemented. Eventually, someone created a kubectl plugin (yashbhutwala/kubectl-df-pv) instead. Implementing it as a kubectl plugin might be one option.

If this feature depends on the Metrics API (where the metrics-server retrieves metrics from the kubelet and exposes them via the Metrics API), I think we need to discuss it with the SIG-Instrumentation (owner of the Metrics API and metrics-server subprojects) and get their agreement. See kubernetes-sigs/metrics-server#1318.

Here are some points I’m personally curious about:

• What are the required permissions to execute kubectl top pvc? Read access to Pods and PVCs?
• Not all PVs are mounted as file systems on Nodes. In the case of PVs using a FUSE CSI driver, what would be the expected result of kubectl top pvc?

[googs1025]

• Not all PVs are mounted as file systems on Nodes. In the case of PVs using a FUSE CSI driver, what would be the expected result of kubectl top pvc?

This is a good point. 🤔 There may be multiple types of PVs and support for third-party to do their own, which may make this implementation more complicated? In addition: kubelet_volume_stats_used_bytes and
kubelet_volume_stats_capacity_bytes are both compatible and feasible for the current mainstream PVs?

[tsmetana]

* What are the required permissions to execute `kubectl top pvc`? Read access to Pods and PVCs?

I guess that would depend on the implementation. The user would have to be able to get to the kubelet stats somehow. Either directly or potentially through the metrics-server. But I would really just keep the implementation confined to present statistics that are already present in the cluster.

• Not all PVs are mounted as file systems on Nodes. In the case of PVs using a FUSE CSI driver, what would be the expected result of kubectl top pvc?

This is a good point. 🤔 There may be multiple types of PVs and support for third-party to do their own, which may make this implementation more complicated? In addition: kubelet_volume_stats_used_bytes and kubelet_volume_stats_capacity_bytes are both compatible and feasible for the current mainstream PVs?

The volume stats are being collected by cAdvisor AFAIK, so whatever it supports would work. That's also where potential new volume types support should come from. Which is another reason I'd like to really stick to using the existing metrics. I assume (I have some data from OpenShift cluster so I'm reasonably certain about it) they'd cover vast majority of use cases.