[Feature Request] Optimize the api _cat/nodes

[kkewwei]

Is your feature request related to a problem? Please describe

Now the method is as follows:

        return channel -> client.admin().cluster().state(clusterStateRequest, new RestActionListener<>(channel) {
                     ......
                    nodesInfoRequest.timeout(request.param("timeout"));
                    client.admin().cluster().nodesInfo(nodesInfoRequest, new RestActionListener<NodesInfoResponse>(channel) {
                               ......
                               // wait all the nodes response
                               nodesStatsRequest.timeout(request.param("timeout"));
                              client.admin().cluster().nodesStats(nodesStatsRequest, new RestResponseListener<NodesStatsResponse>(channel) {
                                      ......
                               }
                    }
        }

It seems has two problems:

1. cluster().nodesInfo() and cluster().nodesStats() use separate timeout, in that case, if timeout from the client is 30s, without adding cluster().state(), the overall time can be 60s, which is 2x times that the expect.
2. Only all nodes return the a NodeInfoResponse in cluster().nodesInfo() can the next cluster().nodesStats() be called. It's normal to have a slow node(such as fullGc) in large clusters, the api will become unresponsive, it means that if some of nodes are blocked in cluster().nodesInfo(), the overrall api will be blocked.

Describe the solution you'd like

1. If timeout is 30s in _cat/nodes, the overall time should be around 30s.
2. If some of nodes are blocked, it doesn't affect the rest nodes to get metrics. Each node separately call cluster().nodesInfo() and cluster().nodesStats().

The code can be like this:

        long time1 = threadPool.relativeTimeInMillis();
        return channel -> client.admin().cluster().state(clusterStateRequest, new RestActionListener<>(channel) {
                     ......
                    long time2 = threadPool.relativeTimeInMillis();
                    nodesInfoRequest.timeout(timeout - (time2-time1)));
                    for (String nodeId : nodeIds) {
                         nodesInfoRequest.nodesIds(nodeId);
                          client.admin().cluster().nodesInfo(nodesInfoRequest, new RestActionListener<NodesInfoResponse>(channel) {
                                    ......
                                    long time3 = threadPool.relativeTimeInMillis();
                                    nodesStatsRequest.timeout(timeout - (time3-time1)));
                                    nodesStatsRequest.nodesIds(nodeId);
                                   client.admin().cluster().nodesStats(nodesStatsRequest, new RestResponseListener<NodesStatsResponse>(channel) {
                                         ......
                                    }
                           }
                    }
                    channel.sendResponse(RestTable.buildResponse(buildTable(fullId, request, clusterStateResponse, nodesInfoResponse, nodesStatsResponse), channel));

        }

Related component

Cluster Manager

Describe alternatives you've considered

No response

Additional context

No response

[shwetathareja]

cc : @aasom143 who was looking into holistic API cancellation across different cat and cluster APIs

[rajiv-kv]

[Triage - attendees 1 2 3 4] - @kkewwei Thanks for creating the issue.
Please check if this can leverage the cancellation framework 13908

[kkewwei]

[Triage - attendees 1 2 3 4] - @kkewwei Thanks for creating the issue. Please check if this can leverage the cancellation framework 13908

@rajiv-kv The the cancellation framework can be used here to solve the first problem, It doesn't solve the second problem.

I wold like to solve the two of the problems within the cancellation framework, cc @aasom143.

[aasom143]

Hi @kkewwei, thanks for following up. With the new cancellation framework, we have added a new timeout(cancel_after_time_interval) that can be used to address the first problem. For the second issue, we already have a timeout which can configured for each node's transport call. By setting this timeout, we can prevent our requests from being blocked by a faulty node. I hope this provides clarity on how to resolve the second problem.

[aasom143]

To address the first issue, could you please add cancellation support for the cat/nodes API? You can refer to the recent PR regarding cancellation support for the cat/shards API.

[kkewwei]

To address the first issue, could you please add cancellation support for the cat/nodes API? You can refer to the recent PR regarding cancellation support for the cat/shards API.

Of course, thank you.

[kkewwei]

@aasom143, It's ok now, please have a look when you are free.#14853