Optimize NodeIndicesStats output behind flag

[Pranshu-S]

Description:

As of today the APIs which fetch Node Stats with indices stats currently returns all shard statistics regardless of the requested level (node, level, or indices) to the coordinator node (node where-in or from which the rest request was created). This unnecessary iteration on the coordinator node leads to overhead and potential health issues. Furthermore, the coordinator node repeats index-level statistics calculation for each node response even when the requested level is indices.

This pull request proposes optimizations to the NodeStats API to pre-compute and return only the necessary information to generate the response. The APIs would now perform pre-computation of shard or index-level statistics on remote nodes (individual nodes which respond to the coordinator node) based on the level parameter in the REST request - nodes, indices or shards. Only for level = shards does the response contains the shard level stats.

The optimisation hides behind a flag transport level flag optimizeNodeIndicesStatsOnLevel part of CommonStatsFlag. This is enabled in RestNodeStatsAction as well as RestNodeAction API calls as we do not require it during these code paths.

Pre-computation on remote nodes minimizes data transfer and processing on the coordinator node, leading to significant performance gains, especially in large clusters. Reduced load on the coordinator node contributes to a healthier state by minimizing GC pauses and CPU spikes.

Testing:

Extensive testing with a 20k shards cluster has validated the effectiveness of these optimizations.

Stats	Without Optimisation	With Optimisation
Average	0.857280048	0.4961762
P90	1.1514673	0.7631599
Max	1.78542	1.423496
Min	0.444184	0.212655

Related Issues

Resolves #13340

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 59a0f5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for f6d3b58: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for b741868: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for a267aef: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 517baa5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for e009a84: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 857e7e1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 9876c13: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 61bf2e0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 73de188: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[Pranshu-S]

[Test Result](https://build.ci.opensearch.org/job/gradle-check/42419/testReport/) (3 failures / -5)
[org.opensearch.discovery.ClusterDisruptionIT.classMethod](https://build.ci.opensearch.org/job/gradle-check/42419/testReport/junit/org.opensearch.discovery/ClusterDisruptionIT/classMethod/)
[org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureLastSuccessfulSettingsUpdate](https://build.ci.opensearch.org/job/gradle-check/42419/testReport/junit/org.opensearch.index/ShardIndexingPressureSettingsIT/testShardIndexingPressureLastSuccessfulSettingsUpdate/)
[org.opensearch.indices.replication.SegmentReplicationIT.testReplicaAlreadyAtCheckpoint](https://build.ci.opensearch.org/job/gradle-check/42419/testReport/junit/org.opensearch.indices.replication/SegmentReplicationIT/testReplicaAlreadyAtCheckpoint/)

For org.opensearch.discovery.ClusterDisruptionIT.classMethod

java.lang.AssertionError: SpanData validation failed for validator org.opensearch.test.telemetry.tracing.validators.AllSpansAreEndedProperly
MockSpanData{spanID='aa1534630900e459', parentSpanID='f8c436a5e84c915f', traceID='d7fd33dd4c374128', spanName='dispatchedShardOperationOnPrimary', startEpochNanos=2792950755631, endEpochNanos=0, hasEnded=false, attributes={index=restart_while_indexing, refresh_policy=false, thread.name=opensearch[node_t1][write][T#1], bulk_request_items=1, shard_id=0, node_id=uRqO2n91TTCp5ocLhE5MQA}, stackTrace=null}

Similar to failures in Flaky Test Issue - #14308

---

For org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureLastSuccessfulSettingsUpdate

java.lang.AssertionError: expected:<0> but was:<2>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.opensearch.index.ShardIndexingPressureSettingsIT.lambda$waitForTwoOutstandingRequests$13(ShardIndexingPressureSettingsIT.java:939)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1128)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1101)
	at org.opensearch.index.ShardIndexingPressureSettingsIT.waitForTwoOutstandingRequests(ShardIndexingPressureSettingsIT.java:938)
	at 

Similar failures in Flaky Test Issue - #14331

---

For org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureLastSuccessfulSettingsUpdate
Flaky Test - #14328

[github-actions]

✅ Gradle check result for fa2373f: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 81.63265% with 18 lines in your changes missing coverage. Please review.

Project coverage is 71.83%. Comparing base (acee2ae) to head (40e0fc9).
Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
.../java/org/opensearch/indices/NodeIndicesStats.java	85.71%	7 Missing and 5 partials ⚠️
...in/java/org/opensearch/indices/IndicesService.java	25.00%	2 Missing and 1 partial ⚠️
...h/action/admin/indices/stats/CommonStatsFlags.java	75.00%	0 Missing and 2 partials ⚠️
...rg/opensearch/rest/action/cat/RestNodesAction.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14454      +/-   ##
============================================
- Coverage     71.87%   71.83%   -0.04%     
- Complexity    63318    63415      +97     
============================================
  Files          5231     5244      +13     
  Lines        296521   296864     +343     
  Branches      42832    42868      +36     
============================================
+ Hits         213113   213250     +137     
- Misses        65948    66097     +149     
- Partials      17460    17517      +57     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

✅ Gradle check result for 15f890d: SUCCESS

[github-actions]

❌ Gradle check result for 2f4187f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 40e0fc9: SUCCESS