Reduce the flaky tests impact by 90%

[joperezr]

Objective: Systematically address and fix the flaky tests by 90%.

Tasks:

• Pull metrics to identify the most problematic flaky tests.
• Prioritize and fix the tests based on their impact.
• Provide regular updates on progress and metrics.
• Collect and report metrics such as time, number of failures, and successful jobs in order to be able to create a curve to visualize the progress in reducing flaky tests.
• Create a curve to visualize the progress in reducing flaky tests.

[davidfowl]

Looking at the stats there are some things that stand out:

https://dev.azure.com/dnceng-public/public/_test/analytics?definitionId=274&contextType=build

We should delete WithDataShouldPersistStateBetweenUsages tests in general (not just elastic search) or find a way to make them more reliable. These tests are super sketchy and do things like

aspire/tests/Aspire.Hosting.Elasticsearch.Tests/ElasticsearchFunctionalTests.cs

Line 93 in be90451

DockerUtils.AttemptDeleteDockerVolume(volumeName, throwOnFailure: true);

Looking at all tests sorted by duration:

The starter template should be a separate job and we should decide what to do with these > 2 minute tests. (PS notice the slow ones are mostly the hosting tests).

I tried messing with a github action designed to help isolate and investigate these tests (#7073).

Ideally, we can work in a way where we dont need an hour per run to diagnoses test failures by staring at lots of console output.

[davidfowl]

Taking this one