You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Run the test runner using Maven with the command mvn clean test. There is used a random flakyness to illustrate the behaviour.
Observe the generated report for missing tags and mismatched titles.
Expected Behavior
After retrying failed scenarios and updating the report:
The tags from the Examples tables should remain associated with the corresponding scenarios.
The titles in the report should correctly match the scenarios that were retried.
Actual Behavior
After retrying failed scenarios and updating the results:
The tags from the Examples tables are missing in the final report.
There is a mismatch in the report titles, where a failed scenario from one Examples table incorrectly shows as another scenario from a different table.
Environment
Karate Version: 1.4.1
Java Version: openjdk 11.0.14.1 2022-02-08 LTS
Additional Information
The mismatch in titles and missing tags only occur after the retry logic is executed.
The initial report generated before the retry does not have these issues.
Relevance of the Bug
The tags in our Karate test scenarios are not merely for organizational purposes; they serve a critical role in integrating our test automation framework with our test management system. Each tag corresponds to a unique identifier in Xray, which allows us to map automated test results directly to the relevant test cases in our management system. The loss of tags from the Examples tables after retrying scenarios disrupts this integration.
Solution Hints
The problem with the titles appears to be related to how the updateResults method in the Suite class compares existing scenarios to determine if they are equal. The comparison logic, which relies only on the section index and the example index, can erroneously consider different scenarios as equal under certain conditions.
Mismatched Titles
The updateResults method (here) uses the equals method of the Scenario class to match scenarios. The isEqualTo method (here) checks if the section index and the example index are the same. This can lead to a situation where different scenarios from separate Example tables are incorrectly identified as the same because they share the same index positions within their respective tables.
A potential solution would be to refine the comparison logic to include a more distinctive identifier that takes into account the unique context of each scenario, ensuring that scenarios from different Example tables are not conflated.
Missing Tags
The issue with the missing tags after retrying scenarios is still under investigation.
Attachments
karate-reports.zip: The karate-reports folder containing the generated report after the retry, showcasing the reported bug.
Screenshot from the Report where we see, that the Example Table Tags are missing, Example 3 is missing and Example 6 is duplicated.
The text was updated successfully, but these errors were encountered:
Owlwasrowk
changed the title
Mismatched Titles and Missing Tags in Reports After Retrying Scenarios
Mismatched Step Titles and Missing Tags in Reports After Retrying Scenarios
Jan 25, 2024
@Owlwasrowk thanks for the details. Karate doesn't yet support retries and this issue is a great example of what all edge cases come in the way. the best option here is for someone to go through the details (I think you have done some good analysis already) and contribute a PR
@ptrthomas Thank you for your prompt response. I understand that Karate does not currently support retries out-of-the-box, and the complexities that come with implementing such a feature.
I appreciate your suggestion to contribute a PR to address this issue. The missing tags may be related to the Suite object not retaining information about the scenario's originating example table during retries.
If you have any starting points or insights that could guide the development of a solution, I would be grateful for your advice.
but anyone with Java experience should be able to dive in. I think you are spot on with the equals implementation of scenario. we are missing the concept of what I propose to call "examplesParentIndex", ideally I would rename "examplesIndex" to "examplesRowIndex" and use "examplesIndex" for what's missing
Summary
Two issues have been observed after retrying failed scenarios in a test suite with a Scenario Outline that includes multiple Examples tables:
Steps to Reproduce
mvn clean test
. There is used a random flakyness to illustrate the behaviour.Expected Behavior
After retrying failed scenarios and updating the report:
Actual Behavior
After retrying failed scenarios and updating the results:
Environment
Additional Information
Relevance of the Bug
The tags in our Karate test scenarios are not merely for organizational purposes; they serve a critical role in integrating our test automation framework with our test management system. Each tag corresponds to a unique identifier in Xray, which allows us to map automated test results directly to the relevant test cases in our management system. The loss of tags from the Examples tables after retrying scenarios disrupts this integration.
Solution Hints
The problem with the titles appears to be related to how the
updateResults
method in theSuite
class compares existing scenarios to determine if they are equal. The comparison logic, which relies only on the section index and the example index, can erroneously consider different scenarios as equal under certain conditions.Mismatched Titles
The
updateResults
method (here) uses theequals
method of theScenario
class to match scenarios. TheisEqualTo
method (here) checks if the section index and the example index are the same. This can lead to a situation where different scenarios from separate Example tables are incorrectly identified as the same because they share the same index positions within their respective tables.A potential solution would be to refine the comparison logic to include a more distinctive identifier that takes into account the unique context of each scenario, ensuring that scenarios from different Example tables are not conflated.
Missing Tags
The issue with the missing tags after retrying scenarios is still under investigation.
Attachments
karate-reports
folder containing the generated report after the retry, showcasing the reported bug.Screenshot from the Report where we see, that the Example Table Tags are missing, Example 3 is missing and Example 6 is duplicated.
The text was updated successfully, but these errors were encountered: