Retries and flaky tests

[pandoras-toolbox]

Does the open-test-reporting format support to display that a test has been retried?

The legacy XML test output supports it, for instance Maven Surefire produces:

<testcase name=".." classname=".." time="0.1">
  <flakyFailure message="exception message" type="assertion exception">
    <stackTrace>flaky failure stack trace</stackTrace>
    <system-out> flaky failure System.out log message </system-out>
  </flakyFailure>
  <system-out> success </system-out>
</testcase>

Tools like Allure Report can identify flaky tests with that information and correctly display retries of a whole test.

I am not sure where the specification for flakyFailure originates from, and if there is also a flakySuccess, meaning that after several retries the test finally succeeded. But these are good test reporting features, not so much for unit tests, more for higher level tests.

Of course retrying a whole tests should not be done in normal circumstances, but sometimes it is good to retry them automatically, especially for end-to-end tests.