support for reporting multiple executions of the same test in a single run

[signed]

Is this something you want to consider?
I know this feature from TestCafe

[stolp]

I consider support for this as mandatory.

[marcphilipp]

@signed @stolp Could you please elaborate?

Would this be multiple executions of a test framework or multiple runs of the same test within the same execution?

If the former, could this be represented by having multiple report files?

If the latter, is there a limitation in the current format that makes representing reruns impossible?

[mmichaelis]

Quoting TestCafe as referenced by @signed:

Quarantine Mode

Enable quarantine mode to eliminate false negatives and detect unstable tests. When a test fails, TestCafe quarantines it and repeats it until the test yields conclusive results.

This pattern in my experience quite typical for system and especially UI tests. Instead of failing after the first run, you give some "grace" to the test to repeat. And this applies to every single test.

In the end, you want to know:

• Has there been a successful execution?

  For TestCafe, it seems to be some "ratio" to configure. For us, it is, that the test is only repeated if failing, and only repeated a given number of times, in which it needs to be successful once.

• Did the test fail at all?

  Unstable tests are expensive (runtime) and may signal an instability of the AUT as well as of the test, which we may want to address at some time. Thus, at least we want to know if we have any tests which required such repetition and see, if we can do better (use a better Wait-Pattern in test or fix unstable AUT).

  For this, we need to have as much information as possible. Especially unstable/flaky tests are hard to debug locally, quite often (Heisenberg Bug). Thus, we collect not only logs of the tests, but, if feasible, also of the AUT (JConsole log, Server log) as well as screenshots.

Does this answer your question, @marcphilipp?

[mpkorstanje]

@mmichaelis the result element is open for extension. Would you be able to include the additional information about the separate executions of a test there?

https://github.com/ota4j-team/open-test-reporting/blob/main/schema/src/main/resources/org/opentest4j/reporting/schema/core.xsd#L78

[stolp]

@marcphilipp: I was referring to the latter, multiple runs of the same test in a single test execution.

Regarding your question, I have no idea if this is possible in the current format as I have not yet used it.

I was simply deducting this as a fact from the issue heading.