Allow multiple test reports

[reddwarf69]

What feature do you want to see added?

Warnings-NG has two parameters: "id" and "name" (https://www.jenkins.io/doc/pipeline/steps/warnings-ng/#recordissues-record-compiler-warnings-and-static-analysis-results) that allow you to have separate reports.

junit has "checksName", that allow you to have separate reports in GitHub... but those are mixed, all together, when you go back to Jenkins.

I would like to have the "id" and "name" options also in junit.

Upstream changes

No response

[timja]

could you give a bigger example? JUnit does use the 'enclosing block'.

So it will include e.g. the stage name in the final checks name

[reddwarf69]

I'm OK with how the Checks API is being used / GitHub Checks are being published.
The problem is in how different sets of tests are presented in the Jenkins UI.

This is the side of my job in the Jenkins UI.

"clang-tidy Warnings" and "clang-tidy servers Warnings" are both created by Warnings NG, but they are clearly split.

They are also split as GitHub Checks

And yes, "Tests" and "Server Tests" are also split as GitHub Checks. I can even see how many of each tests there are (160 "Tests" and 2 "Server tests")

But in the Jenkins UI there is only "Test Result", and if I click on it, I get

a single number: 162 test (160+2).

Since they are split as GitHub checks, I would like them split in the Jenkins UI too. And I can do that with the Warnings NG results with its "id" and "name" options.

    recordIssues 
                 tools: [
                   groovyScript(parserId: 'my_clangtidy', pattern: 'logs/build_full_static_analysis.log'),
                   groovyScript(parserId: 'my_clangtidy', id: 'my_clangtidy_servers', name: 'clang-tidy servers', pattern: 'logs/build_full_servers_static_analysis.log')
                 ]

I can also do it when creating the documentation too

  publishHTML([
    ...
    reportName: 'Server Documentation'
  ]

But I can't do it with the tests (and the coverage report, but that's a different issue in a different plugin).

FWIW this is my junit call: junit checksName: 'Server tests', testResults: 'junit.xml'

[uhafner]

This is an interesting point. Recently I thought that it might make more sense to remove the UI part of that feature from the warnings plugin and show only a single "warnings" link in order to remove the clutter. Using the new application bar one can start within the top level warnings link and then show the individual subsections of the warnings in separate sub-pages. (The same I am planning in the coverage plugin: one link and summary, but multiple results below this entry point). With the 1-link approach it means we would have a better hierarchical decomposition of the Jenkins builds results.

I understand that the current JUnit results view is kind of awful and has a poor UI that should get a facelift. So this feature does not work with the current implementation in the JUnit plugin and requires some extra work. What is not yet clear to me: is your feature request about the option to show filtered results using separate sidebar links, or is the actual requirement to implement a persistent filter of the JUnit results that still can be navigated from one single link to a page that has several results available (e.g. using tabs or any other separation)?

[reddwarf69]

I guess what I want is to have the results of two independent projects into a single Jenkins job, in a way as close as possible to "just twice what I already had when there was a single project".
But to be fair, the problem may be me having two independent projects not only in a single repository, but also building as a single Jenkins job.

Splitting it into multiple repos is not trivial. But I have actually just created a second Jenkinsfile in the repo, to build the second project. Since I now have two independent Jenkins jobs, I can have my "Tests" and "Server Tests" in the Jenkins UI, each one in a different job; which makes more sense to start with.
Also, it didn't make any sense to me for junit to be able to split the checks in GitHub but not in the Jenkins UI. It was either none or both, but not only one. Now that I have two Jenkins jobs, it makes sense! I need to change the name in one of the jobs to avoid a conflict, with two jobs sending the same "check" to GitHub for the same commit.

So... I was asking this because Warnings NG also had it, and it made sense for all the plugins to behave similarly. But since Warnings NG is going to remove it, and I have a better fix for my workflow, I guess this issue could be closed.

[timja]

I think fine to keep open for now.

The test reports should likely be named better / separated. I don't think archiving multiple ones is done terribly well

[yogeek]

we would also benefit from a better separation between different test reports in jenkins UI : is this proposition still active please ?

[Unknown-Guy]

We also were looking for a way to see multiple test results. Our idea was to split Unit tests and Feature tests into own stages, both stages then get post step with junit skipPublishingChecks: true, testResults: 'results/integration.xml', checksName: 'Integration Tests', similar for unit tests. But the results are combined in single Test result page and graph.

[kaddkaka]

We are also looking for having and supporting multiple result files. The reason is that we run our testsuite in a few different configurations and create one report for each run. The number of tests can become quite large (~50 k) which makes the html gui render very slowly.

I created an issue on the Jenkins JIRA but was redirected here. (https://issues.jenkins.io/browse/JENKINS-73594)

When having a long tests result, the Web GUI is too slow to handle it satisfactory. Therefore, the automatic merging of several junit test result xml files is unwanted behavior.

I have a pipeline that generates 10 different result files. It would be great if Jenkins could leave them untouched and just present all 10 as separate test results for the same build.

Similar feature was requested for different purpose here: https://stackoverflow.com/questions/41871285/using-multiple-junit-results-file-on-jenkins-pipeline