"Test suite failed to run" does not report as a test failure in the xml output

[phawxby]

Considering the following console output

 PASS  __tests__\src\common\services\index.test.js
 PASS  __tests__\src\common\services\support.test.js
 FAIL  __tests__\src\features\product\routers\product.js
  ● Test suite failed to run

    brand-engine\src\common\react-app\src\common\templates\cta.mustache:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){<a class="hello" href="{{ href }}">{{ hrefText }}</a>
                                                                                             ^

    SyntaxError: Unexpected token <

      4 | import FrameworkTemplate from '../../../common/framework-template';
      5 |
    > 6 | import * as cta from '../../../common/templates/cta.mustache';
      7 |
      8 | class Home extends React.Component {
      9 |       render() {

      at ScriptTransformer._transformAndBuildScript (node_modules/jest-runtime/build/script_transformer.js:307:17)
      at Object.<anonymous> (src/common/react-app/src/features/test/components/home.js:6:38)
      at Object.<anonymous> (src/common/react-app/src/app.js:6:39)
 PASS  __tests__\src\common\framework-engine\index.test.js

There is no output in the junit.xml referencing product.js test suite providing no indication it failed. Everything else is in there as expected.

[palmerj3]

@phawxby I think in this case the test runner actually stops no?

But this is a clear case where this is an "error" instead of a "failure" and, right now, Jest currently does not provide enough information to a testResultsProcessor to differentiate this case.

Do you have a github repo that reproduces this that I could look at?

[phawxby]

I've quickly thrown together a test case.
jest-junit-testcase.zip

Jest tests don't stop due to a suite failure unless running with the bail option.

Console

PS C:\Users\Paul\Documents\GitHub\jest-junit-testcase> npm run test
yarn run v1.5.1
$ jest
 FAIL  __tests__\failure.test.js
  ● Test suite failed to run

    Cannot find module './math' from 'failure.js'



      at Resolver.resolveModule (node_modules/jest-resolve/build/index.js:169:17)
      at Object.<anonymous> (src/failure.js:8:13)

 PASS  __tests__\maths.test.js
  √ add (7ms)
  √ subtract (1ms)

Test Suites: 1 failed, 1 passed, 2 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        2.286s
Ran all test suites.
error An unexpected error occurred: "Command failed.
Exit code: 1

junit.xml

<testsuites name="jest tests" tests="2" failures="0" time="0.374">
  <testsuite name="\__tests__\maths.test.js" errors="0" failures="0" skipped="0" timestamp="2018-04-05T08:49:29" time="0.374" tests="2">
    <testcase classname=" add" name="maths.test.js - add" time="0.004">
    </testcase>
    <testcase classname=" subtract" name="maths.test.js - subtract" time="0">
    </testcase>
  </testsuite>
</testsuites>

[phawxby]

Ah-ha! There's the problem. Let me see if I can figure out this magic then get a PR in.

[p00j4]

The setup fails with module errors but the xml generated says, all is fine.

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="jest tests" tests="3" failures="0" time="5.1">
  <testsuite name="&gt;&gt;&gt; TESTING ---  REDUCER FOR DROPDOWNS --- " errors="0" failures="0" skipped="0" timestamp="2019-03-04T12:24:02" time="2.269" tests="3">
    <testcase classname="&gt;&gt;&gt; TESTING --- REDUCER FOR DROPDOWNS ---  +++ Add new attr dropdown for an eventname " name="&gt;&gt;&gt; TESTING --- SEGMENT REDUCER FOR DROPDOWNS ---  +++ Add new attr dropdown for an eventname " time="0.004">
    </testcase>
    <testcase classname="&gt;&gt;&gt; TESTING --- REDUCER FOR DROPDOWNS ---  +++ Request for event data (attribute) " name="&gt;&gt;&gt; TESTING --- REDUCER FOR DROPDOWNS ---  +++ Request for event data (attribute) " time="0.001">
    </testcase>
    <testcase classname="&gt;&gt;&gt; TESTING --- REDUCER FOR DROPDOWNS ---  +++ Recieved attrs for a event" name="&gt;&gt;&gt; TESTING --- REDUCER FOR DROPDOWNS ---  +++ Recieved attrs for a event" time="0.004">
    </testcase>
  </testsuite>
</testsuites>

[manuel-woelker]

Currently experiencing the same issue with jest-junit 6.3.0

[jkblume]

Yes, experiencing the same with 6.3.0 - is there a release/version where this was resolved?

[palmerj3]

This is a known limitation of jest and not an issue with jest-junit. If a test suite fails to run then reporters are given very limited information. Not enough for me to be able to write to the xml, "hey test suite blah with test foo failed". If there was a syntax error or something else which prevents jest from even executing the test suite then reporters aren't given enough detailed information for me to be able to reuse the unique identifiers from previous runs (e.g. test suite -> test case name).

[jkblume]

okay. thanks for the information. i hoped for a solution like in other junit runners or reporters: If a test run fails, because of an syntax error, the junit xml report 'simply' does not get generated and i can use the stdout of the test runner to get information for the error.

[palmerj3]

@jkblume can you give me more information on that? What other junit runners are you referring to? And is jest-junit somehow blocking this information? You still see the test failures in stdout/stderr.

[jkblume]

@palmerj3 I am referring to runners in other languages, like in python (unittest-xml-reporting).

We need to further process the xunit xml to parse for informations we need to provide to our users on a test run (like a ci is doing it). Until now, is was enough to just look, if the test runner is generating a xml unit file. If it was generated, we can use the content, otherwise, we used the stdout/stderr of the test runner for further unstructured information why the test run, does not finished.

I think some ci test parsers will have problems to, of processing this xml (like the origin of this issue shows). But i understand, if it's not possible due to limitations of the jest itself.

[palmerj3]

Cool thank you!

So I'm hesitant to take this approach since many CI providers will only give you test feedback (passed/failed) if a junit file is generated. It sucks that those numbers won't always be accurate for the edge cases where a test file cannot be parsed but jest itself will always exit 1 in that case so any status checks will fail.

I'm going to table this for now but thank you for posting the issue and following up with additional details.

[rattrayalex-stripe]

@palmerj3 can we reopen this ticket? It's quite a surprising and unfortunate failure mode. I think the issue was closed because the PR to fix it was opened, but that PR didn't end up getting merged.

[palmerj3]

I closed the ticket because it is not fixable. If jest fails to run a test file then jest-junit is not given any information about the test suites within or the test cases within said file. So I don't have enough information in order to generate a junit.xml file that says "Test suite x,y,z failed".

[rattrayalex-stripe]

Ah, sorry I misunderstood the timeline.

It's certainly true that it's impossible to generate a junit.xml file that says "Test suite x,y,z failed" by the very nature of the bug.

However, I'm not sure that this is a necessary product requirement for this bug; users wish for their CI to fail and provide a clue as to where to look, and are likely to accept that a list of test suite names can't be provided if the test suite setup fails.

Internally, we have a workaround that generates a single testsuite, each with a single testcase, for each testResult with a testExecError containing the failureMessage as CDATA. From what I understand of a quick skim of the PR, it does a much more elegant job of a similar approach.

It may be uncomfortable to invent a "system-err" testsuite out of thin air, especially since it won't mesh with CI failure history (a feature we also use), but given that jestjs/jest#5958 hasn't made progress, it may be a better tradeoff for the time being. Users at least may be able to see a history of test suites which fail to run!

[palmerj3]

I definitely see your point but given that jest-junit allows teams to configure the naming convention of their suite names (based on data passed to us from jest when it executes the suites) there is no way I could reproduce that if the file fails to execute.

So in that case users would have a junit.xml file with a test suite that doesn't match any test suites seen before. That's my main concern. If you do have systems that track test history using junit.xml files then a feature like this could potentially corrupt that data.

But perhaps i'm missing something.

[asheq-svmx]

@palmerj3 Is there an open issue on the jest repo requesting that the information required to create a complete XML file be made available to jest-junit? I would like to follow it to be aware of any updates in this area. Thanks for your help!

[palmerj3]

There isn't an issue with jest that I'm aware of filing this as a bug or a feature request. But the issue I'm linking here is probably the most relevant to the core of the problem.

jestjs/jest#6616

[dl748]

I have this same issue, it is NOT a jest issue. I've tried with the jest-trx-results-processor plugin and it correctly reports all the errors. This plug does not report ANY errors and will always give a 100% success rate and not tell you the errors.

[palmerj3]

It's not exactly just a jest issue and not exactly just a jest-junit issue. A combination of both.

If your test suite fails to run and I write that information to a junit.xml file what would I write exactly? Jest does is not able to tell me what test suite(s) failed or what individual tests contain syntax errors (for example).

Only details I have as a reporter is what file path(s) failed to run.

And given that you can configure jest-junit to write junit.xml files in different ways, not necessarily containing file path, if I attempt to write anything it could just confuse whatever reports and records your tests.

This is why I don't do it. But if you can think of a way to approach this while retaining the users configuration then feel free to submit a PR.

[Blacktiger]

It's not exactly just a jest issue and not exactly just a jest-junit issue. A combination of both.

If your test suite fails to run and I write that information to a junit.xml file what would I write exactly? Jest does is not able to tell me what test suite(s) failed or what individual tests contain syntax errors (for example).

Only details I have as a reporter is what file path(s) failed to run.

And given that you can configure jest-junit to write junit.xml files in different ways, not necessarily containing file path, if I attempt to write anything it could just confuse whatever reports and records your tests.

This is why I don't do it. But if you can think of a way to approach this while retaining the users configuration then feel free to submit a PR.

How about making this an option for people who don't do any complicated configuration. Just drop the failed test suite name into the junit xml. If people don't like it, then they don't have to enable it.

[dl748]

I'm not sure how its a combination of Jest and JUnit, as other plugins that do the same thing have no problems with reporting this information. I've had to remove this plugin and use another to get this information for now.

[palmerj3]

It's described pretty clearly in this thread if you read it

[dl748]

That's fine. I guess if i can get the correct information, albeit in a different file format, which suite and test failed, along with the compilation errors, with a different plugin, I'm all good. All I had to do was write a quick converter from that format to the junit format. I was kind of hoping this would get fixed, so that i didn't have to use/write the converter.

[SamTheisens]

@dl748 @Blacktiger @asheq-svmx @rattrayalex-stripe @jkblume the issue has been addressed in v12.0.0.

[matthewoden]

I'm seeing this issue rearing it's ugly head again in a new way. Tests with errors are simply not recorded.

The test case is where I have a path issue:

    Cannot find module '../../../common/fetchJson' from 'src/components/component.test.js'

      2 | import { id, gridActions } from './Grid'
      3 | import { runSaga } from 'redux-saga'
    > 4 | import fetchJson from '../../../common/fetchJson'
        | ^
      5 |
      6 | jest.mock('../../../../common/fetchJson')
      7 |

      at Resolver.resolveModule (node_modules/jest-resolve/build/resolver.js:324:11)
      at Object.<anonymous> (src/components/component.test.js'.js:4:1)

Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        3.377 s
Ran all test suites matching /src\/components\/component.test.js/i.

Which outputs that zero tests ran, with zero failures. Using [email protected] and [email protected]:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="jest tests" tests="0" failures="0" errors="0" time="1.494">
</testsuites>

[SamTheisens]

I'm seeing this issue rearing it's ugly head again in a new way. Tests with errors are simply not recorded.

Did you set reportTestSuiteErrors?

"jest-junit": {
    "reportTestSuiteErrors": "true"
  }

[matthewoden]

@SamTheisens No, we sure didn't. Thank you.

[Deaniesaur]

Tried adding reportTestSuiteErrors in package.json and error count is displayed right.

However, if I add it using - jest.config.js -test suite's name is undefined and error count is still 0 instead of 1.

reporters: [
"default",
[ "jest-junit", { "reportTestSuiteErrors" : "true" } ]
],
testResultsProcessor: "jest-junit",

[palmerj3]

Try removing testResultsProcessor. If you add jest-junit as both a reporter and testResultsProcessor it likely nullifies the configs.