[Git HEAD] make check keeps exiting with code 0 despite failures

[hartwork]

Hi!

make check keeps exiting with code 0 despite failures which is unfortunate in context of CI and/or distro packaging. It would be great to exit with code 1 in case of (errors and/or) failures. Thank you!

Best, Sebastian

[hartwork]

@henry2cox commit 193cc76 made the CI rely on make check exiting with non-zero code in case of test failures a few hours ago. Was there some other commit that made make check do that or will the CI now just be green even not all tests were successful? That would defeat it's key mission, so I'm wondering. What do you think?

[henry2cox]

Sorry - not trying to confuse things (...but succeeding remarkably effectively) - the fix for exit status is line 161 in the test_run script - which I checked in with the other fixes, in 193cc76.
It should be doing what we want: if tests all pass: zero exit status. If something breaks: nonzero.

[hartwork]

@henry2cox cool! So we can close this as fixed and keep the CI calling make check as is?

[hartwork]

@henry2cox update: Found your message #343 (comment) now — closing as fixed…

[henry2cox]

I think so - yes.
And, in the meantime, we can see if we can make the artifact job more useful/usable - and more paranoid.
(Paranoia in regression/release testing is a Good Thing, IMO.)