Excluding non-zero exit codes from Relative Time comparison.

[zamu-flowerpot]

I usually run a few sets of parameter lists which quickly explodes in number of instances (which is working as intended!). However, all the runs are shown in the summary at the end including those that exited with an non-zero exit code.

Looking at the code base, adding a filter in src/benchmark/relative_speed::compute to remove results with non-zero exit codes from the output should fix it.

I'm not really sure if there are more dependencies on the compute function elsewhere though or I would have just pushed a PR.

---

PS. Thanks for the all the great software! I use hyperfine, fd, and bat all the time!

[Raphalex46]

I was looking for an issue about this, so I'm happy I found yours !

Another case is when you use another command to populate a parameter list for example, and the output is a little bit off, it can be frustrating to have run all of the benchmarks for nothing because of an error like this, but using -i is not ideal because you don't necessarily want the failed command's results in the final output.

However, it seems that including non-zero exit code might still be relevant in some cases, so this should be another flag right ?

(I'm not really familiar with hyperfine besides using it, this is just a thought I had)

[sharkdp]

Thank you for your feedback.

However, it seems that including non-zero exit code might still be relevant in some cases, so this should be another flag right ?

I think so. There are valid use cases where we want to benchmark a program that always returns with a non-zero exit status. So we want it to be included in the comparison. For example, I was once benchmarking fd vs. find in a folder where I did not have full permissions. This leads find to exit with status 1. But fd exits with status 0 in that case.

What we do have is the possibility to retrieve exit codes in post-processing (e.g. through the Python scripts in scripts/). So if you export benchmark results to JSON, you will see the exit code there:

{
  "results": [
    {
      "command": "fd",
      "mean": 0.010896327180000002,
      "stddev": 0.00019708197362518857,
      "median": 0.010896327180000002,
      "user": 0.0025372600000000004,
      "system": 0.00880242,
      "min": 0.010756969180000003,
      "max": 0.011035685180000001,
      "times": [
        0.010756969180000003,
        0.011035685180000001
      ],
      "exit_codes": [
        0,
        0
      ]
    },
    {
      "command": "find",
      "mean": 0.0018015931800000004,
      "stddev": 0.00024363505567138774,
      "median": 0.0018015931800000004,
      "user": 0.00223126,
      "system": 0.0,
      "min": 0.00162931718,
      "max": 0.0019738691800000006,
      "times": [
        0.0019738691800000006,
        0.00162931718
      ],
      "exit_codes": [
        1,
        1
      ]
    }
  ]
}

But I understand you would like to see this feature included in hyperfine itself. If so, I would appreciate any help in designing this feature.

• What possible use cases are there?
• Can we support all use cases without adding new command line options?
• If not, how can we design the new CLI to support all (future) use cases regarding exit codes?
• …

[Raphalex46]

For use cases, I don't know if I can think of anything else outside of what @zamu-flowerpot already mentioned (and maybe messing up the parameter lists like I said).

I can't really think of a way to reconcile "ignore non-zero exit codes" and "exclude runs with non-zero exit codes from the result" in a single option (but again, I'm not that familiar with the project).
Maybe a new option like "exclude-failure" could be added, and for generality, maybe both "ignore-failure" and "exclude-failure" could take an optional list of error-code to ignore/exclude as an argument (the default being "exclude/ignore all non-zero exit codes").

Let me know what you think :)

[SgtPooki]

What we do have is the possibility to retrieve exit codes in post-processing (e.g. through the Python scripts in scripts/). So if you export benchmark results to JSON, you will see the exit code there:

I needed exactly this. Thanks @sharkdp, I was able to get the exact flakiness of a test I was debugging by doing:

hyperfine 'npm run test' --show-output --ignore-failure --export-json=./testRuns.json --runs=10 | tee testRuns.txt
cat testRuns.json | jq '.results[0].exit_codes | {total: . | length, failed: map(select(. == 0|not)) | length }'