MPI Hang at first file output under SMR

[bprather]

When any HDF5 outputs are enabled in KHARMA (i.e. dumps and/or restarts), and it is run with multiple MPI ranks, it can hang at the first file write (just after printing Running X driver...).

This appears to be an issue with Parthenon, possibly introduced by recent MPI changes for new features. Parthenon PR for a possible fix is here: parthenon-hpc-lab/parthenon#979

Disabling all HDF5 outputs avoids this issue, and as it "only" happens ~80% of the time it can sometimes be circumvented just by repeatedly restarting the same run. Disabling SMR (either not refining or using AMR) generally also fixes the issue or at least makes the race condition much less frequent. Might also be an issue of the number of blocks/blocks per rank, though.

[bprather]

This is fixed by the updated form of a Parthenon PR we merged: parthenon-hpc-lab/parthenon#963

Remaining CI failures of feature/parthenon-bump are actual KHARMA bugs, quick stuff related to boundary conditions.