File handling changes

[hellkite500]

When running a large run with BMI models that leak file handles, it is easy to exhaust the OS's available descriptors, which can lead to crashes with little context.

Additions

• Error message from the OS errno when BMI init config files are not readable.

Removals

Changes

• Change the FileStreamHandler to not hold each output's file descriptor for the duration of the program, but instead open and close as needed to write output.

Testing

1. All existing tests pass, tested with faulty BMI module on large regional domain.

Todos

• Consider updating other file reading checks with errno outputs.

Checklist

• PR has an informative and human-readable title
• Changes are limited to a single goal (no scope creep)
• Code can be automatically merged (no conflicts)
• Code follows project standards (link if applicable)
• Passes all existing automated tests
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
• Reviewers requested with the Reviewers tool ➡️

Target Environment support

• Linux
• MacOS

[donaldwj]

I would suggest adding flag that determines if we always close and file after each use. This change would have a rather high performance cost.

[PhilMiller]

Agreed with Donald that this is going to be a disaster for performance, especially running on any sort of HPC system with shared storage, and most especially Lustre.

That may not be a reason not to do this right now, and then optimize the performance impacts later if/as they arise, though.

[PhilMiller]

FileStreamHandler is used for what, CSV output? In the long run, that should go away anyway, in favor of NetCDF or Parquet or whatnot, but those will presumably have the same issues.

Thinking broadly, I don't think we can really expect to ship anything that does file-per-catchment output.