-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use validation files for CI #371
base: master
Are you sure you want to change the base?
Conversation
…or comparison during the CI
…ardyn into validationUpdate
For now the CI fails due to very strict tolerance settings and the example |
@cniethammer @FG-TUM This PR introduces a new way of validation by comparing the simulation output to a reference file. However, in one case of the examples list, the results of the simulation depend on the number of MPI ranks. Does this make sense to you or is it a bug / race condition / etc.? |
Just some short feedback having a first look: |
Description
The way the CI for validation works is changed with this PR.
Formerly, the list of examples was run with the PR as well as the master and the result was compared. This could lead to some issues (see #319).
This is solved by comparing now to validation files that were generated in advance. This speeds up the CI and makes it easier to add items to the list of examples.
Additionally, the new solution checks if the results are consistent for various compilation options. This is done intrinsically since always the same validation files are used for comparison.
What is a good choice for the tolerances?
XX
XX
How Has This Been Tested?
Please describe the tests that you ran to verify your changes.
Documentation
There are two new scripts (both in the
checks
folder):validation_createJSON.py
to generate the validation files based on the Logger output of ls1validation_compare_files.py
to compare the output of a simulation run with the validation file. This script is also used in the CI.It is also possible to include the output of the ResultWriter provided that it gives parsable output (see #368).