Multilevel artifact

[pgrete]

For the INCITE runs (and also by @mfournier01 at smaller scale), some multi level artifacts started to show up recently:

This is data directly extracted from the output files (so it's not a plotting issue).

Interestingly, whatever happens at the boundaries does not grow/extend fast, e.g., it takes several thousands of cycles to show up and even after several thousand more cycles it's still contained to only a few cells width.

The issue can reliably be reproduced when restarting a specific INCITE sim (with binary probably corresponding to 3ce0a88) from last year with a current binary.

Data being read and immediately written does not show the artifact indicating that both reading and writing data is not the cause of the issue.
I also tried to go back to the 1D limited prolongation (versus the current 3D limited one), but this didn't change anything.
Also given that it does not show up for the old version, using cons for prolongation/restriction should also not be the root cause.

I have not been able to reliably reproduce on smaller scales/different pgen.
So next debugging steps could be

• smaller reproducer with same physics
• then/or reduce/limit pgen/physics
• bisect between current and 3ce0a88