TST, BENCH: memory profiling/testing #817
Labels
Comments
tylerjereddy
added a commit
to tylerjereddy/darshan
that referenced
this issue
Sep 27, 2022
* add some `asv` peak memory benchmarks as regression guards against darshan-hpcgh-811; the only thing missing (IMO) here to completely close darshan-hpcgh-817 is a version of the log file with `HEATMAP` in it (see the diff comments for why I don't have this for `asv` suite at the moment) * although this won't automatically run in CI because it takes too long and too much memory, it still seems to do an excellent job of being sensitive to the patched leaks over the recent history of our project: - `asv continuous -e -b "peakmem.*" b6c8599 main` ``` before after ratio [b6c8599] [3fb2c36] <treddy_memleak_cffi_only_pt2~1> <main> - 239M 76M 0.32 cffi_backend.RetrieveLogData.peakmem_log_get_record('LUSTRE') - 241M 76M 0.32 cffi_backend.RetrieveLogData.peakmem_log_get_record('H5F') - 257M 75.8M 0.30 cffi_backend.RetrieveLogData.peakmem_log_get_record('STDIO') - 289M 76M 0.26 cffi_backend.RetrieveLogData.peakmem_log_get_record('MPI-IO') - 303M 76M 0.25 cffi_backend.RetrieveLogData.peakmem_log_get_record('POSIX') - 304M 75.4M 0.25 cffi_backend.RetrieveLogData.peakmem_log_get_record('DXT_POSIX') - 304M 75.3M 0.25 cffi_backend.RetrieveLogData.peakmem_log_get_record('DXT_MPIIO') - 324M 75.8M 0.23 cffi_backend.RetrieveLogData.peakmem_log_get_record('H5D') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('DXT_POSIX') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('H5F') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('H5D') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('STDIO') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('MPI-IO') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('DXT_MPIIO') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('LUSTRE') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('POSIX') SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY. PERFORMANCE INCREASED. ```
tylerjereddy
added a commit
to tylerjereddy/darshan
that referenced
this issue
Oct 19, 2022
* add some `asv` peak memory benchmarks as regression guards against darshan-hpcgh-811; the only thing missing (IMO) here to completely close darshan-hpcgh-817 is a version of the log file with `HEATMAP` in it (see the diff comments for why I don't have this for `asv` suite at the moment) * although this won't automatically run in CI because it takes too long and too much memory, it still seems to do an excellent job of being sensitive to the patched leaks over the recent history of our project: - `asv continuous -e -b "peakmem.*" b6c8599 main` ``` before after ratio [b6c8599] [3fb2c36] <treddy_memleak_cffi_only_pt2~1> <main> - 239M 76M 0.32 cffi_backend.RetrieveLogData.peakmem_log_get_record('LUSTRE') - 241M 76M 0.32 cffi_backend.RetrieveLogData.peakmem_log_get_record('H5F') - 257M 75.8M 0.30 cffi_backend.RetrieveLogData.peakmem_log_get_record('STDIO') - 289M 76M 0.26 cffi_backend.RetrieveLogData.peakmem_log_get_record('MPI-IO') - 303M 76M 0.25 cffi_backend.RetrieveLogData.peakmem_log_get_record('POSIX') - 304M 75.4M 0.25 cffi_backend.RetrieveLogData.peakmem_log_get_record('DXT_POSIX') - 304M 75.3M 0.25 cffi_backend.RetrieveLogData.peakmem_log_get_record('DXT_MPIIO') - 324M 75.8M 0.23 cffi_backend.RetrieveLogData.peakmem_log_get_record('H5D') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('DXT_POSIX') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('H5F') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('H5D') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('STDIO') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('MPI-IO') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('DXT_MPIIO') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('LUSTRE') - 23.7G 75.4M 0.00 cffi_backend.RetrieveLogData.peakmem_log_get_mounts('POSIX') SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY. PERFORMANCE INCREASED. ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is just a reminder to check if our current
asvsuite is sufficient to detect the memory footprint improvements related to fixing gh-811, and if not we should add a peak memory benchmark or two that are sensitive to that suite.That said, it is quite common not to have CI tests for memory leaks, even in quite large projects, so we can probably delay the actual mem profiling code for a little bit as long as we fix soon-ish.
The text was updated successfully, but these errors were encountered: