updating benchmarks

[francispoulin]

I just tried to run benchmark_shallow_water_model.jl and saw that there was an error. Considering it hasn't been touched in 3 years, this is not suprising.

I want to see how things scale nowadays with shallow water and nonhydrostatic models. If anyone has any suggestions when I work on these, please let me know.

I am hoping we can get some updated plots.

[francispoulin]

I'm actually having a heck of a time running any of the benchmarks because of errors that seem to involve SuiteSparse_jll. Since the error suggests to contact them I did, and you can follow the thread here.

JuliaSparse/SparseArrays.jl#536

[francispoulin]

Okay, I downloaded a fresh version of main and followed hte instructions in the README to update the benchmark environment and have a bunch of errors.

Is this a problem for others or just me? I have tried it on a couple of machines and they are consistent.

$ julia -e 'using Pkg; Pkg.activate(pwd()); Pkg.instantiate(); Pkg.develop(PackageSpec(path=joinpath(@__DIR__, "..")))'
  Activating project at `~/Software/TMP2_main/Oceananigans.jl/benchmark`
┌ Warning: The active manifest file is an older format with no julia version entry. Dependencies may have been resolved with a different julia version.
└ @ ~/Software/TMP2_main/Oceananigans.jl/benchmark/Manifest.toml:0
Precompiling project...
        Info Given Benchmarks was explicitly requested, output will be shown live 
ERROR: LoadError: ArgumentError: Package SparseArrays does not have SuiteSparse_jll in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have SparseArrays checked out for development and have
  added SuiteSparse_jll as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with SparseArrays
Stacktrace:
  [1] macro expansion
    @ Base ./loading.jl:1770 [inlined]
  [2] macro expansion
    @ Base ./lock.jl:267 [inlined]
  [3] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1747
  [4] #invoke_in_world#3
...

When I asked SuiteeSparse_jll about this they said that I needed to use the same version of julia. I am using 1.10. Should I be using 1.9?

[glwagner]

Try deleting benchmark/Manifest.toml and starting again?

[francispoulin]

Thanks for the suggestion!

Unfortuantely, that still has errors.

$ julia -e 'using Pkg; Pkg.activate(pwd()); Pkg.instantiate(); Pkg.develop(PackageSpec(path=joinpath(@__DIR__, "..")))'
  Activating project at `~/Software/TMP2_main/Oceananigans.jl/benchmark`
  Installing known registries into `~/.julia`
    Updating registry at `~/.julia/registries/General.toml`
ERROR: Unsatisfiable requirements detected for package GPUCompiler [61eb1bfa]:
 GPUCompiler [61eb1bfa] log:
 ├─possible versions are: 0.1.0-0.26.4 or uninstalled
 ├─restricted by compatibility requirements with CUDA [052768ef] to versions: 0.13.3-0.17.3
 │ └─CUDA [052768ef] log:
 │   ├─possible versions are: 0.1.0-5.3.3 or uninstalled
 │   └─restricted to versions 3.8.0-3 by Benchmarks [ee90295e], leaving only versions: 3.8.0-3.13.1
 │     └─Benchmarks [ee90295e] log:
 │       ├─possible versions are: 0.1.0 or uninstalled
 │       └─Benchmarks [ee90295e] is fixed to version 0.1.0
 └─restricted by julia compatibility requirements to versions: 0.22.0-0.26.4 or uninstalled — no versions left
Stacktrace:
  [1] propagate_constraints!(graph::Pkg.Resolve.Graph, sources::Set{Int64}; log_events::Bool)
    @ Pkg.Resolve ~/Software/julia-1.10.0/share/julia/stdlib/v1.10/Pkg/src/Resolve/graphtype.jl:1072
  [2] propagate_constraints! (repeats 2 times)
    @ Pkg.Resolve ~/Software/julia-1.10.0/share/julia/stdlib/v1.10/Pkg/src/Resolve/graphtype.jl:1008 [inlined]
  [3] simplify_graph!(graph::Pkg.Resolve.Graph, sources::Set{Int64}; clean_graph::Bool)
    @ Pkg.Resolve ~/Software/julia-1.10.0/share/julia/stdlib/v1.10/Pkg/src/Resolve/graphtype.jl:1533
  [4] simplify_graph! (repeats 2 times)
    @ Pkg.Resolve ~/Software/julia-1.10.0/share/julia/stdlib/v1.10/Pkg/src/Resolve/graphtype.jl:1532 [inlined]
  [5] resolve_versions!(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber, installed_only::Bool)
    @ Pkg.Operations ~/Software/julia-1.10.0/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:407
  [6] up(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, level::Pkg.Types.UpgradeLevel; skip_writing_project::Bool, preserve::Nothing)
    @ Pkg.Operations ~/Software/julia-1.10.0/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1537
  [7] up
    @ Pkg.Operations ~/Software/julia-1.10.0/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1519 [inlined]
  [8] up(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, preserve::Nothing, update_registry::Bool, skip_writing_project::Bool, kwargs::@Kwargs{})
    @ Pkg.API ~/Software/julia-1.10.0/share/julia/stdlib/v1.10/Pkg/src/API.jl:351
  [9] up
    @ Pkg.API ~/Software/julia-1.10.0/share/julia/stdlib/v1.10/Pkg/src/API.jl:326 [inlined]
 [10] up
    @ Pkg.API ~/Software/julia-1.10.0/share/julia/stdlib/v1.10/Pkg/src/API.jl:164 [inlined]
 [11] instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Base.BinaryPlatforms.Platform, allow_build::Bool, allow_autoprecomp::Bool, kwargs::@Kwargs{})
    @ Pkg.API ~/Software/julia-1.10.0/share/julia/stdlib/v1.10/Pkg/src/API.jl:1790
 [12] instantiate
    @ ~/Software/julia-1.10.0/share/julia/stdlib/v1.10/Pkg/src/API.jl:1766 [inlined]
 [13] instantiate(; kwargs::@Kwargs{})
    @ Pkg.API ~/Software/julia-1.10.0/share/julia/stdlib/v1.10/Pkg/src/API.jl:1765
 [14] top-level scope
    @ none:1

[glwagner]

I'm sorry you're stuck in depedency hell... you have to keep trying different things to get out of it. I can't give you a magic bullet. One method is to remove packages from the project (eg Pkg.rm("CUDA"), or whatever), and then resolve dependencies, and then add the package back...

[francispoulin]

Thanks! I am now out of dependency hell! Great name by the way.

I removed CUDA, BSON, MPI, BenchmarkTools and then added them back in in the same order and now I'm happy to say the bennchmarks are running.

I hope this doesn't happen for others.

Now to do some benchmarking, at long ast.

[glwagner]

I hope this doesn't happen for others.

It's actually a common experience! Important to know how to get out of it.

[francispoulin]

The benchmark that we have for ShallowWaterModel from three years ago shows that there were speed up from CPU to GPU by a factor of 400.

When I ran it on my laptop, I had a speed p of no more than 17.5 because this is an NVIDA GeForce RTX, nothing fancy.

When I try it on a server with an NVIDA A100-SXM4-60GB we get much better results, over 300 but not quite 400 like we saw previously. I don't think this is a problem but if others do please let me know.

I'll now look at the NonhydrostaticModel.