Use ClusterManagers.ElasticManager instead of SlurmManager

[oschulz]

Now we can do this within an interactive Julia session (plain Julia session outside of an salloc):

using LegendDataManagement, Distributed

@always_everywhere begin
    using Distributions
end

mode = LegendDataManagement.SlurmRun(slurm_flags = `--ntasks=4 --cpus-per-task=8 --mem-per-cpu=8G`)
#LegendDataManagement.worker_start_command(mode)

# Add some workers:
legend_addprocs(mode)

# List resources:
using TypedTables
Table(LegendDataManagement.worker_resources())

# Confirm that Distributions is loaded on workers:
worker = last(workers())
@fetchfrom worker Normal()

# Add some more workers:
legend_addprocs(mode)
Table(LegendDataManagement.worker_resources())

# Add even more workers:
legend_addprocs(mode)
Table(LegendDataManagement.worker_resources())

And we can do SLURM batch scripts like this (e.g. "batchtest.jl"):

#!/usr/bin/env -S julia --project=@SOME_JULIA_ENVIRONMENT --threads=8
#SBATCH --ntasks=4 --cpus-per-task=8 --mem-per-cpu=8G

using LegendDataManagement, Distributed

@always_everywhere begin
    using LegendDataManagement
end

legend_addprocs()

report = lreport(
    "# Batch job report",
    "## Worker resources",
    LegendDataManagement.worker_resources()
)
# show(stdout, MIME"text/markdown"(), report)
show(stdout, MIME"text/plain"(), report)

Should run with a simple

sbatch -o out.txt batchtest.jl

and "out.txt" should then contain a report with the worker resources.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 229 lines in your changes are missing coverage. Please review.

Project coverage is 32.93%. Comparing base (35d10ca) to head (ab130eb).

Files	Patch %	Lines
src/slurm.jl	0.00%	130 Missing ⚠️
src/workers.jl	0.00%	99 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #47      +/-   ##
==========================================
- Coverage   37.33%   32.93%   -4.41%     
==========================================
  Files          24       25       +1     
  Lines        1398     1585     +187     
==========================================
  Hits          522      522              
- Misses        876     1063     +187     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[oschulz]

CC @theHenks , @LisaSchlueter , @JBlaschke

[oschulz]

Closing in favor of oschulz/ParallelProcessingTools.jl#11