Skip to content

Commit

Permalink
Fix use on SLURM by replacing parallel::detectCores()
Browse files Browse the repository at this point in the history
Replaced parallel::detectCores() with length(parallelly::availableWorkers()). Since the former does not respect the allocations of queueing systems such as SLURM, it was not ready to be used on systems such as UBELIX that use e.g. SLURM (see https://stackoverflow.com/a/75923399)
  • Loading branch information
fabern committed Oct 1, 2024
1 parent 7af1031 commit 96e3c2d
Show file tree
Hide file tree
Showing 3 changed files with 4 additions and 4 deletions.
4 changes: 2 additions & 2 deletions R/map2tidy.R
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
#' @param fileprefix A character string specifying the file name prefix.
#' @param ncores Number of cores for parallel execution (distributing
#' extraction of longitude slices). When set to \code{"all"}, the number of
#' cores for parallelisation is determined by \code{parallel::detectCores()}.
#' cores for parallelisation is determined by \code{length(parallelly::availableWorkers())}.
#' Defaults to \code{1} (no parallelisation).
#' @param fgetdate A function to derive the date used for the time dimension
#' based on the file name.
Expand Down Expand Up @@ -114,7 +114,7 @@ map2tidy <- function(


if (ncores=="all"){
ncores <- parallel::detectCores() - 1
ncores <- length(parallelly::availableWorkers()) - 1
}
ncores <- min(ncores, nrow(ilon_arg)) # No need to have more cores if we only produce 1 file

Expand Down
2 changes: 1 addition & 1 deletion man/map2tidy.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion vignettes/parallel_computation.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ print("getting data for longitude indices:")
list_of_LON_str <- gsub(".*(LON_[0-9.+-]*).*.rds","\\1", res_tidy$data) # extract LONGITUDES from res_tidy
# get all available cores
ncores <- 3 # parallel::detectCores()
ncores <- 3 # length(parallelly::availableWorkers())
# set up the cluster, sending required objects to each core
cl <- multidplyr::new_cluster(ncores) %>%
Expand Down

0 comments on commit 96e3c2d

Please sign in to comment.