Merge branch 'main' into psidm-MoreHybridTests

hsinhaoHHuang · Jan 27, 2025 · faa2626 · faa2626
2 parents c536d4c + b9d3fd3
commit faa2626
Show file tree

Hide file tree

Showing 228 changed files with 2,110 additions and 606 deletions.
diff --git a/.gitignore b/.gitignore
@@ -14,3 +14,4 @@ GAMER_ExtractProfile
 .vscode
 src/Makefile
 src/Makefile.log
+src/.local_settings
diff --git a/configs/mac_clang.config b/configs/mac_clang.config
@@ -0,0 +1,41 @@
+# A "typical" macOS installation without GPU
+FFTW2_PATH      /usr/local/fftw-2.1.5
+MPI_PATH        /usr/local/mpich-3.2
+GSL_PATH        /usr/local/gsl-1.16
+HDF5_PATH       ${HOME}/mambaforge/envs/py312
+
+# compilers
+CXX     clang++
+CXX_MPI mpicxx
+
+# clang flags
+CXXFLAG -g
+CXXFLAG -O3
+CXXFLAG -std=c++11
+#CXXFLAG -Ofast
+CXXFLAG -Wall
+CXXFLAG -Wextra
+CXXFLAG -Wno-unused-variable
+CXXFLAG -Wno-unused-parameter
+CXXFLAG -Wno-uninitialized
+CXXFLAG -Wno-unused-but-set-variable
+CXXFLAG -Wno-unused-function
+CXXFLAG -Wno-unused-result
+CXXFLAG -Wno-implicit-fallthrough
+CXXFLAG -Wno-parentheses
+CXXFLAG -Wno-unknown-pragmas
+CXXFLAG -Wno-mismatched-tags
+CXXFLAG -Wno-deprecated-declarations
+CXXFLAG -Wno-missing-braces
+
+OPENMPFLAG -Xclang -fopenmp
+
+LIBFLAG -lomp
+
+NVCCFLAG_COM -O3
+#NVCCFLAG_COM -use_fast_math
+NVCCFLAG_FLU -Xptxas -dlcm=ca -prec-div=false -ftz=true
+NVCCFLAG_POT -Xptxas -dlcm=ca
+
+#gpu
+GPU_COMPUTE_CAPABILITY -1
diff --git a/configs/mac.config → configs/mac_gnu.config b/configs/mac.config → configs/mac_gnu.config
@@ -1,17 +1,17 @@
 # A "typical" macOS installation without GPU
 FFTW2_PATH      /usr/local/fftw-2.1.5
 MPI_PATH        /usr/local/mpich-3.2
-HDF5_PATH       ${HOME}/miniconda3
 GSL_PATH        /usr/local/gsl-1.16
+HDF5_PATH       ${HOME}/mambaforge/envs/py312
 
 # compilers
-CXX     g++
+CXX     g++-12
 CXX_MPI mpicxx
 
 # gnu flags
 CXXFLAG -g
 CXXFLAG -O3
-#CXXFLAG -std=c++11
+CXXFLAG -std=c++11
 #CXXFLAG -Ofast
 CXXFLAG -Wall
 CXXFLAG -Wextra

diff --git a/doc/wiki/Installation-related/Installation:-Machine-Configuration-File.md b/doc/wiki/Installation-related/Installation:-Machine-Configuration-File.md
@@ -1,7 +1,18 @@
+The machine configuration file is located under `configs`. It specifies the library paths, compiler types, compilation flags, and GPU compute capability.
+
 > [!TIP]
-> To set up your machine configuration file, make a copy of `template.config` and modify it.
+> Check the `configs` directory to see if a configuration file is already available for your machine.
+
+## Set Up the Machine Configuration File
+
+To set up your machine configuration file, go to `configs` and make a copy of `template.config` to modify:
+
+```bash
+cd configs
+cp template.config your_machine.config
+```
 
-The machine configuration file is under `configs`. The configuration file contains the library paths, the compiler types, the compilation flags, and the GPU compute capability.
+Please refer to the following sections to set up your machine configuration file.
 
 ### 0. Rules of the configuration file
 * Comments must start with `#`.

diff --git a/doc/wiki/Installation-related/Installation:-Option-List.md b/doc/wiki/Installation-related/Installation:-Option-List.md
@@ -22,9 +22,9 @@ disabled). See the "Restriction" of each option carefully.
 
 | Option | Value | Description |
 |:---:|:---:|---|
-| `-h`        | -               | Show a short help message. |
-| `-lh`       | -               | Show a detailed help message. |
-| `--machine` | Filename string | Select the `*.config` file from the `configs` directory. |
+| `-h`                                 | -               | Show a short help message. |
+| `-lh`                                | -               | Show a detailed help message. |
+| `--machine` <a name="--machine"></a> | Filename string | Select the `*.config` file from the `configs` directory. It will overwrite the default machine set in the [[default setting file \| Installation#default_setting]]. |
 
 &#8192;&#8192;&#8192;&#8192;&#8192;
 &#8192;&#8192;&#8192;&#8192;&#8192;&#8192;&#8192;&#8192;&#8192;&#8192;

diff --git a/doc/wiki/Installation.md b/doc/wiki/Installation.md
@@ -2,33 +2,46 @@
 
    Please see [[Machine Configuration File | Installation:-Machine-Configuration-File]].
 
-2. Go to the source directory
+<a name="default_setting"></a>
+
+2. Set your machine configuration file as default
+
+   ```bash
+   sh tool/config/set_settings.sh --local --machine=your_machine
+   ```
+
+> [!NOTE]
+> If you want to set the default machine configuration file for all GAMER copies under your user account, use the `--global` option instead of `--local`.
+You can still override the global setting for individual GAMER copies using the `--local` option.
+Furthermore, you can override the default setting by passing the [[--machine | Installation:-Option-List#--machine]]=`your_machine` when executing `configure.py`. 
+
+3. Go to the source directory
 
    ```bash
    cd src
    ```
 
-3. Generate `Makefile` using the Python script `configure.py`
+4. Generate `Makefile` using the Python script `configure.py`
 
    To get the `Makefile`, please execute the following command:
 
    ```bash
-   python configure.py --machine=your_configuration_file [--your_arguments]
+   python configure.py [--your_arguments]
    ```
 
-   `your_configuration_file` is the configuration filename you got from step 1, and `[--your_arguments]` should match your simulation requirements. Please check out [[Option List | Installation:-Option-List]] for all the available options.
+   `[--your_arguments]` represent the options that should align with your simulation requirements. Refer to [[Option List | Installation:-Option-List]] for a complete list of available options.
 
-   For example, the following command uses the `configs/pleiades.config` machine configuration, sets the FFTW method to `FFTW2`, and enables gravity and GPU.
+   For example, the following command sets the FFTW method to `FFTW2` and enables gravity and GPU.
 
    ``` bash
-   python configure.py --machine=pleiades --fftw=FFTW2 --gravity=true --gpu=true
+   python configure.py --fftw=FFTW2 --gravity=true --gpu=true
    ```
 
 > [!TIP]
 > An example script `generate_make.sh` to generate Makefile can be found in each test problem folder,
 e.g., `example/test_problem/Hydro/AcousticWave/generate_make.sh`.
 
-4. Compile the code
+5. Compile the code
 
    ```bash
    make clean

diff --git a/doc/wiki/Quick-Start-related/Quick-Start:-1D-Shock-Tube.md b/doc/wiki/Quick-Start-related/Quick-Start:-1D-Shock-Tube.md
@@ -236,7 +236,7 @@ following modifications.
 
 1\. Re-generate `Makefile` by [[configure.py | Installation]] and recompile `gamer`.
 ``` bash
-sh generate_make.sh --machine=YOUR_MACHINE --openmp=true --gpu=true
+sh generate_make.sh --openmp=true --gpu=true
 make clean
 make -j4
 ```

diff --git a/doc/wiki/Quick-Start.md b/doc/wiki/Quick-Start.md
@@ -4,9 +4,21 @@ Follow the instructions in [[Download]] to get the latest version of GAMER.
 
 ### Setup Your Machine Configuration File
 
-Follow the instructions in [[Machine Configuration File | Installation:-Machine-Configuration-File]] to set the configuration file of your machine.
+The machine configuration file is located under `configs`. This file contains the library paths, compiler types, compilation flags, and GPU compute capability. Check the existing configuration files to see if one matches your machine.
+
+If no suitable configuration file is available, you will need to create a new one. Follow the instructions in [[Machine Configuration File | Installation:-Machine-Configuration-File]] to set up a configuration file for your machine.
+
+If a configuration file matches your machine, you can set it as the default by running
+
+```bash
+sh tool/config/set_settings.sh --local --machine=your_machine
+```
+
+For example, setting `--machine=pleiades` with the above command will use the `configs/pleiades.config` machine configuration when compiling the code.
 
 ### Quick Demos
 
+Read the following guides to learn how to configure, compile, and run GAMER.
+
 1. [[1D Shock Tube: OpenMP with/without GPU acceleration | Quick-Start:-1D-Shock-Tube]]
 2. [[3D Blast Wave: hybrid MPI/OpenMP/GPU + yt analysis | Quick-Start:-3D-Blast-Wave]]
diff --git a/doc/wiki/Runtime-Parameters-related/Runtime-Parameters:-All.md b/doc/wiki/Runtime-Parameters-related/Runtime-Parameters:-All.md
@@ -292,15 +292,16 @@ For variables with `Default/Min/Max` labeled as `Depend`, click the parameter na
 | [[ OPT__OUTPUT_LORENTZ \| Runtime-Parameters:-Outputs#OPT__OUTPUT_LORENTZ ]]                         |               0 |            None |            None | output Lorentz factor [0] ##SRHD ONLY## |
 | [[ OPT__OUTPUT_MACH \| Runtime-Parameters:-Outputs#OPT__OUTPUT_MACH ]]                               |               0 |            None |            None | output mach number [0] ##HYDRO ONLY## |
 | [[ OPT__OUTPUT_MODE \| Runtime-Parameters:-Outputs#OPT__OUTPUT_MODE ]]                               |              -1 |               1 |               3 | (1=const step, 2=const dt, 3=dump table) -> edit "Input__DumpTable" for 3 |
-| [[ OPT__OUTPUT_PART \| Runtime-Parameters:-Outputs#OPT__OUTPUT_PART ]]                               |               0 |               0 |               7 | output a single line or slice: (0=off, 1=xy, 2=yz, 3=xz, 4=x, 5=y, 6=z, 7=diag) [0] |
+| [[ OPT__OUTPUT_PART \| Runtime-Parameters:-Outputs#OPT__OUTPUT_PART ]]                               |               0 |               0 |               8 | output a single line or slice: (0=off, 1=xy, 2=yz, 3=xz, 4=x, 5=y, 6=z, 7=diag, 8=entire box) [0] |
 | [[ OPT__OUTPUT_PAR_DENS \| Runtime-Parameters:-Outputs#OPT__OUTPUT_PAR_DENS ]]                       | PAR_OUTPUT_DENS_PAR_ONLY |               0 |               2 | output the particle or total mass density on grids: (0=off, 1=particle mass density, 2=total mass density) [1] ##OPT__OUTPUT_TOTAL ONLY## |
 | [[ OPT__OUTPUT_PAR_MESH \| Runtime-Parameters:-Outputs#OPT__OUTPUT_PAR_MESH ]]                       |          Depend |          Depend |          Depend | output the attributes of tracer particles mapped from mesh quantities -> edit "Input__Par_Mesh" [1] ##PARTICLE ONLY## |
 | [[ OPT__OUTPUT_PAR_MODE \| Runtime-Parameters:-Outputs#OPT__OUTPUT_PAR_MODE ]]                       |               0 |               0 |               2 | output the particle data: (0=off, 1=text-file, 2=C-binary) [0] ##PARTICLE ONLY## |
 | [[ OPT__OUTPUT_POT \| Runtime-Parameters:-Outputs#OPT__OUTPUT_POT ]]                                 |               1 |            None |            None | output gravitational potential [1] ##OPT__OUTPUT_TOTAL ONLY## |
 | [[ OPT__OUTPUT_PRES \| Runtime-Parameters:-Outputs#OPT__OUTPUT_PRES ]]                               |               0 |            None |            None | output gas pressure [0] ##HYDRO ONLY## |
 | [[ OPT__OUTPUT_RESTART \| Runtime-Parameters:-Outputs#OPT__OUTPUT_RESTART ]]                         |               0 |            None |            None | output data immediately after restart [0] |
 | [[ OPT__OUTPUT_TEMP \| Runtime-Parameters:-Outputs#OPT__OUTPUT_TEMP ]]                               |  OutTempDefault |            None |            None | output gas temperature [0 (HD) or 1 (SRHD)] ##HYDRO ONLY## |
-| [[ OPT__OUTPUT_TEXT_FORMAT_FLT \| Runtime-Parameters:-Outputs#OPT__OUTPUT_TEXT_FORMAT_FLT ]]         |       "%24.16e" |            None |            None | string format of output text files [%24.16e] |
+| [[ OPT__OUTPUT_TEXT_FORMAT_FLT \| Runtime-Parameters:-Outputs#OPT__OUTPUT_TEXT_FORMAT_FLT ]]         |       "%24.16e" |            None |            None | string format of floating-point variables in output text files [%24.16e] |
+| [[ OPT__OUTPUT_TEXT_LENGTH_INT \| Runtime-Parameters:-Outputs#OPT__OUTPUT_TEXT_LENGTH_INT ]]         |              12 |               0 |            None | string length of integer variables in output text files [12] |
 | [[ OPT__OUTPUT_TOTAL \| Runtime-Parameters:-Outputs#OPT__OUTPUT_TOTAL ]]                             |               1 |               0 |               2 | output the simulation snapshot: (0=off, 1=HDF5, 2=C-binary) [1] |
 | [[ OPT__OUTPUT_USER \| Runtime-Parameters:-Outputs#OPT__OUTPUT_USER ]]                               |               0 |            None |            None | output the user-specified data -> edit "Output_User.cpp" [0] |
 | [[ OPT__OUTPUT_USER_FIELD \| Runtime-Parameters:-Outputs#OPT__OUTPUT_USER_FIELD ]]                   |               0 |            None |            None | output user-defined derived fields [0] -> edit "Flu_DerivedField_User.cpp" |
@@ -341,6 +342,7 @@ For variables with `Default/Min/Max` labeled as `Depend`, click the parameter na
 | [[ OPT__UM_IC_REFINE \| Runtime-Parameters:-Initial-Conditions#OPT__UM_IC_REFINE ]]                  |               1 |            None |            None | refine UM_IC from level OPT__UM_IC_LEVEL to MAX_LEVEL [1] |
 | [[ OPT__UNIT \| Runtime-Parameters:-Units#OPT__UNIT ]]                                               |               0 |            None |            None | specify code units -> must set exactly 3 basic units below [0] ##USELESS FOR COMOVING## |
 | [[ OPT__VERBOSE \| Runtime-Parameters:-Miscellaneous#OPT__VERBOSE ]]                                 |               0 |            None |            None | output the simulation progress in detail [0] |
+| [[ OUTPUT_DIR \| Runtime-Parameters:-Outputs#OUTPUT_DIR ]]                                           |             "." |            None |            None | set the output directory [.] |
 | [[ OUTPUT_DT \| Runtime-Parameters:-Outputs#OUTPUT_DT ]]                                             |            -1.0 |            None |            None | output data every OUTPUT_DT time interval ##OPT__OUTPUT_MODE==2 ONLY## |
 | [[ OUTPUT_PART_X \| Runtime-Parameters:-Outputs#OUTPUT_PART_X ]]                                     |            -1.0 |            None |            None | x coordinate for OPT__OUTPUT_PART [-1.0] |
 | [[ OUTPUT_PART_Y \| Runtime-Parameters:-Outputs#OUTPUT_PART_Y ]]                                     |            -1.0 |            None |            None | y coordinate for OPT__OUTPUT_PART [-1.0] |

diff --git a/doc/wiki/Runtime-Parameters-related/Runtime-Parameters:-Outputs.md b/doc/wiki/Runtime-Parameters-related/Runtime-Parameters:-Outputs.md
@@ -2,6 +2,7 @@ Parameters described on this page:
 [OPT__OUTPUT_TOTAL](#OPT__OUTPUT_TOTAL), &nbsp;
 [OPT__OUTPUT_PART](#OPT__OUTPUT_PART), &nbsp;
 [OPT__OUTPUT_TEXT_FORMAT_FLT](#OPT__OUTPUT_TEXT_FORMAT_FLT), &nbsp;
+[OPT__OUTPUT_TEXT_LENGTH_INT](#OPT__OUTPUT_TEXT_LENGTH_INT), &nbsp;
 [OPT__OUTPUT_USER](#OPT__OUTPUT_USER), &nbsp;
 [OPT__OUTPUT_PAR_MODE](#OPT__OUTPUT_PAR_MODE), &nbsp;
 [OPT__OUTPUT_PAR_MESH](#OPT__OUTPUT_PAR_MESH), &nbsp;
@@ -31,6 +32,7 @@ Parameters described on this page:
 [OUTPUT_PART_Y](#OUTPUT_PART_Y), &nbsp;
 [OUTPUT_PART_Z](#OUTPUT_PART_Z), &nbsp;
 [INIT_DUMPID](#INIT_DUMPID), &nbsp;
+[OUTPUT_DIR](#OUTPUT_DIR) &nbsp;
 
 
 Parameters below are shown in the format: &ensp; **`Name` &ensp; (Valid Values) &ensp; [Default Value]**
@@ -52,9 +54,9 @@ data dump frequency.
 the HDF5 snapshots of GAMER.
 
 <a name="OPT__OUTPUT_PART"></a>
-* #### `OPT__OUTPUT_PART` &ensp; (0=off, 1=xy, 2=yz, 3=xz, 4=x, 5=y, 6=z, 7=diagonal) &ensp; [0]
+* #### `OPT__OUTPUT_PART` &ensp; (0=off, 1=xy, 2=yz, 3=xz, 4=x, 5=y, 6=z, 7=diagonal, 8=entire box) &ensp; [0]
     * **Description:**
-Output the data of a single line or slice in the text format.
+Output the data of a single line, slice, or entire box in the text format.
 Use [OUTPUT_PART_X](#OUTPUT_PART_X), [OUTPUT_PART_Y](#OUTPUT_PART_Y), and
 [OUTPUT_PART_Z](#OUTPUT_PART_Z) to specify the target coordinates.
 This output option is mainly for quick inspection for simple test problems
@@ -75,6 +77,12 @@ This will affect files generated by, for example,
 [OPT__OUTPUT_BASEPS](#OPT__OUTPUT_BASEPS).
     * **Restriction:**
 
+<a name="OPT__OUTPUT_TEXT_LENGTH_INT"></a>
+* #### `OPT__OUTPUT_TEXT_LENGTH_INT` &ensp; (&#8805;0) &ensp; [12]
+    * **Description:**
+String length of integer data in output text files.
+    * **Restriction:**
+
 <a name="OPT__OUTPUT_USER"></a>
 * #### `OPT__OUTPUT_USER` &ensp; (0=off, 1=on) &ensp; [0]
     * **Description:**
@@ -283,6 +291,14 @@ in most cases, even when restarting simulations, unless you want to overwrite
 the default value.
     * **Restriction:**
 
+<a name="OUTPUT_DIR"></a>
+* #### `OUTPUT_DIR` &ensp; (string) &ensp; [.]
+    * **Description:**
+Specify the output data directory. In most cases, it is unnecessary to set this value
+unless you want to overwrite the default directory.
+    * **Restriction:**
+The string length should be shorter than `MAX_STRING-100`.
+
 
 ## Remarks
 

diff --git a/example/grackle/Make.mach.eureka-intel b/example/grackle/Make.mach.eureka-intel
@@ -0,0 +1,97 @@
+#=======================================================================
+#
+# FILE:        Make.mach.eureka-intel
+#
+# DESCRIPTION: Makefile for the calab-ntu Eureka GPU cluster with Intel
+#              compiler and OpenMP support
+#
+# AUTHOR:      Hsi-Yu Schive (modified by Guan-Ming Su, Barry T. Chiang,
+#              and Chun-Yen Chen)
+#
+# DATE:        April 1, 2016 (August 28, 2024)
+#=======================================================================
+
+MACH_TEXT  = Eureka Intel
+MACH_VALID = 0
+MACH_FILE  = Make.mach.eureka-intel
+
+#-----------------------------------------------------------------------
+# Install paths (local variables)
+#-----------------------------------------------------------------------
+
+LOCAL_HDF5_INSTALL = /software/hdf5/default
+LOCAL_FC_INSTALL   = /software/intel/oneapi/compiler/default/linux/compiler/lib/intel64
+
+#-----------------------------------------------------------------------
+# Compiler settings
+#-----------------------------------------------------------------------
+
+MACH_CPP       = icpc # C preprocessor command
+
+# Compilers
+MACH_CC_NOMPI  = icc # C compiler
+MACH_CXX_NOMPI = icpc # C++ compiler
+MACH_FC_NOMPI  = ifort # Fortran 77
+MACH_F90_NOMPI = ifort # Fortran 90
+MACH_LD_NOMPI  = icpc # Linker
+MACH_LIBTOOL   = libtool
+
+#-----------------------------------------------------------------------
+# Machine-dependent defines
+#-----------------------------------------------------------------------
+
+# Note: When compiling against HDF5 version 1.8 or greater, you need to
+# compile HDF5 with --with-default-api-version=v16, or specify with
+# -DH5_USE_16_API.
+
+MACH_DEFINES   = -DLINUX -DH5_USE_16_API
+
+#-----------------------------------------------------------------------
+# Compiler flag settings
+#-----------------------------------------------------------------------
+
+MACH_OMPFLAGS = -fopenmp # OpenMP flags
+MACH_CPPFLAGS = -P -traditional # C preprocessor flags
+MACH_CFLAGS   = -mp1 -prec_div -fp_port -align # C compiler flags
+MACH_CXXFLAGS = -mp1 -prec_div -fp_port -align # C++ compiler flags
+MACH_FFLAGS   = # Fortran 77 compiler flags
+MACH_F90FLAGS = -mp1 -prec_div -fp_port -align #-save -zero # Fortran 90 compiler flags
+MACH_LDFLAGS  = #-lifcore -lifport -lpthread -ldl # Linker flags
+
+#-----------------------------------------------------------------------
+# Optimization flags
+#-----------------------------------------------------------------------
+
+MACH_OPT_WARN        = # Flags for verbose compiler warnings
+MACH_OPT_DEBUG       = -g -O0 # Flags for debugging
+MACH_OPT_HIGH        = -O2 # Flags for high conservative optimization
+MACH_OPT_AGGRESSIVE  = -O3 #-Mfptrap -Mflushz -fastsse -Mdaz -Mnontemporal -Mnofprelaxed -Mvect=altcode,assoc,prefetch -Kieee # Flags for aggressive optimization
+
+#-----------------------------------------------------------------------
+# Includes
+#-----------------------------------------------------------------------
+
+LOCAL_INCLUDES_HDF5   = -I$(LOCAL_HDF5_INSTALL)/include # HDF5 includes
+MACH_INCLUDES         = $(LOCAL_INCLUDES_HDF5)
+
+#-----------------------------------------------------------------------
+# Libraries
+#-----------------------------------------------------------------------
+
+LOCAL_LIBS_HDF5   = -L$(LOCAL_HDF5_INSTALL)/lib -lhdf5 # HDF5 libraries
+LOCAL_LIBS_MACH   = -L$(LOCAL_FC_INSTALL) -lm -lifcore -lifport # Machine-dependent libraries
+
+MACH_LIBS         = $(LOCAL_LIBS_HDF5) $(LOCAL_LIBS_MACH)
+
+#-----------------------------------------------------------------------
+# Installation
+#-----------------------------------------------------------------------
+
+# if $(HOME)/local does not exist, mkdir before `make install`
+
+### Use this when make omp-on is set
+MACH_INSTALL_PREFIX = <local_grackle_directory>/grackle_eureka_double_omp
+### Use this when make omp-off is set
+#MACH_INSTALL_PREFIX = <local_grackle_directory>/grackle_eureka_double_no_omp
+MACH_INSTALL_LIB_DIR =
+MACH_INSTALL_INCLUDE_DIR =