Skip to content

Commit

Permalink
add option to chunk netcdf history files to keep under 2GB
Browse files Browse the repository at this point in the history
  • Loading branch information
@hklion
hklion committed Jan 13, 2025
1 parent a7f610b commit 42ae26e
Show file tree
Hide file tree
Showing 6 changed files with 107 additions and 37 deletions.
80 changes: 46 additions & 34 deletions Docs/sphinx_doc/Plotfiles.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,49 +21,61 @@ The following options in the inputs file control the generation of plotfiles.
List of Parameters
------------------

+--------------------------------+-----------------------------------+-----------------------+------------+
| Parameter | Definition | Acceptable | Default |
| | | Values | |
+================================+===================================+=======================+============+
| **remora.plotfile_type** | AMReX or NETCDF plotfiles | "amrex" or | "amrex" |
| | | "netcdf / "NetCDF" | |
+--------------------------------+-----------------------------------+-----------------------+------------+
| **remora.file_min_digits** | Minimum number of digits | Integer >= 0 | 5 |
| | in iteration number appended to | | |
| | plotfile and checkpoint files | | |
+--------------------------------+-----------------------------------+-----------------------+------------+
| **remora.write_history_file** | do we write | false or true | true |
| | netcdf files at | | |
| | each timestep | | |
| | or one file for | | |
| | all timesteps? | | |
+--------------------------------+-----------------------------------+-----------------------+------------+
| **remora.plot_file** | prefix for | String | “plt” |
| | plotfiles | | |
+--------------------------------+-----------------------------------+-----------------------+------------+
| **remora.plot_int** | how often (by | Integer | -1 |
| | level-0 time | :math:`> 0` | |
| | steps) to write | | |
| | plot files | | |
+--------------------------------+-----------------------------------+-----------------------+------------+
| **remora.plot_vars** | name of | list of names | None |
| | variables to | (see table below) | |
| | include in | | |
| | plotfiles. Not | | |
| | used for netCDF | | |
+--------------------------------+-----------------------------------+-----------------------+------------+
+------------------------------------+-----------------------------------+-----------------------+------------+
| Parameter | Definition | Acceptable | Default |
| | | Values | |
+====================================+===================================+=======================+============+
| **remora.plotfile_type** | AMReX or NETCDF plotfiles | "amrex" or | "amrex" |
| | | "netcdf / "NetCDF" | |
+------------------------------------+-----------------------------------+-----------------------+------------+
| **remora.file_min_digits** | Minimum number of digits | Integer >= 0 | 5 |
| | in iteration number appended to | | |
| | plotfile, checkpoint, or chunked | | |
| | history files | | |
+------------------------------------+-----------------------------------+-----------------------+------------+
| **remora.write_history_file** | do we write | false or true | true |
| | netcdf files at | | |
| | each timestep | | |
| | or one file for | | |
| | all timesteps? | | |
+------------------------------------+-----------------------------------+-----------------------+------------+
| **remora.chunk_history_file** | do we divide netcdf history | false or true | false |
| | files so that each file contains | | |
| | only a certain number of time | | |
| | steps? | | |
+------------------------------------+-----------------------------------+-----------------------+------------+
| **remora.steps_per_history_file** | Maximum number of steps per | integer | -1 |
| | netcdf history file. If <=0, | | |
| | calculate automatically such that | | |
| | each file is less than 2GB | | |
+------------------------------------+-----------------------------------+-----------------------+------------+
| **remora.plot_file** | prefix for | String | “plt” |
| | plotfiles | | |
+------------------------------------+-----------------------------------+-----------------------+------------+
| **remora.plot_int** | how often (by | Integer | -1 |
| | level-0 time | :math:`> 0` | |
| | steps) to write | | |
| | plot files | | |
+------------------------------------+-----------------------------------+-----------------------+------------+
| **remora.plot_vars** | name of | list of names | None |
| | variables to | (see table below) | |
| | include in | | |
| | plotfiles. Not | | |
| | used for netCDF | | |
+------------------------------------+-----------------------------------+-----------------------+------------+

.. _notes-5:

Notes
-----

- The NeTCDF option is only available if REMORA has been built with USE_PNETCDF enabled.
- The NetCDF option is only available if REMORA has been built with USE_PNETCDF enabled.

- The write_history_file option is only available if **plotfile_type = netcdf**

- If **plotfile_type = netcdf** and **write_history_file = false**, the frequency
will be determined by **plot_int**
- Depending on your PnetCDF build, the code may be unable to write files larger than 2 GB. If the code
crashes when writing a NetCDF history file (or a single time step, if you have a particularly large grid),
consider building with MPICH v4.2.2 or instead outputting a native AMReX plotfile instead.

- Velocity components are defined on faces within the REMORA code, but are averaged onto
cell centers when written in amrex/native plotfiles. They are not averaged when writing
Expand Down
5 changes: 5 additions & 0 deletions Docs/sphinx_doc/building.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ At run-time, you may need to add PnetCDF to the link path, e.g.:
See sections below for compiler-specific instructions for how to enable NetCDF support.

.. note::
**When built with some MPI implementations, PnetCDF will crash when writing files larger than 2GB. Building
with MPICH v4.2.2 will likely solve this issue.**


GNU Make
~~~~~~~~

Expand Down
21 changes: 19 additions & 2 deletions Source/IO/REMORA_NCPlotFile.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ void REMORA::WriteNCPlotFile(int which_step) {
// For right now we assume single level -- we will generalize this later to multilevel
int lev = 0;
int which_subdomain = 0;
int which_step_in_chunk = -1;

// Create filename
std::string plt_string;
Expand All @@ -32,6 +33,12 @@ void REMORA::WriteNCPlotFile(int which_step) {
} else {
plotfilename = Concatenate(plot_file_name, which_step, file_min_digits);
}
// If chunking, concatenate with which file we're in
if (REMORA::write_history_file and REMORA::chunk_history_file) {
int which_chunk = history_count / REMORA::steps_per_history_file;
plotfilename = Concatenate(plotfilename, which_chunk, file_min_digits);
which_step_in_chunk = history_count - which_chunk * REMORA::steps_per_history_file;
}

// Set the full IO path for NetCDF output
std::string FullPath = plotfilename;
Expand All @@ -48,7 +55,7 @@ void REMORA::WriteNCPlotFile(int which_step) {
// have the IOProcessor move the existing
// file/directory to filename.old
//
if ((!REMORA::write_history_file) || (which_step == 0)) {
if ((!REMORA::write_history_file) || (which_step == 0) || (which_step_in_chunk == 0)) {
if (amrex::ParallelDescriptor::IOProcessor()) {
if (amrex::FileExists(FullPath)) {
std::string newoldname(FullPath + ".old." + amrex::UniqueString());
Expand Down Expand Up @@ -111,6 +118,15 @@ void REMORA::WriteNCPlotFile_which(int lev, int which_subdomain, bool write_head
if (is_history && max_step < 0)
amrex::Abort("Need to know max_step if writing history file");
long long int nt = is_history ? static_cast<long long int>(max_step / std::min(plot_int, max_step)) + 1 : 1;
if (chunk_history_file) {
// First index of the last history file
int last_file_index = REMORA::steps_per_history_file * int(nt / REMORA::steps_per_history_file);
if (history_count >= last_file_index) {
nt = nt - last_file_index;
} else {
nt = REMORA::steps_per_history_file;
}
}

n_cells.push_back(nx);
n_cells.push_back(ny);
Expand Down Expand Up @@ -471,7 +487,8 @@ void REMORA::WriteNCPlotFile_which(int lev, int which_subdomain, bool write_head
//
// We compute the offsets based on location of the box within the domain
//
long long local_start_nt = (is_history ? static_cast<long long>(history_count) : static_cast<long long>(0));
long long adjusted_history_count = chunk_history_file ? history_count % steps_per_history_file : history_count;
long long local_start_nt = (is_history ? static_cast<long long>(adjusted_history_count) : static_cast<long long>(0));
long long local_nt = 1; // We write data for only one time

{
Expand Down
10 changes: 9 additions & 1 deletion Source/REMORA.H
View file
Original file line number Diff line number Diff line change
Expand Up @@ -978,6 +978,14 @@ private:
std::string check_file {"chk"};
int check_int = -1;

// Whether to chunk netcdf history file
bool chunk_history_file = false;

// Number of time steps per netcdf history file.
// -1 is the default and means code will automatically compute
// number of steps per file based on grid size to keep files < 2 GB
int steps_per_history_file = -1;

amrex::Vector<std::string> plot_var_names;
const amrex::Vector<std::string> cons_names {"temp", "salt", "scalar"};

Expand Down Expand Up @@ -1017,7 +1025,7 @@ private:
static int sum_interval;
static amrex::Real sum_per;

// Minimum number of digits in plotfile name
// Minimum number of digits in plotfile name or chunked history file
static int file_min_digits;

// Native or NetCDF
Expand Down
22 changes: 22 additions & 0 deletions Source/REMORA.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -794,6 +794,28 @@ REMORA::ReadParameters ()
plotfile_type = PlotfileType::netcdf;
#ifdef REMORA_USE_NETCDF
pp.query("write_history_file",write_history_file);
pp.query("chunk_history_file",chunk_history_file);
pp.query("steps_per_history_file",steps_per_history_file);
// Estimate size of domain for one timestep of netcdf
auto dom = geom[0].Domain();
int nx = dom.length(0) + 2;
int ny = dom.length(1) + 2;
int nz = dom.length(2);
if (write_history_file and chunk_history_file and (steps_per_history_file <= 0)) {
// Estimate number of steps that will fit into a 2GB file.
steps_per_history_file = int((1.6e10 - NCH2D * nx * ny * 64.0_rt)
/ (nx * ny * 64.0_rt * (NC3D*nz + NC2D)));
// If we calculate that a single step will exceed 2GB and the user has
// requested automatic history file sizing, warn about a possible impending
// error, and set steps_per_history_file = 1 to attempt output anyway.
if (steps_per_history_file == 0) {
amrex::Warning("NetCDF output for a single timestep appears to exceed 2GB. NetCDF output may not work. See Documentation for information about tested MPICH versions.");
steps_per_history_file = 1;
}
}
if (write_history_file and chunk_history_file) {
Print() << "NetCDF history files will have " << steps_per_history_file << " steps per file." << std::endl;
}
#endif
} else {
amrex::Print() << "User selected plotfile_type = " << plotfile_type_str << std::endl;
Expand Down
6 changes: 6 additions & 0 deletions Source/REMORA_IndexDefines.H
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@

#define NGROW 2

// Number of 3D/2D variables output in netcdf files
#define NC3D 5
#define NC2D 5
// Number of 2D variables in a netcdf file's header
#define NCH2D 12

namespace BCVars {
enum {
cons_bc = 0,
Expand Down

0 comments on commit 42ae26e

Please sign in to comment.