file_type ="NETCDF_NCCF" for gen_ens_prod?

[benjamin-cash]

Hello,

I am trying to use gen_ens_prod to analyze a set of ensemble netcdf files. I know from plot_data_plane that met does not recognize these files unless I add file_type=NETCDF_NCCF to the field string. However, there isn't an option to pass a field string to gen_ens_prod, and adding file_type to the field entry in my config file leaves me with the same issue of Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_None".

How do I tell gen_ens_prod that my file is netcdf?

My configuration file is:
ens = {
ens_thresh = 1.0;
vld_thresh = 1.0;
field = [{
name = "tref";
level = "(,,*)";
file_type = NETCDF_NCCF;}];
}

and my command is:
gen_ens_prod -ens file_list -out test.nc -config my_config

[georgemccabe]

Hi @benjamin-cash, you cannot specify the file type per field, but you can specify it for all ensemble input.
Moving file_type = NETCDF_NCCF; out of the field array so that it is on the same level as ens_thresh and vld_thresh should give you the results you are expecting.

ens = {
  ens_thresh = 1.0;
  vld_thresh = 1.0;
  file_type = NETCDF_NCCF;
  field = [{
    name = "tref";
    level = "(*,*,*)";
  }];
}

[benjamin-cash]

@georgemccabe - Thanks, that got it to read in the files. :) Is there a way to tell it in the configuration that I want it to do the calculation based on the average over a particular time slice in the file? Right now it looks like it is grabbing the first entry in each file which are identical in some fields.

[JohnHalleyGotway]

@benjamin-cash, @georgemccabe asked me to take a look at your followup question.

I see from your original post that you're using:

  field = [{ name = "tref"; level = "(*,*,*)"; }];

Frankly I'm surprised that configuration didn't error out in Gen-Ens-Prod! When processing gridded NetCDF files, the level string should contain exactly 2 stars to indicate the gridded dimension (i.e. lat/lon). I'll assume that "tref" is a NetCDF variable that looks like this tref(time,lat,lon). You should configure the level string like this:

level = "(5,*,*)"; // to use the 6-th (0-based) timestep from that 3-dimensional variable

Or likely more convenient:

level = "(20210807_12,*,*)"; // to have MET find and use the time index that matches the timestring indicated

To answer your question though, MET does NOT have to ability to compute time-summaries (like mean) of gridded data on-the-fly. I can see how we could potentially enhance the NetCDF library code to so, but it definitely does not do that now.

You could run a PCP-Combine command with the "-derive" option first to compute the mean across multiple time steps before passing the output to Gen-Ens-Prod.

[benjamin-cash]

Thanks. I was able to get it to work for a single 6h period by specifying the timestep. I'm struggling with getting pcp_combine to work for this problem though. Could you give me an example command for calculating, say, the week 3-4 for a given variable that is stored in a single netcdf file? In my case the forecast is initialized 2011111500 and the data is 6 hourly.

[JohnHalleyGotway]

@benjamin-cash can you point me to a sample NetCDF file? One that's either publicly available or you could post it to our anonymous FTP Site following these instructions.

Might as well work with a real example.

[benjamin-cash]

Ok, I'm uploading 2012011500.mem001.subset.sfcf.all.nc following those directions now.

[JohnHalleyGotway]

@benjamin-cash, thanks for posting that sample file. As I understand it, you'd like to compute the average value for all time step between weeks 3 and 4. I pulled down the sample file and ran the following command.

I see that this file contains a whopping 141 hourly timesteps with a reference time of "2012-01-15 00:00:00" and 6-hourly forecast hours out to 840 hours. Week 3 - 4 would include forecast hours 504 to 672. Here's the big downside... you'll need to specify each timestep separately in the pcp_combine command and 1 week would correspond to 28 of them! By way of example, I'll list the command for the mean of 8 timesteps. But I'm sure you'll get the point.

pcp_combine -derive mean \
2012011500.mem001.subset.sfcf.all.nc 'name="tref"; level="(20120129_00,*,*)"; file_type=NETCDF_NCCF;' \
2012011500.mem001.subset.sfcf.all.nc 'name="tref"; level="(20120129_06,*,*)"; file_type=NETCDF_NCCF;' \
2012011500.mem001.subset.sfcf.all.nc 'name="tref"; level="(20120129_12,*,*)"; file_type=NETCDF_NCCF;' \
2012011500.mem001.subset.sfcf.all.nc 'name="tref"; level="(20120129_18,*,*)"; file_type=NETCDF_NCCF;' \
2012011500.mem001.subset.sfcf.all.nc 'name="tref"; level="(20120130_00,*,*)"; file_type=NETCDF_NCCF;' \
2012011500.mem001.subset.sfcf.all.nc 'name="tref"; level="(20120130_06,*,*)"; file_type=NETCDF_NCCF;' \
2012011500.mem001.subset.sfcf.all.nc 'name="tref"; level="(20120129_12,*,*)"; file_type=NETCDF_NCCF;' \
2012011500.mem001.subset.sfcf.all.nc 'name="tref"; level="(20120129_18,*,*)"; file_type=NETCDF_NCCF;' \
tref_mean.nc

And then plot it:

plot_data_plane tref_mean.nc tref_mean.ps 'name="tref_mean"; level="(*,*)";'

Obviously listing each timestep separately is unwieldy. If we need to support this type of processing, we could consider enhancing the MET NetCDF library logic to support time ranges. Something like this:

Note this command DOES NOT work:

/Volumes/d1/projects/MET/MET_development/MET-develop/met/bin/pcp_combine -derive mean \
2012011500.mem001.subset.sfcf.all.nc 'name="tref"; level="(20120219_00-20120226_00,*,*)"; file_type=NETCDF_NCCF;' \
tref_mean.nc

[benjamin-cash]

Hi @JohnHalleyGotway - Thanks for the explanation! If the file was instead broken up into individual files for each 6h interval would it be possible to do the calculation in a single condensed command? I combined the files on purpose to avoid a performance issue with xarray's open_mfdataset in another part of my analysis but the single files are available. I could upload a test sample again?

[JohnHalleyGotway]

Ben, you'd still need to list out the individual input file names, each with its own timestamp on the command line. But since the field info would remain constant for all inputs, you'd only need to list it once using the "-field" option.

I am out of the office this week on PTO. But I'm hoping @georgemccabe could take a look at this discussion.

If the pcp_combine METplus wrapper can construct that command for you to compute the 7-day average, then typing it out on the command line isn't such an issue.

@georgemccabe would it be easier to configure the pcp_combine wrapper to process 28 different input files, each with 1 time-step or 1 input files that contains all of them?

[benjamin-cash]

Sounds like it makes more sense just to do the average with nco, which is my current solution. Thanks for looking into this. :)

[georgemccabe]

Hi @benjamin-cash, it is possible to configure PCPCombine wrapper to read the fields from a single input file. However, when trying to test this out with your data, I discovered a bug that messes up the format of file_type=NETCDF_NCCF; because the ending semi-colon is removed. I created GitHub issue #1534 to fix it. With the bug fixed, I was able to create the command to compute the mean with pcp_combine using a METplus config file. Here is how I configured it for my test:

[config]

PROCESS_LIST = PCPCombine

LOOP_BY = INIT
INIT_TIME_FMT = %Y%m%d%H
INIT_BEG = 2012011500
INIT_END = 2012011500
INIT_INCREMENT = 1M

LEAD_SEQ = 28d

FCST_PCP_COMBINE_INPUT_DIR = /d1/projects/METplus/METplus_Data/development/discussions/1519
FCST_PCP_COMBINE_INPUT_TEMPLATE = {init?fmt=%Y%m%d%H}.mem001.subset.sfcf.all.nc

FCST_PCP_COMBINE_OUTPUT_DIR = {OUTPUT_BASE}/pcp_combine
FCST_PCP_COMBINE_OUTPUT_TEMPLATE = tref_mean.nc

FCST_PCP_COMBINE_RUN = True
FCST_PCP_COMBINE_METHOD = DERIVE
FCST_PCP_COMBINE_STAT_LIST = mean

FCST_PCP_COMBINE_CONSTANT_INIT = TRUE

FCST_PCP_COMBINE_INPUT_ACCUMS = 6H
FCST_PCP_COMBINE_INPUT_NAMES = tref
FCST_PCP_COMBINE_INPUT_LEVELS = "({valid?fmt=%Y%m%d_%H},*,*)"
FCST_PCP_COMBINE_INPUT_OPTIONS = file_type=NETCDF_NCCF;

FCST_PCP_COMBINE_OUTPUT_ACCUM = 7d
FCST_PCP_COMBINE_OUTPUT_NAME = tref_mean