Issues reading NetCDF time information

[jir245]

Hi,

I'm trying to process NetCDF input files through grid_stat. The NetCDF file is CF compliant ( Conventions = "CF-1.0" ; is included in global attributes) and the file includes a "time" variable that describes the forecast valid time as well as a forecast reference time variable to describe the forecast initial time. I've played around with the units and naming of the time variables, but I can't seem to get grid_stat or plot_data_plane to properly read in the valid time, init time and lead time. Part of the header is shown below, including time information:

netcdf test {
dimensions:
sp = 5 ;
latitude = 540 ;
longitude = 1080 ;
time = 1 ;
variables:
string sp(sp) ;
sp:long_name = "aerosol or gas species or type" ;
float latitude(latitude) ;
latitude:_FillValue = NaNf ;
latitude:units = "degrees_north" ;
latitude:standard_name = "latitude" ;
float longitude(longitude) ;
longitude:_FillValue = NaNf ;
longitude:units = "degrees_east" ;
longitude:standard_name = "longitude" ;
int64 time(time) ;
time:standard_name = "time" ;
time:units = "seconds since 2019-09-02 00:00:00" ;
int64 forecast_reference_time(time) ;
forecast_reference_time:standard_name = "forecast_reference_time" ;
forecast_reference_time:units = "seconds since 2019-09-02 00:00:00" ;

The valid time should be 20190903 for a 24 hour forecast with initial time of 20190902. Forecast_reference_time value = 0 and time variable is 86400. The output from plot_data_plane is:

[rubin@heck bin]$ plot_data_plane test.nc test.ps 'name="aot_tot";level="(0,,)"; file_type=NETCDF_NCCF;' -v 4
DEBUG 1: Opening data file: test.nc
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "seconds since 2019-09-02 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "seconds since 2019-09-02 00:00:00"
DEBUG 4: as a reference time of 20190902_000000 and 1 second(s) per time step.
DEBUG 4: get_nc_data(NcVar *, double *) add_offset = 0, scale_factor=1, cell_count=1, is_unsigned_value: 0
DEBUG 1: get_nc_data(NcVar *, double *) type_id: 10 type name: int64
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "seconds since 2019-09-02 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "seconds since 2019-09-02 00:00:00"
DEBUG 4: as a reference time of 20190902_000000 and 1 second(s) per time step.
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 4:
DEBUG 4: Data plane information:
DEBUG 4: plane min: 0.00242465
DEBUG 4: plane max: 8.85684
DEBUG 4: valid time: 20190902_000000
DEBUG 4: lead time: 000000
DEBUG 4: init time: 20190902_000000
DEBUG 4: accum time: 000000
DEBUG 4:
DEBUG 1: Creating postscript file: test.ps
DEBUG 4: draw_map() -> Plotting map data file: /software/depot/met-10.0.1/share/met/map/country_data
DEBUG 4: draw_map() -> Plotting map data file: /software/depot/met-10.0.1/share/met/map/usa_state_data
DEBUG 4: draw_map() -> Plotting map data file: /software/depot/met-10.0.1/share/met/map/major_lakes_data

Is there a different way I should be specifying the time information?

Thanks for your help!

[j-opatz]

Hello,

And thank you for your question. As you may have seen in the MET user's guide, MET tools will use the time variable for the valid time, and the forecast_reference_time as the init time. As you have indicated, these variables are present in your netCDF CF-compliant file, so it was a bit puzzling to see that your valid and init times were being listed as the same by plot-data-plane.
After speaking with one of our lead MET engineers, it sounds like there may be an issue with the aot_tot variable. Specifically, it needs to include the time dimension. Taking a look at a CF-convention example from the CF-conventions website:

variables:
float xwind(time,pres,lat,lon) ;
xwind:long_name = "zonal wind" ;
xwind:units = "m/s" ;

Note how the variable xwind is defined by time, in addition to the lat, lon, and pres variables.

Is the aot_tot variable also defined by the time variable you've created? In your copy of the netcdf content, I don't see where aot_tot is defined. But as suggested in the CF-conventions, making sure aot_tot is defined by the time variable should correct this issue.

[jir245]

Hi,

Thanks for getting back to me. Sorry I failed to include the aot_tot variable in the information above. Here is how it is defined in the netcdf file:

    double aot_tot(time, latitude, longitude) ;
            aot_tot:_FillValue = NaN ;
            aot_tot:units = "1" ;
            aot_tot:wavelength_nm = 550L ;
            aot_tot:standard_name = "atmosphere_optical_thickness_due_to_ambient_aerosol_particles" ;

So it does include the time variable in the dimensions. I've tried variations on the time variables, but nothing seems to work. I have no idea why. I could send you a full data file if that would be helpful?

[j-opatz]

That would be very helpful, thanks! It's strange to see the time dimension in there and the resulting MET tool behavior.

Please see this post in regards to sending along data.

[jir245]

I think I figured it out. MET didn't like the time variables specified as int64. I changed these to int and got output from plot_data_plane as expected:

plot_data_plane test_new.nc test.ps 'name="aot_tot";level="(0,,)";' -v 4
DEBUG 1: Opening data file: test_new.nc
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "seconds since 2019-09-02 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "seconds since 2019-09-02 00:00:00"
DEBUG 4: as a reference time of 20190902_000000 and 1 second(s) per time step.
DEBUG 4: get_nc_data(NcVar *, double *) add_offset = 0, scale_factor=1, cell_count=1, is_unsigned_value: 0
DEBUG 4: apply_scale_factor(double) unpacked data: count=0 out of 1. FillValue(int)=-9999 data range [86400 - 86400] raw data: [86400 - 86400] Positive count: 1
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "seconds since 2019-09-02 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "seconds since 2019-09-02 00:00:00"
DEBUG 4: as a reference time of 20190902_000000 and 1 second(s) per time step.
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 4:
DEBUG 4: Data plane information:
DEBUG 4: plane min: 0.00242465
DEBUG 4: plane max: 8.85684
DEBUG 4: valid time: 20190903_000000
DEBUG 4: lead time: 240000
DEBUG 4: init time: 20190902_000000
DEBUG 4: accum time: 000000

[j-opatz]

That's great to hear, and thank you for continuing to try and find a solution to this.

I'm going to mark this question as answered and lock the discussion, but I will bring this finding (int64 not being a valid type for time variables) to our engineers and see if this is expected and if it can be noted in our documentation going forward.

[j-opatz]

Hi @jir245,

I'd like to reach out about this again, as there has been more information brought to my attention on this.

After our last major MET release (10.1.0), a bug very similar to yours was found and fixed in the MET 10.1.1. Here's a link to the Github issue describing it. There's a very good chance that the error you encountered is fixed in the latest version, but I want to confirm this.

Would you be able to send a copy of the non-working file to us for testing? It would help ensure this has been accounted for in the METplus suite.

[jir245]

test.nc <https://drive.google.com/file/d/1haQufmXwWZDT3oyUebuI-qavdtkVgknp/view?usp=drive_web> Hi, Thanks for letting me know about this. I still have an older version of Met (10.0.1) so I should probably update to a newer version. There is a google drive link for the test.nc file that I used with the original int64 in it. Hope this is helpful for you guys. Thanks for all of your help, Juli

…

On Tue, May 10, 2022 at 1:20 PM j-opatz ***@***.***> wrote: Hi @jir245 <https://github.com/jir245>, I'd like to reach out about this again, as there has been more information brought to my attention on this. After our last major MET release (10.1.0), a bug very similar to yours was found and fixed in the MET 10.1.1. Here's a link to the Github issue <dtcenter/MET#2123> describing it. There's a very good chance that the error you encountered is fixed in the latest version, but I want to confirm this. Would you be able to send a copy of the *non-working* file to us for testing? It would help ensure this has been accounted for in the METplus suite. — Reply to this email directly, view it on GitHub <#1602 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYBBMFMOEL3EY2OFWEHFWWTVJLAJTANCNFSM5VF4F33A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[j-opatz]

Hi Juli,

And thank you for providing the file. I was able to test this out today, both against the MET v10.1.0 and MET v10.1.1 for the plot-data-plane tool.

As was suspected, this was a part of the bug that the issue I linked was discussing.

Here is the -v 4 printout for v10.1.0:

Opening data file: test.nc
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "seconds since 2019-09-02 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "seconds since 2019-09-02 00:00:00"
DEBUG 4: 		 as a reference time of 20190902_000000 and 1 second(s) per time step.
DEBUG 4: get_nc_data(NcVar *, double *) add_offset = 0, scale_factor=1, cell_count=1, is_unsigned_value: 0
DEBUG 1: get_nc_data(NcVar *, double *) type_id: 10 type name: int64
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "seconds since 2019-09-02 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "seconds since 2019-09-02 00:00:00"
DEBUG 4: 		 as a reference time of 20190902_000000 and 1 second(s) per time step.
DEBUG 4: Cannot wrap longitudes since 360 is not evenly divisible by delta_lon = 0.333328.
DEBUG 4: Cannot wrap longitudes since 360 is not evenly divisible by delta_lon = 0.333328.
DEBUG 4: Cannot wrap longitudes since 360 is not evenly divisible by delta_lon = 0.333328.
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 4: 
DEBUG 4: Grid information:
DEBUG 4:    Projection: Lat/Lon
DEBUG 4:    Nx: 1080
DEBUG 4:    Ny: 540
DEBUG 4:    lat_ll: -89.833
DEBUG 4:    lon_ll: 179.833
DEBUG 4:    delta_lat: 0.333
DEBUG 4:    delta_lon: 0.333
DEBUG 4:    wrapLon: false
DEBUG 4: 
DEBUG 4: Data plane information:
DEBUG 4:    plane min: 0.00242465
DEBUG 4:    plane max: 8.85684
DEBUG 4:    valid time: 20190902_000000
DEBUG 4:    lead time: 000000
DEBUG 4:    init time: 20190902_000000
DEBUG 4:    accum time: 000000

Note that it's replicating the behavior that you found, with the valid and init time being the exact same.

Now here's the same file, but using v10.1.1 (aka after the issue was fixed):

Opening data file: test.nc
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "seconds since 2019-09-02 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "seconds since 2019-09-02 00:00:00"
DEBUG 4: 		 as a reference time of 20190902_000000 and 1 second(s) per time step.
DEBUG 4: get_nc_data(NcVar *, double *) add_offset = 0, scale_factor=1, cell_count=1, is_unsigned_value: 0 for time
DEBUG 4: apply_scale_factor(double) unpacked data: count=0 out of 1. FillValue(int64)=-9999 data range [86400 - 86400] raw data: [86400 - 86400] Positive count: 1
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "seconds since 2019-09-02 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "seconds since 2019-09-02 00:00:00"
DEBUG 4: 		 as a reference time of 20190902_000000 and 1 second(s) per time step.
DEBUG 4: Cannot wrap longitudes since 360 is not evenly divisible by delta_lon = 0.333328.
DEBUG 4: Cannot wrap longitudes since 360 is not evenly divisible by delta_lon = 0.333328.
DEBUG 4: Cannot wrap longitudes since 360 is not evenly divisible by delta_lon = 0.333328.
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 4: 
DEBUG 4: Grid information:
DEBUG 4:    Projection: Lat/Lon
DEBUG 4:    Nx: 1080
DEBUG 4:    Ny: 540
DEBUG 4:    lat_ll: -89.833
DEBUG 4:    lon_ll: 179.833
DEBUG 4:    delta_lat: 0.333
DEBUG 4:    delta_lon: 0.333
DEBUG 4:    wrapLon: false
DEBUG 4: 
DEBUG 4: Data plane information:
DEBUG 4:    plane min: 0.00242465
DEBUG 4:    plane max: 8.85684
DEBUG 4:    valid time: 20190903_000000
DEBUG 4:    lead time: 240000
DEBUG 4:    init time: 20190902_000000
DEBUG 4:    accum time: 000000

Now we see the lead time is set correctly (240000) and the valid time is exactly 24 hours after the init time.
It's great to see that you were able to find the alternative solution to this, and hopefully when you're able to obtain the latest version of MET you don't have this worry anymore!