Using Observation Climatology in Point Stat

[mc155123]

I am using point stat in MetPlus/5.0 to verify model against point surface observations, starting with 2 meter temperature for example.

The command is

point_stat -v 5 t2m_2024110200.grb2 t2m_2024110200.nc config_file -outdir output_dir 

metplus has returned successfully, and produced some expected outputs. See attached log file.

Although found it, metplus did not use the climatology, and issued some warnings

DEBUG 2: --------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Reading data for TMP/A0.
DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 3520 Ny: 2875 lat_ll: -46.800 lon_ll: -109.504 delta_lat: 0.013 delta_lon: 0.013 wrapLon: false
DEBUG 2: Processing masking regions.
DEBUG 3: Processing grid mask: FULL
DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
DEBUG 2: Processing geography data.
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_None".
WARNING:
WARNING: read_ct2m_2024110200.nclimo_file() -> Trouble reading climatology file "obs_climatology_00.nc"
WARNING:
DEBUG 3: Found 0 climatology fields.
DEBUG 2: For TMP/A0 found 1 forecast levels, 0 climatology mean levels, and 0 climatology standard deviation levels.
DEBUG 2:
DEBUG 2: --------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Searching 605 observations from 605 messages.
DEBUG 4: For TMP/A0 versus TMP/Z0, skipping observation with bad data value:
DEBUG 4:    ADPSFC 7600 20241102_000000 -28.11560 -117.84250 407.00000 TMP 0.00000 408.53427 NA -9999.00000
DEBUG 4: For TMP/A0 versus TMP/Z0, skipping observation with bad data value:
DEBUG 4:    ADPSFC 27054 20241102_000000 -10.05110 -143.06860 3.50000 TMP 0.00000 0.00000 NA -9999.00000
DEBUG 4: For TMP/A0 versus TMP/Z0, skipping observation with bad data value:
DEBUG 4:    ADPSFC 33307 20241102_000000 -19.41680 -146.53619 556.00000 TMP 0.00000 539.88690 NA -9999.00000

The obs climatology file is provided via variables below. However, I am not sure how to define the format of the obs climatology file, and what metplus expects it to contain.

POINT_STAT_CLIMO_MEAN_INPUT_DIR
POINT_STAT_CLIMO_MEAN_INPUT_TEMPLATE

I hoped that metplus may accept the obs climatology with the same format and content as the file for observations. That is how the obs climatology file was created.

I have included following files in an ttached zip file

• log file from metplus
• final config file
• t2m from model in grib2 format
• t2m observations in netcdf format, which was created with ascii2nc tool from metplus. The csv file used by ascii2nc is also included.

Thanks for looking into this problem. I am new to MetPlus. The problem I have maybe due to incorrect configuration, or file format, or something else.

point_stat_clim.zip

[CPKalb]

Hi:

When looking at your log file (and as you posted above), I see the following:

WARNING:
WARNING: read_ct2m_2024110200.nclimo_file() -> Trouble reading climatology file "obs_climatology_00.nc"
WARNING:

This means that MET was not able to read your climatology file. I notice in the files that you sent over that the climatology file contains point observations. At this time MET is not able to use point data as the climatology; it has to be gridded data (with the exception of the SEEPS score). This is why MET is unable to read you file.

You may be able to create gridded observations using MET's point2grid tool. I'm listing below a few examples of how to set up the configuration for point2grid:
https://metplus.readthedocs.io/en/main_v6.0/generated/met_tool_wrapper/Point2Grid/Point2Grid.html
https://metplus.readthedocs.io/en/main_v6.0/generated/model_applications/short_range/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect.html

Your climatology file is missing some needed global attributes, so it will not go directly through point2grid. If you look in the t2m_2024110200.nc file you sent, it has the following attributes:

// global attributes:
		:MET_Obs_version = "1.02" ;
		:use_var_id = "true" ;
		:FileOrigins = "File / generated 20241231_115646 UTC on host gadi-login-09.gadi.nci.org.au by the MET ascii2nc tool" ;
		:MET_version = "V11.0.2" ;
		:MET_tool = "ascii2nc" ;

Something similar needs to be tin the obs_climatology_00.nc for it to be able to be read by point2grid. Do you have access to NCO tools? If so, you can easily add these attributes using ncatted.

It's possible that at some point in the future we could consider enhancing MET to use point observations for climatology, but this need has not previously come up and so it is not available at this time.

Christina

[mc155123]

Hi Christina,

Thanks so much for looking into my problem, and prviding very informative suggestions.

Following your suggestions, I used point2grib tool, and created grided obs netcdf file. I only made very small change to the example Point2Grid.conf, point2gril succeeded when using t2m obs file, as you pointed out, the obs climatology mean file missing global attributes. However, I am not sure about settings, particular, regrid method. below is what I have, nd full final config is attached.

# Value to pass with the -method argumen - Default is UW_MEAN, other examples are
#POINT2GRID_REGRID_METHOD = MAX 
POINT2GRID_REGRID_METHOD =

I then cpplied output file from point2grid, grid_obs_climatology_00.nc, to point_stat, and I did not change anything else. This time, the climatology filel is read in, even though point_stat failed in the end, complaint about climatology file

DEBUG 2: Computing Scalar Partial Sums and Continuous Statistics.
DEBUG 2: Processing TMP/A0 versus TMP/Z0, for observation type ADPSFC, over region FULL, for interpolation method HiRA Ensemble NBRHD(9), using 257 matched pairs.
DEBUG 3: Computing ensemble statistics relative to the climatological mean.
ERROR  : 
ERROR  : derive_climo_prob() -> The range of climatological probability values [9.18012, 35.9382] falls outside the expected range of [0, 1].
ERROR  : When verifying a probabilistic forecast using climatology data, either supply a probabilistic climo_mean field or non-probabilistic
ERROR  : climo_mean and climo_stdev fields from which a normal approximation of the climatological probabilities should be derived.
ERROR  : 

I think it is possible that the obs climatology file grid_obs_climatology_00.nc is not quite as what metplus expects, it is supposed to contains mean observed temperature over a period of time. I did not provide climo_stdev.

It is very likely configurations I provided to metplus, both in point2grib and in point_stat, have mistakes. I have attached following
files to this discussion: log and final config for both point2grib and point_stat, also file grid_obs_climatology_00.nc

point2grib/grid_obs_climatology_00.nc		point_stat/metplus.log.20250110062850
point2grib/metplus.log.20250110165733		point_stat/metplus_final.conf.20250110062850
point2grib/metplus_final.conf

I look forward to any suggestions you can offer.
Thanks again,
MC

metplus_p2g_and_ps.zip

[CPKalb]

For point2grid, the regrid method is a choice meaning that there isn't one "correct" one to use. It's really about which one makes the most sense for the data you have and what you want to evaluate. I don't see any obvious problems with the Point2Grid configurations. The main thing to make sure here is that it is going on the same grid as your model data.

The error with the climatology file has to do with how probability data is handled in MET. As it stats, MET expects probabilistic data to be a range of 0 to 1. Your data isn't probabilistic and so your temperatures are not 0 to 1. However, it looks like you have HIRA turned on, which is what computes the probabilistic output. I'm guessing this is why MET wants the climatology to be probabilities.

@JohnHalleyGotway could you take a look at this Monday? I'm not certain exactly what MET needs for climatologies with HIRA.

Christina

[mc155123]

Hi Christina,

Thanks for answering my questions. I made a small progress, which also raised another question.

I created stdev for obs climatology, and converted both mean and stdev point netcdf file to grid file using point2grid tool. I then use them in point_stat. This time, metplus succeeded, however, more than half of observation points (341 out of 605) were rejected. See log messages below.

DEBUG 2: Processing TMP/A0 versus TMP/Z0, for observation type ADPSFC, over region FULL, for interpolation method BILIN(4), using 257 matched pairs.
DEBUG 3: Number of matched pairs   = 257
DEBUG 3: Observations processed    = 605
DEBUG 3: Rejected: station id      = 0
DEBUG 3: Rejected: obs var name    = 0
DEBUG 3: Rejected: valid time      = 0
DEBUG 3: Rejected: bad obs value   = 7
DEBUG 3: Rejected: off the grid    = 0
DEBUG 3: Rejected: topography      = 0
DEBUG 3: Rejected: level mismatch  = 0
DEBUG 3: Rejected: quality marker  = 0
DEBUG 3: Rejected: message type    = 0
DEBUG 3: Rejected: masking region  = 0
DEBUG 3: Rejected: bad fcst value  = 0
DEBUG 3: Rejected: bad climo mean  = 341
DEBUG 3: Rejected: bad climo stdev = 0
DEBUG 3: Rejected: mpr filter      = 0
DEBUG 3: Rejected: duplicates      = 0
DEBUG 2: Computing Categorical Statistics.

when using point2grid tool, natually many grid points are filled with missing values. (see attached orginal nc file and the grid nc file). I think this is why many observations were rejected.

I am not sure how point2grid tool works. Is it possible to config the tool to create grid file that retains all points within the input point nc file, put them on the nearest grid points in the output grid nc file?

I attached point_stat log and final config file, and input point nc and output grid nc file.

grid_stdev_obs_climatology_00.nc	metplus_final.conf.20250110114214
metplus.log.20250110114214		stdev_obs_climatology_00.nc 

Thanks so uch for your help.

MC
point_stat_use_clim.zip

[JohnHalleyGotway]

@mc155123 thanks for passing along your data. I work with @CPKalb who's out of the office this week at the American Met Society conference. I see that you have a point climatology data source that you'd like to use in MET. As @CPKalb points out, MET only supports gridded climatologies at this time, so you are attempting to "grid" this climatology data and have had some limited success in doing so.

You ran point2grid, something like this:

met-11.1.0/bin/point2grid t2m_2024110200.nc t2m_2024110200.grb2 t2m_2024110200_grid.nc -field 'name="TMP"; level="Z0";'  -v 3

And the result of doing so, is that only 595 of ~10 million grid points have values:

DEBUG 2: process_point_met_data() -> var_count=605, grid: 595 out of 10120000   

You're wondering whether point2grid provides a "nearest-neighbor" option so that each point value is used for all grid points for which its closest. It does not have that option. Instead, logic is provided to determine what to do with multiple point obs falling in the same grid box.

It is possible to run regrid_data_plane to "fatten-up" the values, replacing the value at each grid point with the "max" of the values in some radius around it. But that wouldn't work well for nearby point obs. You'd want to use the closest one rather than the "max" value.

I suspect the best path forward will be to process the gridded "climo" points in Python to what you'd like... defining the value at each grid point using the value of the nearest climo point observation.

[mc155123]

Hi @JohnHalleyGotway ,
Thanks for your suggestions. your command point2grid below is interesting

met-11.1.0/bin/point2grid t2m_2024110200.nc t2m_2024110200.grb2 t2m_2024110200_grid.nc -field 'name="TMP"; level="Z0";'  -v 3

if this command succeeded abd created grid nc file that retained 595 obs, that will be very promissing. I think my previous attempt must only created TMP on 257 grib points, and the rest points have missing values.

I tried your command

echo point2grid $f $fn grid_${f} \-field 'name="TMP"; level="Z0";' -v 3
point2grid mean_obs_climatology_00.nc t2m_2024110921.grb2 grid_mean_obs_climatology_00.nc -field name="TMP"; level="Z0"; -v 3

but the command did not work for me see message below. I am using metplus/5.0.1

DEBUG 1: Start point2grid by ycx548(5420) at 2025-01-14 07:46:38Z  cmd: point2grid mean_obs_climatology_00.nc t2m_2024110921.grb2 grid_mean_obs_climatology_
00.nc -field name=TMP
ERROR  : 
ERROR  : yyerror() -> syntax error in file "/tmp/met_config_778465_1"
ERROR  :    
ERROR  :    line   = 2
ERROR  :    
ERROR  :    column = 3
ERROR  :    
ERROR  :    text   = "(nul)"
ERROR  :
ERROR  : 
ERROR  : (nul)
ERROR  : (nul)
ERROR  : 
/build/entrypoint.sh: line 34: -v: command not found

any suggestions?

Thanks
MC

[JohnHalleyGotway]

@mc155123, I had assumed that that was the point2grid command you were running when you originally said...

I created stdev for obs climatology, and converted both mean and stdev point netcdf file to grid file using point2grid tool.

That runtime error is most likely due to missing single quotes on the command line. The presence of double and single quotes can get rather finicky. The entire field string should be enclosed in single quotes, with double-quotes strings on the inside, like this:

point2grid mean_obs_climatology_00.nc t2m_2024110921.grb2 grid_mean_obs_climatology_00.nc -field 'name="TMP"; level="Z0";' -v 3

I'm not really sure what the previous echo line is doing, but if you want the single quotes to be preserved in the output, you'll probably need to escape them with a backslash, like this:

echo point2grid $f $fn grid_${f} \-field \'name="TMP"; level="Z0";\' -v 3

I want to point out that I DID NOT generate 595 matched pairs. I was just pointing out this log message printed from point2grid:

DEBUG 2: process_point_met_data() -> var_count=605, grid: 595 out of 10120000

This is stating that your very dense grid over Australia has 10,120,000 grid points in it... and only 595 of those grid points contain non-empty values, which is not very many!

I'll point out that MET does include logic to automatically regrid gridded climo data to the forecast domain on the fly. So your climo data does NOT need to be on the same grid as your forecast. You could run point2grid to create a much lower resolution climatology (e.g. perhaps 0.25 degree or 0.5 degree lat/lon or something). I'm guessing with that approach, you would get matched pairs for all 605 points. However, the big problem with this approach is that you most likely won't actually be using the correct climo value.

The best solution would be for us to enhance MET to support point climatology data directly, rather than just gridded climos. Does BOM typically use point climos instead of grids? But that would require writing up a development issue, identifying a funding source, and working it into our future development timeline.

Another (somewhat kludgy and rather involved) approach would be this...

1. Run point_stat WITHOUT specifying any climo information. That'll get you all 605 matched pairs. And write the MPR line type to the output. The FCST/OBS_CLIMO_MEAN/STDEV columns will all contain NA since you provided no climo data.
2. Write a Python script to read the MPR output from Point-Stat, and for each line, extract the climo mean and std dev from your point obs climo dataset... and overwrite the FCST/OBS_CLIMO_MEAN/STDEV columns with your real values.
3. Run the stat_analysis tool to read those patched MPR lines and compute continuous and/or categorical statistics.

Honestly, I think it'd probably be simpler to write a Python script to read the NetCDF output from point2grid and replace each NA value with the nearest non-NA value. And then you could run point_stat using the climo inputs directly.

[mc155123]

Hi @JohnHalleyGotway ,

Thanks for your explaination. Being new to metplus, I did not realise that metplus only takes grided climatology data. AS you pointed out, met does regrid on the fly, I can get some lower resolution grided climatology data and feed it to point_stat. That is perhaps easiest at this point.

Our point based verification is somewhat limited. I am not aware if anyone is using point base climotology in verification. I will switch to use low res climatology.

Thanks to both of you for your help.
MC

[JohnHalleyGotway]

Sounds great. Thanks for following up. I'll go ahead and close this discussion.

I'll note that I did just add a note about this to our (brand new) METplus Idea Catcher Project Board. Although it sounds like you don't see an immediate need for METplus to support point climatology data at this time.