How to specify METPLUS_OBS_PREPBUFR_MAP

[BinbinZhou-NOAA]

I compared the file PB2NCConfig_wrapped in METplus4.0.0 and METplus4.1.0-beta5, and found the explicit definition for obs_prefbufr_map is removed in beta-5 version, and use METPLUS_OBS_PREPBUFR_MAP Instead.

I tried to find how to specify METPLUS_OBS_PREPBUFR_MAP in the PB2NC conf file but can not find it in the METplus_4.1.0-beta5 online documentation (key = , val = ; pairs}. In the PB2NC use case example of the document, METPLUS_OBS_PREPBUFR_MAP is not specified. This means there are some default values for all possible variables (key)? What is the default val for PBL height, ceiling, visibility, total cloud, ect? I prefer to define the these vals by myself in the conf file.

Binbin Zhou
EMC

[georgemccabe]

PB2NC_OBS_PREPBUFR_MAP is the METplus variable that is used to set this value. The table at the bottom of this section shows the relationship between the settings: https://metplus.readthedocs.io/en/develop/Users_Guide/wrappers.html#pb2nc-met-conf

At the top of the PB2NC: MET Configuration section is a link to the default config file where you can see the default value of the obs_prepbufr_map variable:

To add a new key/val pair to this map, you will need to copy the default value and add the new item(s) to the end of the list. You do not need to include the square braces from the value. A multiple line METplus variable value must contain the same indentation at every line of the value. The line directly after must not have any indentation to signify the end of the value.

PB2NC_OBS_PREPBUFR_MAP =
   { key = "POB";     val = "PRES";  },
   { key = "QOB";     val = "SPFH";  },
   { key = "TOB";     val = "TMP";   },
   { key = "ZOB";     val = "HGT";   },
   { key = "UOB";     val = "UGRD";  },
   { key = "VOB";     val = "VGRD";  },
   { key = "D_DPT";   val = "DPT";   },
   { key = "D_WDIR";  val = "WDIR";  },
   { key = "D_WIND";  val = "WIND";  },
   { key = "D_RH";    val = "RH";    },
   { key = "D_MIXR";  val = "MIXR";  },
   { key = "D_PRMSL"; val = "PRMSL"; },
   { key = "D_PBL";   val = "PBL";   },
   { key = "D_CAPE";  val = "CAPE";  },
   { key = "TOTALCLOUD";  val = "TCDC"; }

It is likely that these new additions to your obs_prepbufr_map will be useful to other users, so it would make sense to add these new key/values to the default config file.

Could you please create a new MET GitHub issue and list the new key/value pairs that are needed to be added to the default?

[BinbinZhou-NOAA]

George,

I removed the space between the quotation mark and semi-colon, then running is successful. But the key ~val match pairs for those new added 4 variables are not fine, shown as below
obs_var =
"TMP",
"UGRD",
"VGRD",
"DPT",
"RH",
"CAPE",
"HOVI",
"TOCC",
"CEILING",
"PMO" ;

The last 4 should be VIS, TCDC, HGT, PRSML

[malloryprow]

@BinbinZhou-NOAA I set

PB2NC_MET_CONFIG_OVERRIDES = obs_prefbufr_map = [ { key = "POB"; val = "PRES"; }, { key = "QOB"; val = "SPFH"; }, { key = "TOB"; val = "TMP"; }, { key = "UOB"; val = "UGRD"; }, { key = "VOB"; val = "VGRD"; }, { key = "D_DPT"; val = "DPT"; }, { key = "D_WDIR"; val = "WDIR"; }, { key = "D_WIND"; val = "WIND"; }, { key = "D_RH"; val = "RH"; }, { key = "D_MIXR"; val = "MIXR"; }, { key = "D_PBL"; val = "HPBL"; }, { key = "D_PRMSL"; val = "PRMSL"; }, { key = "D_CAPE"; val = "CAPE"; }, { key = "TDO"; val = "DPT"; }, { key = "PMO"; val = "PRMSL"; }, { key = "TOCC"; val = "TCDC"; }, { key = "HOVI"; val = "VIS"; }, { key = "CEILING"; val = "HGT"; }, { key = "MXGS"; val = "GUST";} ];

and in my pb2nc netCDF file I see

obs_var =
"SPFH",
"TMP",
"UGRD",
"VGRD",
"DPT",
"RH",
"PRMSL",
"PRMSL",
"GUST",
"DPT",
"VIS",
"HGT",
"TCDC" ;
}

[BinbinZhou-NOAA]

@malloryprow Thanks! I'll test it. What METplus versiosn you used for this case?

[malloryprow]

@BinbinZhou-NOAA I am using MET version 10.1.0-beta5 and METplus version 4.1.0-beta5

[BinbinZhou-NOAA]

@malloryprow
Yes, it works in METplus_10.1.0-beta5! Thanks!
I still hope to update the MET's PB2NC's default config file to inlude PMO (for PRMSL), TDO (for DPT), HOVI (for VIS), CEILING (for HGT), TOCC (for TCDC) to match the field strings used in EMC's forecast files. PMO and TDO are MNEMONIC in PREPBUFR file for mean sea level pressure and dew point. I prefer to use them directly instead of deriving them for observation data

[JohnHalleyGotway]

@BinbinZhou-NOAA, George and I did some more digging and now understand what's going on here. There are a few things to clarify.

1. The reason why it's not working as expected it due to a typo. METplus is defining "obs_prep_bufr_map"... BUT on this line of the PB2NC config file, I see that its named "obs_prefbufr_map" with an 'F' instead of a 'P'!

2. Next, we realize that METplus should NOT actually be setting obs_prefbufr_map (or obs_prepbufr_map). Based on this description of obs_bufr_map, it should instead be defining PB2NC_OBS_BUFR_MAP to set obs_bufr_map in MET.

So @georgemccabe will update METplus to support PB2NC_OBS_BUFR_MAP and will removed PB2NC_OBS_PREPBUFR_MAP.

3. With this issue for MET version 10.1.0, @hsoh-u will also actually fix that typo, changing "obs_prefbufr_map" to "obs_prepbufr_map". And when he does he'll make sure to keep it backward compatible.

Sorry for all the confusion caused by the typo!

[BinbinZhou-NOAA]

@JohnHalleyGotway

That's great to find the reason. If also add following 5 { key ~ val } pairs. it is more convenient for us to generate our pb2nc conf file.
{ key = "HOVI"; val = "VIS"; },
{ key = "TOCC"; val = "TCDC"; },
{ key = "CEILING"; val = "HGT"; },
{ key = "PMO"; val = "PRMSL"; }
{ key = "TDO"; val = "DPT"; }
Dew point and mean sea level pressure can be derived through D_DPT and D_PRMSL,
But since the NCEP's prepbufr files already have dew point (TDO) and mean sea level pressure (PMO) data, we prefer using the observed instead of derived data.

Binbin

[JohnHalleyGotway]

@BinbinZhou-NOAA, thanks for providing this list.

However, I'm concerned about this one:

{ key = "CEILING"; val = "HGT"; },

The model output contains several different types of heights... ceiling, cloud base, cloud top, freezing level. All of these are HGT in the GRIB1/2 outputs but are differentiated by the level "type" value. Renaming "CEILING" obs as "HGT" will likely cause confusion.

I'm also not sure how to handle these 2 you referenced:

{ key = "PMO"; val = "PRMSL"; }
{ key = "TDO"; val = "DPT"; }

If we add those 2, should we REMOVE these existing ones?

   { key = "D_DPT";   val = "DPT";   }
   { key = "D_PRMSL"; val = "PRMSL"; }

Or should we just continue writing out PMO and TDO without renaming them and let the user pick which to use based on their config file settings? You can configure METplus to verify fcst "DPT" vs obs "TDO"... or fcst "DPT" vs obs "DPT".

[BinbinZhou-NOAA]

@JohnHalleyGotway
D_DPT and D_PRMSL can be kept and let user select derived or direct observed.
As for CEILING, I just consider that the ceiling height in NCEP's model outout grib files is represented by "HGT" and "L0". If this is confused with other heights, using CEILING should be fine:
{ key = "CEILING"; va l= "CEILING"; }

Binbin

[JohnHalleyGotway]

@BinbinZhou-NOAA, we discussed this topic at the METplus development meeting today. Our primary concern is that we do not want to change the results for users from MET version 10.0.0 to version 10.1.0. Looking at the proposed changes to the PB2NC default configuration file we've discussed, some of them WOULD change results. Below I've listed a summary of what we'd like to change (or not) and the reasoning for it:

1. Add HOVI and TOCC to OBS_PREPBUFR_MAP in the default PB2NC config file:

{ key = "HOVI"; val = "VIS"; },
{ key = "TOCC"; val = "TCDC"; },

Renaming these to VIS and TCDC makes verifying them against GRIB outputs more convenient.

2. DO NOT add an entry for CEILING. If the input obs name is what we actually want written to the output, there is no need to rename it. Just leave it as is.

3. DO NOT add/modify the settings for D_DPT or D_PRMSL. This is the messy one. If we were to rename PMO to PRMSL and TDO to DPT, that would CHANGE the results for other users. I definitely agree that verifying against the raw obs (PMO or TDO) versus derived obs (D_PRMSL and D_DPT), is a very important consideration. However, right now, in all existing PB2NC output, PRMSL and DPT are actually the derived versions (not the raw ones). We should not repurpose those names in MET version 10.1.0.

However, you still can configure/run PB2NC and Point-Stat to do what you'd like. Just verify forecasts of PRMSL against PMO obs and forecasts of DPT against TDO obs. You'll just need to specify the FCST and OBS variable names in METplus separately. But that usage should work fine.

Or you could, of course, configure METplus to do whatever you'd like. Any settings in the METPLUS_OBS_BUFR_MAP take precedence over the default settings in OBS_PREPBUFR_MAP.

I'd also recommend that the NOAA/EVS team give this some thought. What conventions do you want to define/follow for differentiating between raw observations and derived types? If those conventions do require changes to the PB2NC config file, we could consider making them in a major release, like MET version 11.0.0, rather than a minor one, like version 10.1.0.

[BinbinZhou-NOAA]

John, Thanks for this update and explanation. . No problem to just add VIS and TCDC and keep others unchanged. We can handle this on our side and won't mess up current MET codes and scripts due to our requests. Binbin

…

On Tue, Feb 22, 2022 at 1:03 PM johnhg ***@***.***> wrote: @BinbinZhou-NOAA <https://github.com/BinbinZhou-NOAA>, we discussed this topic at the METplus development meeting today. Our primary concern is that we do not want to change the results for users from MET version 10.0.0 to version 10.1.0. Looking at the proposed changes to the PB2NC default configuration file we've discussed, some of them WOULD change results. Below I've listed a summary of what we'd like to change (or not) and the reasoning for it: 1. *Add* HOVI and TOCC to OBS_PREPBUFR_MAP in the default PB2NC config file: { key = "HOVI"; val = "VIS"; }, { key = "TOCC"; val = "TCDC"; }, Renaming these to VIS and TCDC makes verifying them against GRIB outputs more convenient. 1. *DO NOT* add an entry for CEILING. If the input obs name is what we actually want written to the output, there is no need to rename it. Just leave it as is. 2. *DO NOT* add/modify the settings for D_DPT or D_PRMSL. This is the messy one. If we were to rename PMO to PRMSL and TDO to DPT, that would CHANGE the results for other users. I definitely agree that verifying against the raw obs (PMO or TDO) versus derived obs (D_PRMSL and D_DPT), is a very important consideration. However, right now, in all existing PB2NC output, PRMSL and DPT are actually the derived versions (not the raw ones). We should not repurpose those names in MET version 10.1.0. However, you still can configure/run PB2NC and Point-Stat to do what you'd like. Just verify forecasts of PRMSL against PMO obs and forecasts of DPT against TDO obs. You'll just need to specify the FCST and OBS variable names in METplus separately. But that usage should work fine. Or you could, of course, configure METplus to do whatever you'd like. Any settings in the METPLUS_OBS_BUFR_MAP take precedence over the default settings in OBS_PREPBUFR_MAP. I'd also recommend that the NOAA/EVS team give this some thought. What conventions do you want to define/follow for differentiating between raw observations and derived types? If those conventions do require changes to the PB2NC config file, we could consider making them in a major release, like MET version 11.0.0, rather than a minor one, like version 10.1.0. — Reply to this email directly, view it on GitHub <#1417 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFLYEPLTMX7CH2MH7XDU4PFVZANCNFSM5NX6SACA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou IMSG at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683