Need help on the configuration setting for older NCEP AQM output verification #1558
-
|
I have a scripts that works for the current NCEP operational AQM producing hourly point_stat statistic. I want to go back to 2019 to re-process the operational AQM (a different one) at that time for additional statistic. The same script found the obs and forecast files but fail to find any match. I try to change the FCST_LEVEL from A1 to L1 but did not work. I need expert from the DTC to help spot the error in the point_stat statistic. The batch job script and output logfile can be found in https://www.emc.ncep.noaa.gov/mmb/hchuang/methelp/ My config setting can be found on the github page The data location is at WCOSS, Problem for all ozone1, ozone8, and PMTF processing |
Beta Was this translation helpful? Give feedback.
Replies: 8 comments 2 replies
-
|
Thank you for sending the log files and parameter file locations. In your log file, I see lines that look like this:
This means that the PB2NC steps are not running because the data already exists and you have your configuration set to skip files that already exist. This may be why the results for PointStat are not including the new output you expect. To get the PB2NC portion to run, you can either:
Let me know if you continue to have issues and I can take another look. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, George:
Perry please watch this issue.
I repeat the same run so many times so I skip the reproduction of pb2nc.
Please correct me if I am wrong, they should be the same when I re-run for
the 2nd time. It is located in a different directory from experiment
v160a. If you feel this is a necessary step, please let me know again.
My key issue is the point-stat does not find a match between obs input and
fcst input and therefore zero line of statistics. It should have no
relation to pb2nc already created, unless it is created wrong. I believe
the log file finds both the FCST and OBS file correctly.
for example
DEBUG 1: Forecast File:
/gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/verification/aqm/prod/aqm.20190730/aqm.t06z.pm25.f42.148.grib2
DEBUG 1: Observation File:
/gpfs/dell2/emc/verification/noscrub/Ho-Chun.Huang/metplus_aq/aqm/conus_sfc/prod/pm/
prepbufr.pm.20190731.nc
Ho-Chun Huang, Ph.D.
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
***@***.*** ***@***.***>
301-683-3958
…On Wed, Apr 6, 2022 at 3:09 PM George McCabe ***@***.***> wrote:
Hi @Ho-ChunHuang-NOAA <https://github.com/Ho-ChunHuang-NOAA>,
Thank you for sending the log files and parameter file locations. In your
log file, I see lines that look like this:
04/06 16:32:02.584 metplus (command_builder.py:933) DEBUG: Skip writing
output
/gpfs/dell2/emc/verification/noscrub/Ho-Chun.Huang/metplus_aq/aqm/conus_sfc/prod/pm/
prepbufr.pm.20190731.nc because it already exists. Remove file or change
PB2NC_SKIP_IF_OUTPUT_EXISTS to False to process
This means that the PB2NC steps are not running because the data already
exists and you have your configuration set to skip files that already
exist. This may be why the results for PointStat are not including the new
output you expect. To get the PB2NC portion to run, you can either:
- Set PB2NC_SKIP_IF_OUTPUT_EXISTS = False to overwrite the existing
files
- Remove the data in the output directory for PB2NC (set in
PB2NC_OUTPUT_DIR)
- Change the OUTPUT_BASE value and write to an entirely new directory
Let me know if you continue to have issues and I can take another look.
—
Reply to this email directly, view it on GitHub
<#1558 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALPHE3GU7MJ5AQ7JRE67IVDVDXONHANCNFSM5SWYD3SQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi, George:
The run includes two parts, first the PB2NC and followed by Point_Stat. I
believe the PB2NC part is okay. The problem I asked is the 2nd part , the
Point_Stat.
Thanks.
Ho-Chun Huang, Ph.D.
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
***@***.*** ***@***.***>
301-683-3958
On Wed, Apr 6, 2022 at 3:20 PM Ho-Chun Huang - NOAA Affiliate <
***@***.***> wrote:
… Hi, George:
Perry please watch this issue.
I repeat the same run so many times so I skip the reproduction of pb2nc.
Please correct me if I am wrong, they should be the same when I re-run for
the 2nd time. It is located in a different directory from experiment
v160a. If you feel this is a necessary step, please let me know again.
My key issue is the point-stat does not find a match between obs input and
fcst input and therefore zero line of statistics. It should have no
relation to pb2nc already created, unless it is created wrong. I believe
the log file finds both the FCST and OBS file correctly.
for example
DEBUG 1: Forecast File:
/gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/verification/aqm/prod/aqm.20190730/aqm.t06z.pm25.f42.148.grib2
DEBUG 1: Observation File:
/gpfs/dell2/emc/verification/noscrub/Ho-Chun.Huang/metplus_aq/aqm/conus_sfc/prod/pm/
prepbufr.pm.20190731.nc
Ho-Chun Huang, Ph.D.
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
***@***.*** ***@***.***>
301-683-3958
On Wed, Apr 6, 2022 at 3:09 PM George McCabe ***@***.***>
wrote:
> Hi @Ho-ChunHuang-NOAA <https://github.com/Ho-ChunHuang-NOAA>,
>
> Thank you for sending the log files and parameter file locations. In your
> log file, I see lines that look like this:
>
> 04/06 16:32:02.584 metplus (command_builder.py:933) DEBUG: Skip writing
> output
> /gpfs/dell2/emc/verification/noscrub/Ho-Chun.Huang/metplus_aq/aqm/conus_sfc/prod/pm/
> prepbufr.pm.20190731.nc because it already exists. Remove file or change
> PB2NC_SKIP_IF_OUTPUT_EXISTS to False to process
>
> This means that the PB2NC steps are not running because the data already
> exists and you have your configuration set to skip files that already
> exist. This may be why the results for PointStat are not including the new
> output you expect. To get the PB2NC portion to run, you can either:
>
> - Set PB2NC_SKIP_IF_OUTPUT_EXISTS = False to overwrite the existing
> files
> - Remove the data in the output directory for PB2NC (set in
> PB2NC_OUTPUT_DIR)
> - Change the OUTPUT_BASE value and write to an entirely new directory
>
> Let me know if you continue to have issues and I can take another look.
>
> —
> Reply to this email directly, view it on GitHub
> <#1558 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALPHE3GU7MJ5AQ7JRE67IVDVDXONHANCNFSM5SWYD3SQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
Beta Was this translation helpful? Give feedback.
-
|
Yes, it is OK to skip the PB2NC step if you don't need to reprocess that data. If you had modified the configuration to add data to convert in that step, then you would need to rerun it. From the log output from Point-Stat, it looks like the matched pairs are being rejected based on the valid time:
@JohnHalleyGotway pulled some of your files from WCOSS to investigate. The valid time of the forecast data you are reading is:
and this is the file that contains the observations that are used:
Checking the timestamps in that file:
I see hourly output from 20190801_010000 to 20190802_000000. The PointStat config file you are using (PointStatConfig_ANOWPM) does not contain many of the environment variables that are set by METplus. It is hard-coded to use
so even though your METplus config is setting the obs_window to +/- 86400, this is not what is actually being used by the point_stat app. That's why there's 0 matches. The forecast is valid at 20190801_000000 but there are no obs with that timestamp. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, George:
You are right, my example is not a good one. I might pick the wrong
message,
my verification time window is
export DATE=20190731
export DATEp1=20190801
cat << EOF > shared.conf_aq
[config]
VALID_BEG = ${DATE}01
VALID_END = ${DATEp1}00
OBS_WINDOW_BEGIN = -86400
OBS_WINDOW_END = 86400
This is the last (or last two) conf file so it should overwrite the default.
It is strange that all valid valid hours produced null statistics. Let me
say it again, the current setting works for other AQM experiments.
For valid time of 20190801_000000, it should find the forecast file
aqm.20190731/aqm.t06z.awpozcon_bc.f18.148.grib2,
aqm.20190731/aqm.t12z.awpozcon_bc.f12.148.grib2, and so on.
So there must be some matches. Can you look for other matches?
Ho-Chun Huang, Ph.D.
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
***@***.*** ***@***.***>
301-683-3958
…On Thu, Apr 7, 2022 at 11:26 AM George McCabe ***@***.***> wrote:
Hi @Ho-ChunHuang-NOAA <https://github.com/Ho-ChunHuang-NOAA>,
Yes, it is OK to skip the PB2NC step if you don't need to reprocess that
data. If you had modified the configuration to add data to convert in that
step, then you would need to rerun it.
From the log output from Point-Stat, it looks like the matched pairs are
being rejected based on the valid time:
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 21273
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 0
DEBUG 2: Rejected: valid time = 21273
@JohnHalleyGotway <https://github.com/JohnHalleyGotway> pulled some of
your files from WCOSS to investigate. The valid time of the forecast data
you are reading is:
DEBUG 4: valid time: 20190801_000000
and this is the file that contains the observations that are used:
/gpfs/dell2/emc/verification/noscrub/Ho-Chun.Huang/metplus_aq/aqm/conus_sfc/prod/pm/
prepbufr.pm.20190731.nc
Checking the timestamps in that file:
ncdump -v hdr_vld_table prepbufr.pm.20190731.nc | grep 2019 | sort -u
I see hourly output from 20190801_010000 to 20190802_000000. The PointStat
config file you are using (PointStatConfig_ANOWPM) does not contain many of
the environment variables that are set by METplus. It is hard-coded to use
obs_window = { beg = 0; end = 0; }
so even though your METplus config is setting the obs_window to +/- 86400,
this is not what is actually being used by the point_stat app.
That's why there's 0 matches. The forecast is valid at 20190801_000000 but
there are no obs with that timestamp.
—
Reply to this email directly, view it on GitHub
<#1558 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALPHE3B22CHA5AQSYM2PKBDVD35A3ANCNFSM5SWYD3SQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
It looks like some of your PointStatConfig files reference environment variables to set the obs_window and some do not. The example you provided does not (beg and end are 0). Here is one that does: This is one of the reasons why we now recommend that users always use the wrapped MET config files that are provided with the METplus wrapper repository, such as PointStatConfig_wrapped. All overrides from the default values can be done in the METplus config file instead of having to set them in your own MET config files. I outlined how to make these updates for the CAM config files in this discussion: #996 (comment) From the example you supplied earlier, the METplus config file parm/use_cases/perry/point_stat_aq.conf is where POINT_STAT_CONFIG_FILE is set:
Instead of using this file, you can use the one that is provided with METplus:
This config file will reference the environment variable ${METPLUS_OBS_WINDOW_DICT}, which according to your logs is set in your run to:
There are other values in PointStatConfig_AIRNOW that differ from the defaults config file (found in the MET repository and listed in the point_stat log output). I compared these files to determine the following variables should be added to the point_stat_aq.conf file: The following variables in point_stat_aq.conf don't correspond with the values set in PointStatConfig_AIRNOW, so they should be changed if they need to match what is used in that file: Change:
to
Change:
to
Those changes should use the same configurations as before but with the properly values for the obs_window used so that the observations will not be rejected based on the valid time. Let me know if that does not work. It would be good to update all of your use cases so that they match this format. It will make it much easier to debug issues going forward. I'm happy to help with this process if desired. Let me know what you think. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, George:
Thanks, that is a lot for me to comprehend. I will take some time to test
and coordinate with Perry for any changes.
Ho-Chun Huang, Ph.D.
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
***@***.*** ***@***.***>
301-683-3958
…On Thu, Apr 7, 2022 at 3:51 PM George McCabe ***@***.***> wrote:
It looks like some of your PointStatConfig files reference environment
variables to set the obs_window and some do not. The example you provided
does not (beg and end are 0). Here is one that does:
https://github.com/Ho-ChunHuang-NOAA/EMC_AQM_Verification/blob/4f2d6430a105a4dcaca667e82f71629b0c5811f4/parm/met_config/PointStatConfig_cam#L90-L93
obs_window = {
beg = ${OBS_WINDOW_BEGIN};
end = ${OBS_WINDOW_END};
}
This is one of the reasons why now recommend that users always use the
wrapped MET config files that are provided with the METplus wrapper
repository, such as PointStatConfig_wrapped. All overrides from the default
values can be done in the METplus config file instead of having to set them
in your own MET config files. I outlined how to make these updates for the
CAM config files in this discussion: #996 (comment)
<#996 (comment)>
From the example you supplied earlier, the METplus config file
parm/use_cases/perry/point_stat_aq.conf is where POINT_STAT_CONFIG_FILE is
set:
POINT_STAT_CONFIG_FILE ={PARM_BASE}/met_config/PointStatConfig_AIRNOW
Instead of using this file, you can use the one that is provided with
METplus:
POINT_STAT_CONFIG_FILE ={PARM_BASE}/met_config/PointStatConfig_wrapped
This config file will reference the environment variable
${METPLUS_OBS_WINDOW_DICT}, which according to your logs is set in your run
to:
INFO: METPLUS_OBS_WINDOW_DICT=obs_window = {beg = -86400;end = 86400;}
There are other values in PointStatConfig_AIRNOW that differ from the
defaults config file (found in the MET repository and listed in the
point_stat log output). I compared these files to determine the following
variables should be added to the point_stat_aq.conf file:
POINT_STAT_REGRID_METHOD = BILIN
POINT_STAT_REGRID_WIDTH = 2
POINT_STAT_INTERP_TYPE_METHOD = BILIN
POINT_STAT_INTERP_TYPE_WIDTH = 2
POINT_STAT_OUTPUT_FLAG_FHO = STAT
POINT_STAT_OUTPUT_FLAG_CTC = STAT
POINT_STAT_OUTPUT_FLAG_SL1L2 = STAT
POINT_STAT_OUTPUT_PREFIX = {MODEL}_AIRNOW
POINT_STAT_MET_CONFIG_OVERRIDES = cat_thresh = [ >50, >60, >65, >70, >75, >85, >105, >125, >150 ];
The following variables in point_stat_aq.conf don't correspond with the
values set in PointStatConfig_AIRNOW, so they should be changed if they
need to match what is used in that file:
Change:
POINT_STAT_MESSAGE_TYPE = ONLYSF
to
POINT_STAT_MESSAGE_TYPE = AIRNOW
Change:
POINT_STAT_POLY =
to
POINT_STAT_POLY = "MET_BASE/poly/CONUS.poly", "MET_BASE/poly/EAST.poly",
"MET_BASE/poly/WEST.poly",
"MET_BASE/poly/NEC.poly","MET_BASE/poly/SEC.poly",
"MET_BASE/poly/NWC.poly", "MET_BASE/poly/SWC.poly",
"MET_BASE/poly/NMT.poly", "MET_BASE/poly/SMT.poly",
"MET_BASE/poly/GRB.poly", "MET_BASE/poly/SWD.poly",
"MET_BASE/poly/NPL.poly", "MET_BASE/poly/SPL.poly",
"MET_BASE/poly/MDW.poly", "MET_BASE/poly/LMV.poly",
"MET_BASE/poly/GMC.poly", "MET_BASE/poly/APL.poly"
Those changes should use the same configurations as before but with the
properly values for the obs_window used so that the observations will not
be rejected based on the valid time. Let me know if that does not work.
It would be good to update all of your use cases so that they match this
format. It will make it much easier to debug issues going forward. I'm
happy to help with this process if desired. Let me know what you think.
—
Reply to this email directly, view it on GitHub
<#1558 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALPHE3ARABQD7CZX3NGZXJDVD44CNANCNFSM5SWYD3SQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi @Ho-ChunHuang-NOAA , I am just following up on this discussion. Were you able to test the changes and get the correct result? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, George:
Please close the ticket. I decided not to change my code/setting. It
works for other periods and the new Inline CMAQ model.
Ho-Chun Huang, Ph.D.
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
***@***.*** ***@***.***>
301-683-3958
…On Mon, May 2, 2022 at 5:06 PM George McCabe ***@***.***> wrote:
Hi @Ho-ChunHuang-NOAA <https://github.com/Ho-ChunHuang-NOAA> , I am just
following up on this discussion. Were you able to test the changes and get
the correct result?
—
Reply to this email directly, view it on GitHub
<#1558 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALPHE3A6K7WBQ4W3EAAKSOLVIA7TZANCNFSM5SWYD3SQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
This comment describes how the METplus configuration files could be updated to use the wrapped version of the MET config file that is provided with the repository and set all of the configurations in METplus configuration files to avoid inconsistency between multiple MET config files. |
Beta Was this translation helpful? Give feedback.
This comment describes how the METplus configuration files could be updated to use the wrapped version of the MET config file that is provided with the repository and set all of the configurations in METplus configuration files to avoid inconsistency between multiple MET config files.