Statanlysis config file

[perthsb]

Hi,
I am having an issue with Stat Analysis configuration used after running GridStat component. I did not specify any output stat template for GridStat, so it creates stat files with templates. Now I am trying to use that output to run Stat Analysis but giving error that 05/15 18:35:46.122Z metplus (command_builder.py:257) ERROR: (stat_analysis_wrapper.py:1341) No value found for lookin dir
05/15 18:35:46.123Z metplus (command_builder.py:257) ERROR: (stat_analysis_wrapper.py:248) Could not get runtime settings dict list

Files are in Cactus : run gridstat using /lfs/h2/emc/vpppg/noscrub/Partha.Bhattacharjee/Aerosol/METPlus/ICAP/run_metplus_icapaod.sh followed by
/lfs/h2/emc/vpppg/noscrub/Partha.Bhattacharjee/Aerosol/METPlus/ICAP/run_metplus_StatAnly_ICAPaod.sh

Any help on how to sort this issue.
Thanks.

[georgemccabe]

Hi @perthsb, it looks like the StatAnalysis variable names in your config file should be named with MODEL1_ instead of MODEL_, e.g.:

MODEL_STAT_ANALYSIS_LOOKIN_DIR should be MODEL1_STAT_ANALYSIS_LOOKIN_DIR
MODEL_STAT_ANALYSIS_DUMP_ROW_TEMPLATE should be MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE
MODEL should be MODEL1
OBTYPE should be MODEL1_OBTYPE

I realize that this is confusing if you are only providing 1 model to process via StatAnalysis. We could enhance the wrapper to look for the model variables without a number if no variables with numbers are found. I created issue #2176 to track this enhancement. It is not marked as high priority because there is a workaround, but please let us know if this should be made a higher priority.

Please let me know if you are still having issues after renaming the config variables.

[perthsb]

Thanks @georgemccabe for your answer. I have modified the changes and run stat analysis but it did not provide any output from statanalysis. config file I transferred to Hera /scratch1/NCEPDEV/stmp2/Partha.Bhattacharjee/METplus/Scripts/VIIRS

[georgemccabe]

According to the latest log, none of the stat lines are being retained:

DEBUG 2: Processing 21 STAT files.
DEBUG 2: STAT Lines read = 42
DEBUG 2: STAT Lines retained = 0
DEBUG 2:
DEBUG 2: Processing Job 1: -job aggregate_stat -model GEFSAERO -fcst_valid_hour 030000 -fcst_var AOTK -obtype VIIRS -vx_mask FULL -line_type SL1L2 -out_line_type CNT -out_alpha 0.05000 -rank_corr_flag 0

This means that your job arguments are filtering out all of the data. The warning log that follows reiterates this:

WARNING: do_job_aggr_stat() -> no matching STAT lines found for job: -job aggregate_stat -model GEFSAERO -fcst_valid_hour 030000 -fcst_var AOTK -obtype VIIRS -vx_mask FULL -line_type SL1L2 -out_line_type CNT -out_alpha 0.05000 -rank_corr_flag 0

You can increase the verbosity for stat_analysis (using LOG_STAT_ANALYSIS_VERBOSITY) and/or compare the values in the job arguments to your data to see why all of the lines are being discarded.

In this case it looks like you are requesting fcst valid hour 3 (-fcst_valid_hour 030000):

FCST_VALID_HOUR_LIST = 3

but the valid hours of your GridStat output are 0, 6, 12, 18. Adjusting this value (or unsetting it) should prevent the data to be filtered out due to valid time. The all of the lines are still being filtered out, increasing the verbosity should give you a hint as to why.

[georgemccabe]

Hi @perthsb, were you able to get the desired result from StatAnalysis? Please let me know if you need any more assistance.