time format issue in config file

[perthsb]

Hi,
I am trying to run Asciitonc and poinstat script using the latest build on Cactus and I am getting valid time format issue which I am not able to resolve.
Location : /lfs/h2/emc/vpppg/noscrub/Partha.Bhattacharjee/Aerosol/METPlus/Aeronet_download
script : run_metplus_PointstatAeronet_Glb.sh
log : /lfs/h2/emc/vpppg/noscrub/Partha.Bhattacharjee/Aerosol/METplus/Aeronet_download/output_Poinstat/logs/metplus.log.20230428164813

Any suggestion on this will be helpful.

[jprestop]

HI @perthsb. @JohnHalleyGotway is on Discussions support on Monday and will either reply or assign this Discussion to someone then. Since I am the only one with access to Cactus, I grabbed your directory and put it on the ftp server using the following commands, following the instructions from How to Send Us Data #954.

mkdir -p /lfs/h2/emc/vpppg/noscrub/julie.prestopnik/user_support/partha_2152
cd !$
cp -r /lfs/h2/emc/vpppg/noscrub/Partha.Bhattacharjee/Aerosol/METPlus/Aeronet_download/ .
tar -cvf Aeronet_download.tar
ftp -p ftp.rap.ucar.edu
When prompted for my name, I put "anonymous"
When prompted for my password, I typed my email address.
cd incoming/irap/met_help
mkdir bhattacharjee_data
put Aeronet_download.tar
bye

See below for more detail:

julie.prestopnik@clogin06:/lfs/h2/emc/vpppg/noscrub/julie.prestopnik/user_support/partha_2152> cp -r /lfs/h2/emc/vpppg/noscrub/Partha.Bhattacharjee/Aerosol/METPlus/Aeronet_download/ .
julie.prestopnik@clogin06:/lfs/h2/emc/vpppg/noscrub/julie.prestopnik/user_support/partha_2152> tar -cvf Aeronet_download.tar Aeronet_download/
julie.prestopnik@clogin06:/lfs/h2/emc/vpppg/noscrub/julie.prestopnik/user_support/partha_2152> ls
Aeronet_download  Aeronet_download.tar
julie.prestopnik@clogin06:/lfs/h2/emc/vpppg/noscrub/julie.prestopnik/user_support/partha_2152> ftp -p ftp.rap.ucar.edu
Wrapper for lftp to simulate compatibility with lukemftp
Name (julie.prestopnik): anonymous
Password:                                
lftp [email protected]:~> cd incoming/irap/met_help
---- Connecting to ftp.rap.ucar.edu (128.117.14.132) port 21
<--- 220-                                    
<--- 220-   ***               National Center for Atmospheric Research              ***
<--- 220-   ***                    Research Applications Program                    ***
<--- 220-
<--- 220-   ***   Use of this system for unauthorized purposes is a violation of    ***
<--- 220-   ***   federal law. See Public Law 99-474. Any activity may be logged    ***
<--- 220-   ***   and monitored. By using this system, you consent to such logging  ***
<--- 220-   ***   and monitoring.                                                   ***
<--- 220-
<--- 220-   The local time is: Fri Apr 28 12:02:03 2023
<--- 220-                                          
<--- 220-   If you have problems logging in, please try using a dash (-) as the first
<--- 220-   character of your password -- this will turn off the continuation messages
<--- 220-   that may be confusing your ftp client.
<--- 220-
<--- 220-   If you have unusual problems, please contact: [email protected]
<--- 220-
<--- 220-   ***  NCAR users: *Do not* use your RAL, CIT or UCAS password here!!!    ***
<--- 220-   ***  Please contact the RAL Helpdesk if you require assistance using    ***
<--- 220-   ***  this FTP service.                                                  ***
<--- 220-
<--- 220 128.117.14.132 FTP server ready
<--- 230-Login access granted.                     
<--- 230-   Authenticated as user: anonymous
<--- 230 Anonymous access granted, restrictions apply
<--- 250 CWD command successful                    
cd ok, cwd=/incoming/irap/met_help
lftp [email protected]:/incoming/irap/met_help> mkdir bhattacharjee_data
mkdir ok, `bhattacharjee_data' created                
lftp [email protected]:/incoming/irap/met_help> cd bhattacharjee_data/
lftp [email protected]:/incoming/irap/met_help/bhattacharjee_data> put Aeronet_download.tar
1403054080 bytes transferred in 141 seconds (9.49 MiB/s)                                       
lftp [email protected]:/incoming/irap/met_help/bhattacharjee_data> ls
-rw-r--r--   1 ftp      ftp      1403054080 Apr 28 17:52 Aeronet_download.tar
lftp [email protected]:/incoming/irap/met_help/bhattacharjee_data> bye
julie.prestopnik@clogin06:/lfs/h2/emc/vpppg/noscrub/julie.prestopnik/user_support/partha_2152> 

For reference, here is the error Partha is receiving:

...
04/28 16:48:13.904Z metplus (config_metplus.py:644) DEBUG: Setting [config] LOG_ASCII2NC_VERBOSITY to default value (2)
04/28 16:48:13.905Z metplus.ASCII2NC (run_util.py:158) ERROR: Fatal error occurred
Traceback (most recent call last):
  File "/apps/ops/para/libs/intel/19.1.3.304/metplus/5.0.1/metplus/util/run_util.py", line 99, in run_metplus
    getattr(module, f"{process}Wrapper")(config,
  File "/apps/ops/para/libs/intel/19.1.3.304/metplus/5.0.1/metplus/wrappers/ascii2nc_wrapper.py", line 35, in __init__
    super().__init__(config, instance=instance)
  File "/apps/ops/para/libs/intel/19.1.3.304/metplus/5.0.1/metplus/wrappers/command_builder.py", line 92, in __init__
    self.c_dict = self.create_c_dict()
  File "/apps/ops/para/libs/intel/19.1.3.304/metplus/5.0.1/metplus/wrappers/ascii2nc_wrapper.py", line 60, in create_c_dict
    c_dict['OBS_INPUT_DIR'] = self.config.getdir('ASCII2NC_INPUT_DIR',
  File "/apps/ops/para/libs/intel/19.1.3.304/metplus/5.0.1/metplus/util/config_metplus.py", line 686, in getdir
    dir_path = super().getstr('config', dir_name, default=None,
  File "/apps/ops/para/libs/intel/19.1.3.304/metplus/5.0.1/produtil/config.py", line 1266, in getstr
    return self._get(sec,opt,str,default,badtypeok,morevars,taskvars=taskvars)
  File "/apps/ops/para/libs/intel/19.1.3.304/metplus/5.0.1/produtil/config.py", line 1211, in _get
    s=self._interp(sec,opt,morevars=morevars,taskvars=taskvars)
  File "/apps/ops/para/libs/intel/19.1.3.304/metplus/5.0.1/produtil/config.py", line 1178, in _interp
    return self._formatter.format(got,
  File "/apps/spack/python/3.10.4/intel/19.1.3.304/xqft4d45h4dp4xnbz2ue3nbxv65i6bgp/lib/python3.10/string.py", line 161, in format
    return self.vformat(format_string, args, kwargs)
  File "/apps/spack/python/3.10.4/intel/19.1.3.304/xqft4d45h4dp4xnbz2ue3nbxv65i6bgp/lib/python3.10/string.py", line 165, in vformat
    result, _ = self._vformat(format_string, args, kwargs, used_args, 2)
  File "/apps/spack/python/3.10.4/intel/19.1.3.304/xqft4d45h4dp4xnbz2ue3nbxv65i6bgp/lib/python3.10/string.py", line 205, in _vformat
    obj, arg_used = self.get_field(field_name, args, kwargs)
  File "/apps/spack/python/3.10.4/intel/19.1.3.304/xqft4d45h4dp4xnbz2ue3nbxv65i6bgp/lib/python3.10/string.py", line 270, in get_field
    obj = self.get_value(first, args, kwargs)
  File "/apps/ops/para/libs/intel/19.1.3.304/metplus/5.0.1/produtil/config.py", line 169, in get_value
    raise KeyError(key)
KeyError: 'valid?fmt=%Y%m%d'
...

[perthsb]

Thanks @jprestop . Let me know if you need any more data to send.

[JohnHalleyGotway]

Thanks @perthsb for providing sample data and thanks @jprestop for grabbing it.

The good news is that this error is easily fixed. I checked in with @georgemccabe to confirm. Please take a look at these settings in Pointstat_AeronetAOD2.conf:

ASCII2NC_INPUT_DIR = /lfs/h1/ops/dev/dcom/{valid?fmt=%Y%m%d}/validation_data/aq/aeronet
ASCII2NC_INPUT_TEMPLATE = {ASCII2NC_INPUT_DIR}/{valid?fmt=%Y%m%d}.lev15

Per George, template tags can only be found in the _TEMPLATE variable, not the _DIR variable. Also, they don't need to include ASCII2NC_INPUT_DIR in the _TEMPLATE path. They are combined automatically.

Please try reconfiguring/rerunning with the following settings instead:

ASCII2NC_INPUT_DIR = /lfs/h1/ops/dev/dcom
ASCII2NC_INPUT_TEMPLATE = {valid?fmt=%Y%m%d}/validation_data/aq/aeronet/{valid?fmt=%Y%m%d}.lev15

We searched through the METplus User's Guide and found that this detail is NOT well documented. So we'll figure out where to add it.

Thanks for helping to point out that short-coming in the docs.

[JohnHalleyGotway]

FYI, @perthsb, I wrote up GitHub issue #2157 to make sure we remember to update the documentation prior to the next release.

[georgemccabe]

Hi @perthsb, I have made some changes to the logic so that filename template tags should be supported in _DIR variables and not just the corresponding _TEMPLATE variables. This should resolve some of the confusion of where the filename template tags can be used in the METplus configuration files.