[BUG] geo2rdr did not converge for GMAO - think it's about Availability?

[cmarshak]

Describe the bug
This is related to hyp3 deployment and the current script to update GUNWs.

Could be potentially very simple, i.e. due to the fact that the reference date is 20230106 and GMAO is not available. If such is the case, may be helpful to create slightly better error message here.

Here is the job dict:

{'job_id': '5225cd05-0869-4031-a697-15778a29635f',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-02-13T22:40:20+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Tibet_48_beta_gmao',
  'job_parameters': {'estimate_ionosphere_delay': True,
   'frame_id': 7420,
   'granules': ['S1A_IW_SLC__1SDV_20230106T235934_20230107T000001_046670_059804_D8E3',
    'S1A_IW_SLC__1SDV_20230106T235959_20230107T000026_046670_059804_74DB'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20221213T235935_20221214T000002_046320_058C2F_7DFC',
    'S1A_IW_SLC__1SDV_20221214T000000_20221214T000027_046320_058C2F_9AD4'],
   'weather_model': 'GMAO'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/5225cd05-0869-4031-a697-15778a29635f/5225cd05-0869-4031-a697-15778a29635f.log'],
  'expiration_time': '2023-08-14T00:00:00+00:00',
  'processing_times': [8891.189, 23.475]}

The log (as above) is here: https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/5225cd05-0869-4031-a697-15778a29635f/5225cd05-0869-4031-a697-15778a29635f.log

The GUNW is here (without tropo layer): https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/5225cd05-0869-4031-a697-15778a29635f/S1-GUNW-D-R-048-tops-20230106_20221214-000000-00090E_00040N-PP-c254-v2_0_6.nc

So using the GUNW, this script could be re-run.

To Reproduce
Steps to reproduce the behavior:

raider.py ++process calcDelaysGUNW --model GMAO <GUNW_ID>

See ASFHyP3/hyp3#1416 for exactly how raider is called.

[cmarshak]

Here are the failed jobs for reference (there are 3 / 18 submitted that failed):

[{'job_id': '5225cd05-0869-4031-a697-15778a29635f',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-02-13T22:40:20+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Tibet_48_beta_gmao',
  'job_parameters': {'estimate_ionosphere_delay': True,
   'frame_id': 7420,
   'granules': ['S1A_IW_SLC__1SDV_20230106T235934_20230107T000001_046670_059804_D8E3',
    'S1A_IW_SLC__1SDV_20230106T235959_20230107T000026_046670_059804_74DB'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20221213T235935_20221214T000002_046320_058C2F_7DFC',
    'S1A_IW_SLC__1SDV_20221214T000000_20221214T000027_046320_058C2F_9AD4'],
   'weather_model': 'GMAO'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/5225cd05-0869-4031-a697-15778a29635f/5225cd05-0869-4031-a697-15778a29635f.log'],
  'expiration_time': '2023-08-14T00:00:00+00:00',
  'processing_times': [8891.189, 23.475]},
 {'job_id': '14dae435-36fb-4797-9ebd-250434d3d25c',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-02-13T22:40:20+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Tibet_48_beta_gmao',
  'job_parameters': {'estimate_ionosphere_delay': True,
   'frame_id': 7420,
   'granules': ['S1A_IW_SLC__1SDV_20221225T235935_20221226T000002_046495_059224_A539',
    'S1A_IW_SLC__1SDV_20221225T235959_20221226T000026_046495_059224_1CFC'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20221213T235935_20221214T000002_046320_058C2F_7DFC',
    'S1A_IW_SLC__1SDV_20221214T000000_20221214T000027_046320_058C2F_9AD4'],
   'weather_model': 'GMAO'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/14dae435-36fb-4797-9ebd-250434d3d25c/14dae435-36fb-4797-9ebd-250434d3d25c.log'],
  'expiration_time': '2023-08-14T00:00:00+00:00',
  'processing_times': [8888.974, 23.301]},
 {'job_id': 'a3162b13-8591-4cd4-93e2-a02b56100d04',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-02-13T22:40:20+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Tibet_48_beta_gmao',
  'job_parameters': {'estimate_ionosphere_delay': True,
   'frame_id': 7420,
   'granules': ['S1A_IW_SLC__1SDV_20230106T235934_20230107T000001_046670_059804_D8E3',
    'S1A_IW_SLC__1SDV_20230106T235959_20230107T000026_046670_059804_74DB'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20221225T235935_20221226T000002_046495_059224_A539',
    'S1A_IW_SLC__1SDV_20221225T235959_20221226T000026_046495_059224_1CFC'],
   'weather_model': 'GMAO'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/a3162b13-8591-4cd4-93e2-a02b56100d04/a3162b13-8591-4cd4-93e2-a02b56100d04.log'],
  'expiration_time': '2023-08-14T00:00:00+00:00',
  'processing_times': [8891.929, 23.342]}]

[cmarshak]

It is easy to get the relevant GUNW (without tropo) from the failed jobs. Replace the <log id> in the log url with the GUNW Id (the GUNW id is found in the log). Know that's a bit circular - hopefully makes sense.

https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/a3162b13-8591-4cd4-93e2-a02b56100d04/a3162b13-8591-4cd4-93e2-a02b56100d04.log --> https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/a3162b13-8591-4cd4-93e2-a02b56100d04/S1-GUNW-D-R-048-tops-20230106_20221225-000000-00090E_00040N-PP-212a-v2_0_6.nc

[dbekaert]

I think the orbits and SLC seems to overlap.
e.g. Compare the SLCs with the validity of the orbit:
'secondary_granules': ['S1A_IW_SLC__1SDV_20221213T235935_20221214T000002_046320_058C2F_7DFC',
'S1A_IW_SLC__1SDV_20221214T000000_20221214T000027_046320_058C2F_9AD4'],
with:
S1A_OPER_AUX_POEORB_OPOD_20230103T081835_V20221213T225942_20221215T005942.EOF

@jlmaurer we might need to have a debug look at it?

[bbuzz31]

Yeah, this is more nefarious than the availability unfortunately. Possibly related to the padding of the orbit file, although the specific bug is:

Traceback (most recent call last):
  File "/Users/buzzanga/Software_InSAR/RAiDER_git/tools/RAiDER/delay.py", line 373, in _build_cube_ray
    nParts = int(np.ceil(ray_length.max() / MAX_SEGMENT_LENGTH)) + 1
ValueError: cannot convert float NaN to integer

ray_length is all nans, perhaps @piyushrpt knows what's going on?

[bbuzz31]

This bug is indeed related to the padding parameter (which slices the state vectors in the orbit file around the GUNW reference UTC time).

If extra padding is used (>41 minutes; the default is 10 minutes) then geo2rdr converges.
It's not clear why this is needed: the reference time at 00:00:00 UTC, so adding padding of +/- 10 minutes is intuitively sufficient, as the total SLCs range is 23:59:34 to 00:00:27.
And if this extra padding is used, previously converging locations fail (e.g., test_scenario_3.py)

@piyushrpt could you help us understand what's happening here?

[cmarshak]

@bbuzz31 - looks like it's happening near a midnight crossing (or maybe I am just being Mr. Obvious) - might be helpful to see if that has something to do with it either requesting data within Raider, insider Raider's processing with ISCE3, or even ISCE3 itself. I know @rzinke had issue with Tibet AOIs in ISCE2 due to the midnight crossing. Maybe it is similar.

[cmarshak]

I specifically selected this sample due to its midnight crossing ;) FYI - Thanks, Rob!

[bbuzz31]

@cmarshak yeah this may have something to do with it... did any of the 15 that succeeded have a midnight crossing?

[piyushrpt]

If 10 minutes around the reference time in insufficient, I would check the source granule. With SAR, time is same as spatial extent - i.e, where the satellite is looking. If it fails to converge with the right time window - it probably means that the assigned reference time is off or that when building the orbit data structure wrong times are being assigned to the state vector.

[cmarshak]

Thank youk @piyushrpt - Brett, do not use the GUNW name for date and time, use the source granule for date/time.

We can fix the GUNW name (where you are looking) but that will not solve the problem for older products (lots over Tibet with midnight crossing). There will one SLC in reference and secondary to get data you need. That will be sufficient.

Here is the issue above using the first failed job.

gunw_path = 'S1-GUNW-D-R-048-tops-20230106_20221214-000000-00090E_00040N-PP-c254-v2_0_6.nc'

The first date is 20230106 but the center time is in the next day.

S1A_IW_SLC__1SDV_20230106T235934_20230107T000001_046670_059804_D8E3

Use the SLC id to figure out date and time together.

https://github.com/dbekaert/RAiDER/blob/dev/tools/RAiDER/aria/prepFromGUNW.py#L65-L75

[cmarshak]

So we can fix GUNW to be center date as well as center time (the former is earliest date).

Here is the issue we are seeing:

import os
from dateparser import parse

def get_dates(path):
    """ Get the ref/sec date from the filename """
    ref, sec = os.path.basename(path).split('-')[6].split('_')

    return int(ref), int(sec)


def get_time(path):
    """ Get the center time of the secondary date from the filename """
    tt = os.path.basename(path).split('-')[7]
    return f'{tt[:2]}:{tt[2:4]}:{tt[4:]}'
        

gunw_path = 'S1-GUNW-D-R-048-tops-20230106_20221214-000000-00090E_00040N-PP-c254-v2_0_6.nc'
slc_id_ref = 'S1A_IW_SLC__1SDV_20230106T235934_20230107T000001_046670_059804_D8E3'

get_dates(gunw_path), get_time(gunw_path)

Outputs

((20230106, 20221214), '00:00:00')

The first date is the reference date (start) and the latter is the acq time (center). They are in different dates. So raider is thinking the acq is 24 hours earlier than it should be.

This would be fixed if (a) GUNW had both as center or (b) this datetime is read using SLC IDs - you only need one reference and one secondary granule to get the orbit datetimes you need. We will do the fix for (a) but to ensure its correct for older GUNWs - please read the SLC id.

This is thanks to @dbekaert.

[cmarshak]

For everyone's knowledge, I updated the Hyp3 Plugin that generates gunw so the final id of this product would be something like 'S1-GUNW-D-R-048-tops-20230106_20221214-235959-00090E_00040N-PP-c254-v2_0_6'.

Here is the PR: ACCESS-Cloud-Based-InSAR/DockerizedTopsApp#112

[bbuzz31]

Fixed in #500