Update TNRIS Lidar dataset & database against new projects in TNRIS S3 bucket

[dhardestylewis]

Update TNRIS Lidar dataset & database against new projects in TNRIS S3 bucket

for example, stratmap21-50cm is reported in the 2022/05/09 TNRIS availability file but not in terrain_aggregator DB

[dhardestylewis]

include in 2nd grand retiling if possible

[dhardestylewis]

• Bexar & Travis Counties Lidar 2021

[dhardestylewis]

updating TACC's TNRIS Lidar DEM collection against the current TNRIS AWS S3 now

All TNRIS Lidar DEMs & LPCs can be found at the following location at TACC:
stampede2.tacc.utexas.edu:/scratch/projects/tnris/tnris-lidardata

Please contact me if you have any issues with accessing the data.

The following Lidar collections are being pulled now:

stratmap-2021-28cm-50cm-bexar-travis
stratmap-2021-laguna-madre-bathy
stratmap-2022-50cm-ellis-hill-johnson-navarro
stratmap-2022-address-points
stratmap-2022-land-parcels
usgs-hydrography
utbeg-2017-70cm-laguna-madre

Here is how they are being pulled:

cd /scratch/projects/tnris
while IFS= read -r line ; do aws s3 cp s3://tnris-public-data/production-data/${line}/ tnris-lidardata/${line} --recursive & done < tnris-missing-data.csv

tnris-missing-data.csv was produced using

cd /scratch/projects/tnris
aws s3 ls s3://tnris-public-data/production-data/ > tnris-public-data.csv
## use vim magic to modify above file slightly to strip out everything but the collection names
ls tnris-lidardata > tnris-tacc-data.csv
diff --suppress-common-lines -y tnris-public-data.csv tnris-tacc-data.csv > tnris-missing-data.csv
## use vim magic to capture the missing collections on the left side of the diff

I have applied the following permissions against this dataset:

cd /scratch/projects/tnris
find . -type d -exec bash -c "chgrp G-822180 {} ; chmod 755 {} ; chmod g+s {} ; setfacl -d -m g::rx,o::rx {}" \;
find . -type f -exec bash -c "chgrp G-822180 {} ; chmod 644 {}" \;

[dhardestylewis]

• once downloaded, next step is to upload this data using the instructions in
  https://github.com/dhardestylewis/terrain_aggregator#updating-the-database-with-new-lidar-tiles

[dhardestylewis]

#79

• ⏸️ inclusion of latest TNRIS Lidar DEM tiles in terrain_aggregator DB paused to return TNRIS, TDIS, & CWE requests for small-scale 1m accuracy results

[dhardestylewis]

#39 (comment)

• TNRIS Lidar DEM tile collections have been updated successfully!

• This includes all TNRIS Lidar point-cloud data!

cd /scratch/projects/tnris
while IFS= read -r directory ; do echo "${directory} $(~/local/bin/aws/aws s3 ls s3://tnris-public-data/production-data/${directory}/ --recursive | wc -l)" ; done < tnris-missing-data.csv

stratmap-2021-28cm-50cm-bexar-travis 48922
stratmap-2021-laguna-madre-bathy 189742
stratmap-2022-50cm-ellis-hill-johnson-navarro 13156
stratmap-2022-address-points 19459
stratmap-2022-land-parcels 18532
usgs-hydrography 956
utbeg-2017-70cm-laguna-madre 680

cd /scratch/projects/tnris
while IFS= read -r directory ; do echo "${directory} $(find tnris-lidardata/${directory} -type f | wc -l)" ; done < tnris-missing-data.csv

stratmap-2021-28cm-50cm-bexar-travis 48922
stratmap-2021-laguna-madre-bathy 189742
stratmap-2022-50cm-ellis-hill-johnson-navarro 13156
stratmap-2022-address-points 19459
stratmap-2022-land-parcels 18532
usgs-hydrography 956
utbeg-2017-70cm-laguna-madre 680

[dhardestylewis]

• cron job of Update TNRIS Lidar dataset & database against new projects in TNRIS S3 bucket #39 (comment)