HPC integration option 1 - ssh

[rlskoeser]

First and simpler approach for HPC integration is to use ssh access and ssh keys so our app user can login to the cluster as users and start the slurm job as them.

Note that CAS integration (included in #10) is a prerequisite for this.

implementation details

• request access to ssh without duo from test vm to hpc machine
• ensure access to ssh from vm to hpc (may require PUL .lib domain firewall change)
• add a vaulted ssh key to deploy and write instructions for adding to authorized keys on hpc machine
• write setup instructions for hpc conda env
  • Bash script to simplify/automate user setup on hpc #37
• write remote versions equivalent to gpu celery tasks to kick off training jobs: export needed data/model, use ssh to log in as the current user, start the slurm job
  • command-line utility to export data, run segmentation training, and update model with eScriptorium API and slurm #24
  • expand htr2hpc script to support recognition training #27
  • implement method to check status of remote slurm job
  • when the job completes, update the refined model back in escriptorium and report on status
• write htr2hpc celery replacement tasks for train and segtrain to run htr2hpc script on hpc via ssh
• modify escriptorium to call our remote version of the task instead of running locally
• update escriptorium task to report on status of remote slurm job
• update working directory name to use a timestamp, to ensure uniqueness
• delete model if training errors (confirm this is always a new model)
• fix formatting for link to mydella when sending user notifications

[linear]

RSE-100 HPC integration option 1 - ssh

First and simpler approach for HPC integration is to use ssh access and ssh keys so our app user can login to the cluster as users and start the slurm job as them.

Note that CAS integration (included in #10) is a prerequisite for this.

implementation details

• request access to ssh without duo from test vm to hpc machine
• ensure access to ssh from vm to hpc (may require PUL .lib domain firewall change)
• add a vaulted ssh key to deploy and write instructions for adding to authorized keys on hpc machine
• write remote versions equivalent to gpu celery tasks to kick off training jobs: export needed data/model, use scp/rsync to transfer files and ssh to log in as the current user, start the slurm job
• modify escriptorium to call our remote version of the task instead of running locally (think about how to make configurable but this version doesn't have to be elegant)
• implement method to check status of remote slurm job
• modify escriptorium task monitoring to handle remote slurm job
• when the job completes, update the refined model back in escriptorium and report on status

[rlskoeser]

For the new model check (to ensure we don't delete an existing model being updated), I added some logging to compare model creation time with the start time of the task, thought we could use the time delta. Here's the output:

model was created at 2025-01-15 22:19:20.983751+00:00; task started at 2025-01-15 22:19:21.4227
49+00:00; delta: 0:00:00.438998