Revise training script to update model

[rlskoeser]

work related to #39 #34 and closing out #15

• add update option to htr2hpc-train script
  • set accuracy and file-size when uploading model
• finish implementing train celery task / htr2hpc integration
• delete model on training error (unless pre-existing / overwrite)

[cmroughan]

There's a bug causing the script to crash with a "'NoneType' object has no attribute 'file'" report. It's occurring after the run.py portion -- I think it might be happening in the portion of tasks.py dealing with refreshing model data from the database, though I have not had the time to fully isolate it.

Cases where it's occurring:

• segmentation/transcription jobs training from scratch
• segmentation/transcription jobs refining on a model with "overwrite" checked

It does not occur on seg/trans jobs refining on a model but without "overwrite" checked. It also does not occur if an earlier step of the script returns "No best model found".

Update:

Found the bug in the logs, it is indeed happening in tasks.py around where I thought:

File "/srv/www/escriptorium/app/env/lib/python3.11/site-packages/htr2hpc/tasks
.py", line 383, in train
    if model.file is None or model.file == model.parent.file:

[cmroughan]

I would request adjusting tasks.py and run.py to pass forward information about whether the "overwrite" checkbox is ticked or not. If it is ticked, it would be good to pass that info into upload_best_model() and get_best_model(). The logic f"Must be better than original model {original_model.name} accuracy {best_accuracy:0.3f}" etc should only run if "overwrite" is checked. If it is not checked, the new model can be uploaded into eScriptorium even if the accuracy is lower than the original model we're refining on (because it won't replace it). If it is checked, then the new model should not be uploaded (which is the current behavior).

This logic is because someone could select some arbitrary model to use as a foundation for refining on and they will still want the resulting model even if its accuracy is lower than the foundation model's. For example, my taking an existing model for Latin and using it to jumpstart training a Greek model -- I don't care if the output Greek model is 96% to the original Latin's 98% because it's a wholly new model for me. But if "overwrite" is checked, then in that use case I am presumably training a Latin model on top of the old Latin model and then I do want to make sure I'm not deleting a older but better model by overwriting on top of it.