Corrupted checkpoint

[Linardos]

My process is getting killed way too often for some reason (even though the GPU usage is barely above 60% of its total when I check it at nvidia-smi). It's just a baseline run of partitioning_1.csv with fixed_Batch_size, all_collaborators, simple weight average. This time it happened to get killed during storage and resulted in a corrupted pickle file when I ran it again:

Is there a way to store multiple intermediate checkpoints to avoid losing the entire experiment in case this happens again?

[sarthakpati]

You could potentially edit the checkpoint_utils module to add more locations for saving. Be warned that none of these customizations can be part of the submission.