[Question] Safetensors seem to block the main thread -- but torch.save does not?

[vedantroy]

I have the following code in my training loop:

                if rank == 0:
                    t = Thread(
                        target=save_file,
                        args=(model_sd, f"{cfg.model_dir}/model_{step + 1}.safetensors"),
                        daemon=True
                    )
                    t.start()

Which saves the checkpoint to disk using safetensors. However, I notice that this blocks the training loop, even though the thread should be running in the background.

When I switch the code to use torch.save, there's no issue. What should I do?

[Narsil]

This is hard to say without being able to reproduce anything.

This sounds like a GIL issue although it's hard to prove or assert. Rust needs to own the GIL while it does something with some Python values.
This should be easy to assert using multiprocessing instead of threading though.

If it's indeed GIL issues, I don't have any great idea on how to avoid them while using threads.