MAINT: dynesty - reduce number of calls to add_live_points

[ColmTalbot]

A bug in dynesty (joshspeagle/dynesty#490) means that the chain can be corrupted if we write a checkpoint after catching an interrupt.

Currently, we call the unsafe method (add_live_points) more frequently than we need to. This PR makes it so that we only call this when making trace plots and after writing our main checkpoint file. This should reduce the number of calls drastically and practically eliminate our exposure to the issue.

Longer term, we may want to think about moving away from writing a new checkpoint file when exiting and rely on previously written checkpoints as pointed out in the linked issue, writing a checkpoint on exit will always be fundamentally unsafe, even if it can be made practically safe.

[GregoryAshton]

@ColmTalbot it looks like the CI is failing because dynamic doesn't have the add_live method - should this only apply to standard dynesty?

[mj-will]

Following the change to added_live, the changes look good to me.

@ColmTalbot have you had a chance to test this via bilby_pipe? I appreciate the tests should cover this but I think it would be good to be sure. What do you think @GregoryAshton ?

[ColmTalbot]

@ColmTalbot have you had a chance to test this via bilby_pipe?

I haven't, but it's a good idea. I'll try to set something up today.

[GregoryAshton]

All looks good to me too.

[mj-will]

@ColmTalbot have you had a chance to test this via bilby_pipe?

I haven't, but it's a good idea. I'll try to set something up today.

Just following up on this @ColmTalbot, did you start a test(s) with this change?

[ColmTalbot]

@ColmTalbot have you had a chance to test this via bilby_pipe?

I haven't, but it's a good idea. I'll try to set something up today.

Just following up on this @ColmTalbot, did you start a test(s) with this change?

Yeah, I ran one test that previously triggered the issue that completed fine, it isn't an exhaustive test, but I can't think of a better one.

[mj-will]

@ColmTalbot have you had a chance to test this via bilby_pipe?

I haven't, but it's a good idea. I'll try to set something up today.

Just following up on this @ColmTalbot, did you start a test(s) with this change?

Yeah, I ran one test that previously triggered the issue that completed fine, it isn't an exhaustive test, but I can't think of a better one.

Sounds good, I think we can get this in then. If there are issues, hopefully the review will pick up on them.