Add the ability to automatically select best batch size

[sarthakpati]

Is your feature request related to a problem? Please describe.
Currently, GaNDLF requires the user to select the optimal batch_size for the hardware they are training on, which is not very efficient in the case of hyper-parameter tuning.

Describe the solution you'd like
Adding the ability to choose the best batch_size automatically based on every other hyper-parameter would make the process much smoother.

Describe alternatives you've considered
N.A.

Additional context

• https://stackoverflow.com/a/66429367
• https://discuss.pytorch.org/t/how-to-determine-the-largest-batch-size-of-a-given-model-saturating-the-gpu/146075/2
• https://github.com/BlackHC/toma

[github-actions]

Stale issue message

[github-actions]

Stale issue message

[benmalef]

@sarthakpati you can assign this to me

[sarthakpati]

@sarthakpati you can assign this to me

Thanks! 👍🏽

[github-actions]

Stale issue message

[github-actions]

Stale issue message

[github-actions]

Stale issue message

[sarthakpati]

Using Lightning, this option [ref] should work. From the API docs [ref], it should return an int type.

[szmazurek]

Yeah, I used this option myself like last year, it can be useful but from what I have seen it often goes too far with the batch size, I remember that even despite tuning some OOMs happened to me. the tuner is configurable (at least via lightning), so maybe we just should keep in mind that we should stop a little earlier than the tuner would do (i.e. if the tuner chose some value decrease it by like 5% and the use during training).

[sarthakpati]

Sounds perfect, thank you!

[github-actions]

Stale issue message