Status for Pytorch 1.3

[sshkhr]

Hi

Thanks for the excellent repo. I was wondering what's the status of this repo for Pytorch v1.3. I want to use the Newton-CG method using line search to reimplement a meta-learning paper. Will your code support parameters from conv layers and such?

Thanks

[fmeirinhos]

Cheers, @sshkhr

I have tested the optimiser for PyTorch 1.3 and it seemed to work fine. It just doesn't support parameter groups (so optimising for different parameters with different optimiser-hyper-parameters would not work.)
It seems I have forgotten to raise an error for those cases but if you need that functionality let me know and I'll see if it can be implemented.

Best

[sshkhr]

Thanks for clarifying. I'm not particularly looking to use different optimiser-wise hyper parameters for the different network parameters so that should be okay for now. I'll try and see if I can get it working before this weekend and update you. Thanks again

[opooladz]

I am also interested in extending this for use on a CNN. @sshkhr have you had any luck?
I want to run it on a CNN with this architecture for example. https://colab.research.google.com/github/rpi-techfundamentals/fall2018-materials/blob/master/10-deep-learning/04-pytorch-mnist.ipynb#scrollTo=0mB6qGuYiwnX

[fmeirinhos]

The CNN architecture you posted should work just fine, @opooladz.
See the hf_test.py file for how to apply the optimiser to a torch.nn.Module

[opooladz]

Thank you for your reply @fmeirinhos . Yes, I actually followed that file and tried to extend it for CNNs to use with the Hv hessian method and fisher diag (I basically want an implementation of the Levenberg-Marquart update rule), but after a few iterations the loss simply goes to nan if I try with the inverse preconditioner and if i set M_inv = None, the loss just blows up. Is it possible to get another example file with a CNN? I can also send u my code as well.