Feat/L2 Regularization

[L-M-Sherlock]

related PR:

• Expt/regularization of parameters fsrs-optimizer#157

[Expertium]

I'm currently trying out a different regularization method (well, it wasn't really made for regularization, but it kind of acts like a regularizer and improves generalization). If it works well, we can ditch L2 which barely improves the metrics. So don't merge this just yet. And you'd have to un-merge related PRs, but oh well.

[Expertium]

The results ended up being underwhelming.
I implemented AutoClip for gradients. Relevant repo: https://github.com/pseeth/autoclip. The code is very simple, I implemented it in other.py in just 16 lines.
The key idea is simple: keep track of observed gradient norms, calculate the p-th percentile (clip_value = np.percentile(grad_history, clip_percentile)) of them, then if next time the gradient norm exceeds that value, use torch.nn.utils.clip_grad_norm_(self.model.parameters(), clip_value). This is kind of like outlier filtering, but with gradients.
From their paper:

The authors recommend using p=10 aka 10th percentile, which is extremely aggressive, but perhaps suitable for large neural nets.

I tried different values of clip_percentile on FSRS-5, and sadly it barely improves the metrics.
So let's use L2 regularization.

[L-M-Sherlock]

Wow, L2 regularization has a very significant impact on the distribution of w[12], w[14], w[15], w[16]