+
404
+
+
Page not found :(
+
The requested page could not be found.
+
+
+
+
+
+
+
+## Abstract
+> Deep neural networks (NNs) are known to lack uncertainty estimates and struggle to incorporate new data. We present a method that mitigates these issues by converting NNs from weight space to function space, via a dual parameterization. Importantly, the dual parameterization enables us to formulate a sparse representation that captures information from the entire data set. This offers a compact and principled way of capturing uncertainty and enables us to incorporate new data without retraining whilst retaining predictive performance. We provide proof-of-concept demonstrations with the proposed approach for quantifying uncertainty in supervised learning on UCI benchmark tasks.
+
+
+## TL;DR
+- `SFR` is a "posthoc" Bayesian deep learning method
+ - equip any trained NN with uncertainty estimates
+
+
+
+
+- `SFR` can be viewed as a function-space Laplace approximation for NNs
+- `SFR` has several benefits over [weight-space Laplace approximation for NNs](https://arxiv.org/abs/2106.14806):
+ - Its function-space representation is effective for regularization in continual learning (CL)
+ - It can incorporate new data without retraining the NN
+ - It has good uncertainty estimates
+ - We use them to guide exploration in model-based reinforcement learning (RL)
+
+
+
+| | **SFR** | **GP** | **Laplace BNN** |
+|-------------------------------|:-------:|:------:|:--------------------------:|
+| **Function-space** | ✅ | ✅ | ❌ (*weight space*) |
+| **Image inputs** | ✅ | ❌ | ✅ |
+| **Large data** | ✅ | ❌ | ✅ |
+| **Incorporate new data fast** | ✅/❌ | ✅ | ❌ (*requires retraining*) |
+
+## Useage
+See the [notebooks](https://github.com/AaltoML/sfr/tree/main/notebooks) for how to use our code for both regression and classification.
+
+### Minimal example
+Here's a short example:
+{% highlight python %}
+import src
+import torch
+
+torch.set_default_dtype(torch.float64)
+
+def func(x, noise=True):
+ return torch.sin(x * 5) / x + torch.cos(x * 10)
+
+# Toy data set
+X_train = torch.rand((100, 1)) * 2
+Y_train = func(X_train, noise=True)
+data = (X_train, Y_train)
+
+# Training config
+width = 64
+num_epochs = 1000
+batch_size = 16
+learning_rate = 1e-3
+delta = 0.00005 # prior precision
+data_loader = torch.utils.data.DataLoader(
+ torch.utils.data.TensorDataset(*data), batch_size=batch_size
+)
+
+# Create a neural network
+network = torch.nn.Sequential(
+ torch.nn.Linear(1, width),
+ torch.nn.Tanh(),
+ torch.nn.Linear(width, width),
+ torch.nn.Tanh(),
+ torch.nn.Linear(width, 1),
+)
+
+# Instantiate SFR (handles NN training/prediction as they're coupled via the prior/likelihood)
+sfr = src.SFR(
+ network=network,
+ prior=src.priors.Gaussian(params=network.parameters, delta=delta),
+ likelihood=src.likelihoods.Gaussian(sigma_noise=2),
+ output_dim=1,
+ num_inducing=32,
+ dual_batch_size=None, # this reduces the memory required for computing dual parameters
+ jitter=1e-4,
+)
+
+sfr.train()
+optimizer = torch.optim.Adam([{"params": sfr.parameters()}], lr=learning_rate)
+for epoch_idx in range(num_epochs):
+ for batch_idx, batch in enumerate(data_loader):
+ x, y = batch
+ loss = sfr.loss(x, y)
+ optimizer.zero_grad()
+ loss.backward()
+ optimizer.step()
+
+sfr.set_data(data) # This builds the dual parameters
+
+# Make predictions in function space
+X_test = torch.linspace(-0.7, 3.5, 300, dtype=torch.float64).reshape(-1, 1)
+f_mean, f_var = sfr.predict_f(X_test)
+
+# Make predictions in output space
+y_mean, y_var = sfr.predict(X_test)
+{% endhighlight %}
+
+
+## Citation
+Please consider citing our workshop paper.
+
+{% highlight bibtex %}
+@inproceedings{scannellSparse2023,
+ title = {Sparse Function-space Representation of Neural Networks},
+ maintitle = {ICML 2023 Workshop on Duality Principles for Modern Machine Learning},
+ author = {Aidan Scannell and Riccardo Mereu and Paul Chang and Ella Tami and Joni Pajarinen and Arno Solin},
+ year = {2023},
+ month = {7},
+}
+{% endhighlight %}
+
diff --git a/regression.png b/regression.png
new file mode 100644
index 0000000..b45895c
Binary files /dev/null and b/regression.png differ