Fast Adaptive Weight Noise

Deep learning methods have started to become practical for a wide range of tasks where very many labeled examples for supervised training are available, especially in the domains of sensory processing (e.g., vision or audio tasks). Yet, methods which work well on data sets with few training cases in the context of regression of continuous quantities remain scarce. Frequentist schemes such as weight decay or heuristics such as dropout have so far not been able to deliver significant improvements over methods not stemming from the connectionist paradigm, such as Gaussian processes or random forests; consequently, deep learning methods are generally not considered in fields where learning should be realised on small data sets.

We consider neural networks with parameters $\theta$ as weights and biases. If we treat the parameters $\theta$ not as points, but summarise our belief about them via a distribution $q(\theta)$, the data is explained by marginalising out that distribution, i.e.

In the case of $p(\theta)=q(\theta)$, i.e. $q$ is a prior, this is commonly referenced to as the marginal likelihood. We will consider supervised data only, that is $\mathcal{D}_{\text{train}}=\{({}^{i}\mathbf{x},{}^{i}\mathbf{z})\}_{i=1}^{N}$ where the functional relationship $\mathbf{x}\rightarrow\mathbf{z}$ is of interest. In Bayesian learning, a prior $p(\theta)$ is used for $q$ to obtain a posterior via Bayes’ rule

which can then be used to form a predictive distribution for unseen data points

In practice, Bayesian models are designed in a hierarchical way, where the prior is specified with the help of an additional hyperprior, i.e., $p(\theta)=\int_{\eta}p(\theta|\eta)p(\eta)$.

In all but the most trivial cases, Bayesian learning comes with several difficulties which require approximations. For neural networks, not only will the posterior $p(\theta|\mathcal{D}_{\text{train}})$ be highly multimodal due to symmetries in the weight space, but it will also be intractable to find the normalisation constant $p(\mathcal{D}_{\text{train}})=\int p(\mathcal{D}_{\text{train}}|\theta)p(\theta)d\theta$, i.e., the marginal likelihood.

Due to this intractability for all but the simplest cases, neural network practitioners have to resort to approximation schemes such as sampling (e.g., via Markov chain Monte Carlo Neal (1993)), variational inference (Hinton & Van Camp, 1993), combinations thereof (Graves, 2011), or Gaussian approximations (MacKay, 1992).

The contributions of this work will be as follows. We will extend the idea of fast dropout (Wang & Manning, 2013) to the marginalisation of distributions over the weights of a neural network in Section 2, and introduce an efficient way to respect the correlations between outputs units in Section 3.1. This will be used to perform variational Bayes for the special case of a Gaussian likelihood function in Section 3.3. In Section 3.4 we will then propose a novel method to find a distribution over weights, namely the minimisation of the negative log-likelihood of the data plus a regularisation term. The proposed methods will be verified experimentally in Section 4. We model the distributions over weights using diagonal Gaussian as well as Bernoulli distributions.

Adaptive weight noise is a practical method to perform VB (VB) in neural networks (Graves, 2011). The method is based on the approach of Hinton & Van Camp (1993), who utilise the MDL (MDL) principle (Rissanen, 1985; Grünwald, 2007) as an inductive bias.

As usual in the Bayesian setting, the parameters of the model under consideration are not found via point estimates, but represented as a distribution over the weight space. Here, each parameter $\theta_{i}$ will be represented by a Gaussian, i.e. $q(\theta_{i})=\mathcal{N}(\mu_{i},\sigma^{2}_{i})$.

If we are given a likelihood function and we consider $q$ as a variational approximation to the true posterior over the parameters having seen the data, the training criterion can be derived by means of VI:

where the outer sum is over the training samples. The “trick” that Hinton & Van Camp (1993) introduce is that the prior $p(\theta)$ is not set or further specified by a hyper-prior but instead learned as any other parameter in the model and thus essentially set by data. The contribution of Graves (2011) was then to approximate the expectation in Equation 7 by Monte Carlo sampling with Equation 8.

Here we use the previously introduced techniques to find a closed-form approximation to adaptive weight noise. Consider a single layer with $\theta=\{\mathbf{w}\}$, $y=f(\mathbf{x}^{T}\mathbf{w})$, where we have no dropout variables and the weights are Gaussian distributed with $\mathbf{w}\sim\mathcal{N}(\mu_{\mathbf{w}},\sigma^{2}_{\mathbf{w}})$, with covariance diagonal and organised into a vector. Again, we assume a Gaussian density for $a=\mathbf{x}^{T}\mathbf{w}$. Using the rules from Section 2, we find that

A perspective that we have not taken on so far is that this is a convolution of point predictions, each performed by a slightly different neural network with weights drawn from their respective distributions. Consider a neural network $f(\mathbf{x},\theta)$ with $\theta=\{\theta_{i}\}$, where each $\theta_{i}$ is a Gaussian distributed random variable with mean $\mu_{i}$ and variance $\sigma^{2}_{i}$. Let the network represent a distribution $p(\mathbf{z}|\theta)$ for the random variable $\mathbf{y}$, which is the network’s output. The output of the network with marginalised weights will be approximated as such:

where $q(\theta)$ depicts the joint over all weights and the moments of the Gaussian variable on the RHS are obtained as in Equations 9 and 10.

We evalutated FAWN-VI and FAWN-ROPD from Sections 3.3 and 3.4 respectively on a range of static regression tasks using FFN. We are interested in finding not only a point prediction but a whole predictive distribution. These tasks are typically not where neural networks excel and practicioners resort to GP instead, which is why we compare to those.

To this end we used a global univariate Gaussian for the prior and a Gaussian as a variational approximation for each of the parameters:

The KL-divergence is then given by¹¹ 1 Obtained with the help of the Q&A community “crossvalidated” at http://stats.stackexchange.com/questions/7440/kl-divergence-between-two-univariate-gaussians.:

Additionally, we chose a Gaussian likelihood where we assumed that

which resembles a Gaussian distributed measurement error with variance $\hat{\sigma}_{i}$ for output dimension $i$. We integrate the $\hat{\sigma}_{i}$ into the set of parameters and optimise it jointly with all other parameters.

All experiments were performed using a similar protocol to the one used in Hernández-Lobato & Adams (2015): we used single-layer networks with 50 hidden units using the rectifier transfer function. We report the negative log likelihood of the data with means and standard deviations coming from ten different random splits into 90% training and 10% testing data. The parameters of neural networks using FAWN were drawn from a zero-centred Gaussian with standard deviation 0.2.

Training was performed using Adam (Kingma & Ba, 2014) with a step rate of $\alpha=0.001$ until convergence of the training loss. No separate validation set was used. Gradients were estimated using 128 samples in a single mini batch.

The results for GP were obtained using a the sum of a linear and a squared exponential kernel using automatic relevance determination. Three random restarts were performed. We used GPy (GPy authors, 2012–2014) for the experiments.

The results are summarised in Table 1. The proposed methods place themselves well among alternative approaches, where FAWN-ROPD is better than FAWN-VI in all cases.

We have proposed a method to approximate the marginal likelihood of a distribution over neural network weights up to its mean and variance. This enabled us to derive a deterministic approximation of variational Bayes for Gaussian likelihoods and propose a novel, less subjective flavour of variational inference, FAWN-ROPD. The experimental results show that FAWN-ROPD obtains competitive performance over a wide range of regression tasks. These tasks include ones with very little samples (order of a few hundred) as well as many samples (several thousands) and range from domains such as robotics, predictive maintenance, computational biology and others.

The method requires further evaluation: we will experimentally investigate more common deep-learning architectures such as recurrent neural networks and deep multilayer perceptrons. Further, the suitability of FAWN for tasks where model uncertainty in the predictions is of interest, such as active learning or reinforcement learning, needs to be tested. On the theoretical side, the exact relationship of FAWN-ROPD to reference priors remains unclear and a theoretically founded motivation for FAWN-ROPD is an important next step.