A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Abstract

Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit. Dropout is a widely used tool for regularisation in deep models, but a long strand of empirical research has claimed that it cannot be applied between the recurrent connections of an RNN. The argument is that noise hinders the network's ability to model sequences and therefore dropout should be applied to the RNN's inputs and outputs alone. But without regularisation in recurrent layers, existing techniques overfit quickly. In this paper we make use of a recently developed theoretical framework casting dropout as approximate variational inference. Based on the framework we derive mathematically grounded tools to apply dropout within the recurrent layers of RNNs, eliminating model overfitting. We apply our new variational inference based dropout technique in LSTM and GRU networks, evaluating the technique empirically. We show that the new approach outperforms existing techniques on sentiment analysis and language modelling tasks, extending our arsenal of variational tools in deep learning.