Faster Gated Recurrent Units via Conditional Computation

Abstract

In this work, we apply the idea of conditional computation to the gated recurrent unit (GRU), a type of recurrent activation function. With slight modifications to the GRU, the number of floating point operations required to calculate the feed-forward pass through the network may be significantly reduced. This allows for more rapid computation, enabling a trade-off between model accuracy and model speed. Such a trade-off may be useful in a scenario where real-time performance is required, allowing for powerful recurrent models to be deployed on compute-limited devices.