The Mogrifier LSTM is an extension of the LSTM (Long Short-Term Memory) algorithm used in machine learning. The Mogrifier LSTM adds a gating mechanism to the input of the LSTM, where the gating is conditioned on the output of the previous step. Then, the gated input is used to gate the output of the previous step. After a few rounds of this mutual gating, the last updated inputs are fed to the LSTM. This process is called "modulating," and it allows the Mogrifier LSTM to learn patterns in the data more efficiently.
How does the Mogrifier LSTM work?
The Mogrifier LSTM uses a gating mechanism to determine which values to keep and which to discard at each step. The key difference between the Mogrifier LSTM and the standard LSTM is that in the Mogrifier LSTM, the inputs and outputs modulate one another in an alternating fashion before the usual LSTM computation takes place.
The Mogrifier LSTM has two inputs, $\mathbf{x}$, and $\mathbf{h}\_{prev}$. These inputs modulate one another in an alternating fashion before the usual LSTM computation takes place. The modulated inputs are defined as the highest indexed $\mathbf{x}^{i}$ and $\mathbf{h}^{i-1}\_{prev}$ from the interleaved sequences, where $i$ is an odd or even number.
For odd index $i \in \left[1\dots r\right]$, the modulated input $\mathbf{x}^{i}$ is computed as $ \mathbf{x}^{i} = 2\sigma\left(\mathbf{Q}^{i}\mathbf{h}^{iā1}\_{prev}\right) \odot x^{i-2}$. Here, $\mathbf{Q}^{i} \in \mathbb{R}^{m\times{n}}$, where $m$ is the number of rows in $\mathbf{Q}^{i}$, $n$ is the number of columns in $\mathbf{Q}^{i}$, and $k < \min\left(m, n\right)$ is the rank. The constant $2$ ensures that randomly initialized $\mathbf{Q}^{i}$ matrices result in transformations close to identity. To reduce the number of additional model parameters, we typically factorize the $\mathbf{Q}^{i}$ matrices as products of low-rank matrices: $\mathbf{Q}^{i}$ = $\mathbf{Q}^{i}\_{left}\mathbf{Q}^{i}\_{right}$ with $\mathbf{Q}^{i}\_{left} \in \mathbb{R}^{m\times{k}}$, $\mathbf{Q}^{i}\_{right} \in \mathbb{R}^{k\times{n}}$.
For even index $i \in \left[1\dots r\right]$, the modulated input $\mathbf{h}^{i-1}\_{prev}$ is computed as $ \mathbf{h}^{i-1}\_{prev} = 2\sigma\left(\mathbf{R}^{i}\mathbf{x}^{i-1}\right) \odot \mathbf{h}^{i-2}\_{prev}$. Here, $\mathbf{R}^{i} \in \mathbb{R}^{n\times{m}}$, where $m$ is the number of rows in $\mathbf{R}^{i}$, $n$ is the number of columns in $\mathbf{R}^{i}$ and $k < \min\left(m, n\right)$ is the rank.
The number of "rounds," $r \in \mathbb{N}$, is a hyperparameter. When $r = 0$, the Mogrifier LSTM recovers the standard LSTM.
What are the benefits of the Mogrifier LSTM?
The Mogrifier LSTM has several benefits over the standard LSTM. First, it can learn patterns more efficiently because it modulates the input and output of the LSTM. This can lead to better performance on tasks where long-term dependencies are important.
Second, the Mogrifier LSTM reduces the number of additional model parameters compared to other LSTM extensions. By factorizing the $\mathbf{Q}^{i}$ and $\mathbf{R}^{i}$ matrices as products of low-rank matrices, the Mogrifier LSTM reduces the number of additional parameters needed to describe the model. This can make the Mogrifier LSTM faster and more efficient than other LSTM extensions.
Third, the Mogrifier LSTM can be used in a variety of applications. For example, it can be used in natural language processing, speech recognition, and image processing. The Mogrifier LSTM has been shown to achieve state-of-the-art performance on a variety of tasks in these fields.
The Mogrifier LSTM is an extension of the LSTM algorithm that uses a gating mechanism to modulate the input and output of the LSTM. This can lead to better performance on tasks where long-term dependencies are important. The Mogrifier LSTM is also more efficient and faster than other LSTM extensions because it reduces the number of additional model parameters needed. Overall, the Mogrifier LSTM is a powerful tool that can be used in a variety of applications.
