What is AdaShift?
AdaShift is an adaptive stochastic optimizer that helps to solve a problem with the Adam optimizer. It is designed to help models converge and produce more accurate output.
Why was AdaShift created?
Adam is a commonly used optimizer in deep learning models. However, it has a problem with correlation between the gradient and second-moment term. This means that large gradients can end up with small step sizes, while small gradients can end up with large step sizes. This issue can lead to non-convergence of Adam.
To solve this problem, AdaShift was created. It uses temporal shifting to decorrelate the gradient and second-moment term, resulting in more appropriate step sizes for different gradient sizes.
How does AdaShift work?
AdaShift works by using the temporally shifted gradient to calculate $v\_{t}$. First, AdaShift calculates the gradient ($g\_{t}$) for each time step ($t$). Then it calculates the mean of past $n$ gradients ($m\_{t}$).
Next, AdaShift uses the temporally shifted gradient $g\_{t-n}$ to calculate $v\_{t}$, which will be used to adjust the step size. The $v\_{t}$ is calculated with the following formula:
$$ v\_{t}\left[i\right] = \beta\_{2}v\_{t-1}\left[i\right] + \left(1-\beta\_{2}\right)\phi\left(g^{2}\_{t-n}\left[i\right]\right) $$
Finally, AdaShift uses $v\_{t}$ and $m\_{t}$ to update the model weights ($\theta\_{t}$) with the following formula:
$$ \theta\_{t}\left[i\right] = \theta\_{t-1}\left[i\right] - \alpha\_{t}/\sqrt{v\_{t}\left[i\right]}\cdot{m\_{t}\left[i\right]} $$
Overall, AdaShift is designed to help models converge more quickly and produce more accurate output by adjusting the step size based on the size of the gradient.
AdaShift is an adaptive stochastic optimizer that addresses a problem with the Adam optimizer. It helps models converge more quickly and produce more accurate output by adjusting the step size based on the size of the gradient. AdaShift is a valuable tool for deep learning practitioners who want to improve the performance of their models.
