The mathematics behind Adam Optimizer

Question

I have read the paper "ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION".

The PDF link is below: https://arxiv.org/pdf/1412.6980.pdf

The section 2.1 gives the explanation and the intuition in ADAM, but the statements does not make sense for me.

At first, it claims that $| \Delta_t | \le \alpha \cdot (1-\beta_1) / \sqrt{1-\beta_2}$, if $(1-\beta_1)>\sqrt{1-\beta_2}$

In the suggested value of $\beta_1=0.9$, $\beta_2=0.999$, it is in the case.

Latter it claims that $| \Delta_t | \le \alpha$, in common scenarios, but I think that if $\beta_1$, $\beta_2$ is fixed, if would never be in this so called "common scenarios".

Did I miss something? I am really confused.

Dunedubby · Answer 1 · 2018-07-23T13:26:13.663

2

Reading the paper as it is, I'm fairly sure their upper bounds here are incorrect. For instance, consider the counter example $g_1 = 2$, $g_2 = 1$, $\beta_1 = \beta_2 = 0.5$, $\alpha = 1$, $\epsilon \rightarrow 0$.

Then $m_1 = 1$, $v_1 = 1$, $m_2 = 0.5 \cdot 1 + 0.5 \cdot 1 = 1$ and $v_2 = 0.5 \cdot 1 + 0.5 \cdot 1 = 1$

Therefore, $\widehat{m}_2 = 1/(1 - 1/4) = 4/3$ and $\widehat{v}_2 = 1/(1 - 1/4) = 4/3$ and $|\Delta_2| = 1\cdot (4/3)/\sqrt{4/3} = \sqrt{4/3}$, which is more than both suggested upper bounds.

It's possible that this is a typo in favor of $\alpha \cdot \sqrt{1 - \beta_2}/(1 - \beta_1)$, which seems to have a lot more going for it.

edited Jul 23 '18 at 13:26

answered Jul 23 '18 at 11:21

Dunedubby

33

1

If $g_1=2$, then $g_1^2=4$ and $v_1=0+0.5\cdot 4 = 2$ and $v_2=0.5 \cdot 2 + 0.5 \cdot 1^2=1.5$ – Alex Aug 03 '20 at 16:54
My previous comment was wrong - with the correct values, the bounds are correct. – Maverick Meerkat Mar 22 '23 at 17:43
Are there correction published? – High GPA Apr 23 '25 at 20:03

score 0 · Answer 2 · answered Jun 02 '17 at 21:21

0

My understanding is that those two expressions are two possible upper bounds (applicable depending the situation).

According to the paper (https://arxiv.org/pdf/1412.6980.pdf)

The first case only happens in the most severe case of sparsity: when a gradient has been zero at all timesteps except at the current timestep.

In that situation the upper bound will be:

$| \Delta_t | \le \alpha \cdot (1-\beta_1) / \sqrt{1-\beta_2} $

Otherwise the upper bound will be:

$|\Delta_t | \le \alpha$

answered Jun 02 '17 at 21:21

Juan Leni

150

There's a question arose, why does it has these two upper bounds? – user3148602 Jun 11 '17 at 03:45
1

@user3148602 I believe these bounds are wrong, as I tried very hard to derive it but did not succeed, how on earth the authors are allowed to skip such non-trivial bounds??? – Fei Cao Oct 09 '21 at 22:21
@MathandYuGiOhlover did you get any response? Do you have an update? Empirically these bounds do seem to be right. – Maverick Meerkat Mar 22 '23 at 17:42

The mathematics behind Adam Optimizer

2 Answers2