12

My question comes from the paper Adam: A Method for Stochastic Optimization published on ICLR, which has been cited over $80,000$ times up to now. Specifically, in page 13, I think there is a 'bug' in the proof of Lemma 10.4 (see the following snapshot of my annotated version of that page):

enter image description here

I believe that the inequality stated after "Similarly, we can upper bound the rest of the terms in the summation" is wrong, I think the authors missed an important term in the summation that can not be handled easily. I am not sure whether such bug can be remedied to be honest...

Sometimes I feel super mad at some mathematical derivations or statements in these "machine-learning" paper, as I think some steps are really not that trivial and given the fact that the paper is not even that long, so why don't the authors present the full details to HELP the readers? Thank you very much for your kind help!

Edit: There is another question on The mathematics behind Adam Optimizer posted on MSE $4$ years ago, and I am also thinking the bound on $\Delta_t$ stated in their paper is wrong (see the bottom of page 2 of their original paper), as I tried very hard to derive it but did not succeed, how on earth the authors are allowed to skip the details of such non-trivial bounds?

Edit: It seems that their Lemma 10.4 is absolutely wrong, as pointed out for instance in Conjecture 4.2 of this paper. I feel ashamed of this area, as it would be a "miracle" for such paper (with obvious flaw) to get published on any mathematical journal!

Fei Cao
  • 3,018
  • bug? It is a program? Lol –  Oct 09 '21 at 09:34
  • 2
    @Masacroso lol I was influenced by my Ph.D advisor – Fei Cao Oct 09 '21 at 17:12
  • Just came across this post many years later, thanks for sharing your findings! It's been several years since the original post, have there been any consequences since? Also in a practical sense, does this/did this change how efficient ADAM is compared to other gradient-descent methods? From my limited understanding, it seems that although there are mathematical issues with the paper, that somehow ADAM still kinda works well a lot of the times? – sheesymcdeezy Aug 23 '24 at 08:23
  • I agree with almost all parts of your argument, although I am confused with your last paragraph. In my humble opinion, mistakes are very common in applied math. I'd say two out of ten top papers contain some math errors, minor or major. However, the nature of a conference paper in an applied fields implies that the paper is more applied focused (rather than math focused), and a conference is mean to let applied paper to be known before a lengthy journal-reviewing process. It serves as a "fast-track", and it is naturally less rigorous than a top math journal. – High GPA Apr 10 '25 at 01:12
  • A top math journal takes years to publish a paper. With the fast-advancing technology, the time-frame is too lengthy. By focusing on conferences more, we are indeed sacrificing some level of rigidity. There is nothing we need be ashamed about. It is science – High GPA Apr 10 '25 at 01:14
  • 1
    @HighGPA Hey thanks for the comments. It has been a while after this post, and I think I got your points now – Fei Cao Apr 15 '25 at 19:42
  • @FeiCao Thanks for your reply. Are there any correction published? – High GPA Apr 23 '25 at 20:02

1 Answers1

6

I believe there is a typo in the paper and the $t$ you circled is supposed to be a $j$. Not that this fixes the problem. My attempted derivation agrees with you:

From $$\frac{\widehat m_{t,i}^2}{\sqrt{t\widehat v}_{t,i}} \le \frac{t}{\sqrt{t(1-\beta_2)}} \sum_{k=1}^t \gamma^{t-k} \lVert g_{k,i}\rVert_2,$$ we sum over $1 \le t \le T$, obtaining $$\begin{align*}\sum_{t=1}^T\frac{\widehat m_{t,i}^2}{\sqrt{t\widehat v}_{t,i}} &\le \sum_{t=1}^T \sum_{k=1}^t\frac{t}{\sqrt{t(1-\beta_2)}} \gamma^{t-k} \lVert g_{k,i}\rVert_2 \\ &= \sum_{k=1}^T\sum_{t=k}^T \frac{t}{\sqrt{t(1-\beta_2)}} \gamma^{t-k} \lVert g_{k,i}\rVert_2 \\ &= \sum_{k=1}^T \lVert g_{k,i}\rVert_2\sum_{t=k}^T \frac t{\sqrt{t(1-\beta_2)}} \gamma^{t-k} \\ &= \sum_{k=1}^T \lVert g_{k,i}\rVert_2\sum_{j=0}^{T-k} \frac {k+j}{\sqrt{(k+j)(1-\beta_2)}} \gamma^j \\ &= \sum_{t=1}^T \lVert g_{t,i}\rVert_2\sum_{j=0}^{T-t} \frac {t+j}{\sqrt{(t+j)(1-\beta_2)}} \gamma^j \\ &= \sum_{t=1}^T \frac{\lVert g_{t,i}\rVert_2}{\sqrt{t(1-\beta_2)}}\sum_{j=0}^{T-t} \frac {(t+j)\sqrt{t}}{\sqrt{t+j}} \gamma^j \\ &\le \sum_{t=1}^T \frac{\lVert g_{t,i}\rVert_2}{\sqrt{t(1-\beta_2)}}\sum_{j=0}^{T-t} (t+j) \gamma^j. \end{align*}$$ To obtain the claimed inequality, we appear to need $j$ to be an upper bound for $\dfrac {(t+j)\sqrt{t}}{\sqrt{t+j}} = \sqrt{t(t+j)}$, but it isn't unless $j$ is large enough relative to $t$.

Eric M. Schmidt
  • 3,926