1

I was watching a talk of Michael Jordan on Youtube. In the lecture, he mentioned that for a convex-Lipschitz function $f$, the gradient descent

$$x^{k+1} = x^k - \beta \nabla f(x^k)$$ for a constant step-size achieves the following rate,

$$f(x^k) - f(x^*) \leq O(1/k)$$

where $x^*$ is a local minimizer of $f$.

And Nesterov came up with an accelerated version, $$y^{k+1} = x^k - \beta \nabla f(x^k)$$ $$x^{k+1} = (1-\lambda^k)x^{k+1} + \lambda^k y^k$$

which acheives the following upper bound

$$f(x^k) - f(x^*) \leq O(1/k^2)$$

At this point, I am terribly confused as to why this upper bound would imply acceleration. Consider the original argument given by Nesterov,

enter image description here

This theorem doesn't say anything about upper bound, it is clearly a theorem on theoretical lower bound $\Omega(1/k^2)$ (unless that $\geq$ should have been a $\leq$). Furthermore, it is not a statement on the existence of a hypothetical algorithm, but that of a particular function. In fact, in the proof, Nesterov explicitly constructs the function that satisfies this condition. It doesn't say there exists an algorithm that achieves this lower bound. Also, there are strange conditions in the theorem such as $t \leq (n-1)/2$ that is never mentioned in the rate of convergence proofs of gradient descent, etc.

I guess I would be more convinced if the result instead was stated as follows: "Prop: there exists a first-order method such that for convex-Lipschitz functions, achieves $f(x^k) - f(x^*) \leq O(1/k^2)$". This seems to be a more logically coherent motivation as to why the rate for gradient descent is suboptimal. However, this is not what the theorem says and I don't see how the theorem implies this proposition.... (as in comment)

Therefore, in order to show that an algorithm achieves acceleration, we must show that it achieves a lower bound of $\Omega(1/k^2)$. So why is that in $>99\%$ of the literature on accelerated algorithms, an upper bound is provided instead of a lower bound?

Plus, an upper bound means the "worst-case" upper bound. Therefore, even if we do show the gradient descent achieves $O(1/k)$ rate, unless that $1/k$ rate is tight, it does not mean that it will experience that worst-case rate in practice. Since all these rates are worst-case rates of convergence, what is preventing the gradient descent from vastly surpassing Nesterovs algorithm in practice?

Can someone please provide a simple argument why upper bound implies acceleration, when the acceleration result is stated neither in terms of an upper bound nor a statement regarding the existence of a particular "accelerated" algorithm.

Olórin
  • 5,753

2 Answers2

2

If method 2 has a better upper bound than method 1, and both bounds are tight, then method 2's worst case is better than the worst case of method 1. The worst cases might be in totally different problems, so method 1 might sometimes be faster than method 2 anyway. In practice, rigorously showing that a method's bound is tight is usually more difficult than proving the bound itself, so usually you just try to not use too blunt of estimates in the course of deriving the upper bound and then don't bother to derive a lower bound.

What you showed in the image seems to be of a somewhat different character, it says "if $x_{s+1}$ is given by $x_1$ plus any linear combination of all the previous gradients, then there is an $f$ which decays at $\Omega(1/t^2)$ up until $t>(n-1)/2$". Thus it is expressing a fundamental limitation on all methods in this class.

Ian
  • 105,152
  • 1
    Hi, I don't understand the following arguments used in many references: "Since GD achieves $O(1/k)$ rate, and Nesterov shows that for first-order methods, there exists a function, such that for this function, first-order methods obtain $\Omega(1/k^2)$, therefore there exists a gap between the GD's rate and the optimal rate". – Olórin Mar 09 '18 at 22:25
  • For precise statement, see p 284 of https://arxiv.org/pdf/1405.4980.pdf – Olórin Mar 09 '18 at 22:27
  • 3
    @Olórin There are problems with $\Theta(1/k)$ convergence with gradient descent. There are no methods in this particular class with better than $O(1/k^2)$ convergence by the theorem given. Based on that theorem, you might hope that there is actually a method in this class with $O(1/k^2)$. And there is, but that's not implied by that theorem alone. – Ian Mar 09 '18 at 22:28
  • I guess I would be more convinced if the result instead was stated as follows: "Prop: there exists a first-order method such that for convex-Lipschitz functions, achieves $f(x^k) - f(x^*) \leq O(1/k^2)$". This seems to be a more logically coherent motivation as to why the rate for gradient descent is suboptimal. However, this is not what the theorem says and I don't see how the theorem implies this proposition.... – Olórin Mar 09 '18 at 22:43
  • 1
    @Olórin Indeed the theorem you quoted in your question does not state that there exists an algorithm which achieves $f(x^k) - f(x^*) \leq O(1/k^2)$. The theorem you quoted is making a different point entirely. – littleO Mar 09 '18 at 22:44
  • @littleO But that is what is claimed in this reference, for example http://awibisono.github.io/2016/06/20/accelerated-gradient-descent.html, in addition to the notes in the second comment. A quote from the blog "In any case, the result above means the fastest convergence rate we can hope to achieve using a first-order algorithm is O(1/ϵk2) when the objective function is (1/ϵ)-smooth." – Olórin Mar 09 '18 at 22:46
  • @Olórin No, that statement says exactly what it means: $1/k^2$ is the fastest you can hope for. It does not give a recipe for finding a $O(1/k^2)$ method. – Ian Mar 09 '18 at 22:49
  • 1
    There are two separate points to be made here: 1) No algorithm of this type can do better than $O(1/k^2)$. 2) There actually exists an algorithm of this type which achieves the $O(1/k^2)$ convergence rate. Both points are true, but the theorem from Nesterov that you quoted only addresses point 1. We have to look elsewhere to find Nesterov's proof of point 2. – littleO Mar 09 '18 at 22:49
  • @Ian Hi when you said "there are problems with $\Theta(1/k)$ converge with GD", does that mean there exists a result that says for GD, "$f(x^k) - f(x^*) \geq \Omega(1/k)$? I think this would greatly resolve my confusion – Olórin Mar 09 '18 at 23:08
  • @Olórin There's a "bad" $f$ with that property, yes. – Ian Mar 09 '18 at 23:19
2

Nesterov's theorem that you quoted is showing that no algorithm of this type can do better than $O(1/k^2)$. So a convergence rate of $O(1/k^2)$ is in some sense optimal for this type of algorithm. Nesterov separately provided an algorithm that actually achieves this optimal $O(1/k^2)$ convergence rate.

Of course, if the error decreases like $1/k^2$, that's a dramatic improvement over an algorithm for which the error goes down like $1/k$.

littleO
  • 54,768
  • Hi, can you be more explicit as to why showing $f(x^k) - f(x^) \geq \dfrac{3\beta}{32(k+1)^2|x_1 - x^|^2} = \Omega(1/k^2)$ would imply no algorithm of this type can "do better" than $O(1/k^2)$? I'm still not quite understanding why a lower bound $\Omega$ result implies existence of lower upper bound than $O(1/k)$. – Olórin Mar 09 '18 at 22:35
  • It seems the more logical implication to the quoted theorem is: "Since no first-order algorithm can break the $\Omega(1/k^2)$ bound, since GD satisfies a 'lower bound' of $\Omega(1/k)$, therefore there exists a gap between the GD and the optimal rate." In other words, everything stated in terms of lower bound, instead the optimal rate stated in terms of lower bound, and the GD result is stated in terms of upper bound. – Olórin Mar 09 '18 at 22:55
  • The lower bound of $\Omega(1/k^2)$ does not imply a gap with GD. If you only know this lower bound of $\Omega(1/k^2)$ and the fact that GD has a rate of $O(1/k)$, all you can say is that the optimal rate is somewhere between $1/k$ and $1/k^2$ (ignoring constants, and asymptotically). But the fact that Nesterov subsequentially obtained an other algorithm (i.e., an upper bound) with a rate of $O(1/k^2)$ shows the optimal rate is $\Theta(1/k^2)$. – Clement C. Mar 09 '18 at 23:06
  • @ClementC Great, this makes a lot more sense. I was confused because in this reference https://arxiv.org/pdf/1405.4980.pdf at the bottom of pg 284, it states that there exists a gap due to my previous reasoning. "So far our results leave a gap in the case of smooth optimization: gradient descent achieves an oracle complexity of $O(1/ε)$ (respectively $O(κ log(1/ε))$ in the strongly convex case) while we proved a lower bound of $Ω(1/\sqrt(\epsilon))$ (respectively $Ω(\sqrt(k) log(1/\epsilon))$). In this section we close these gaps" – Olórin Mar 09 '18 at 23:11
  • @Olórin If you are talking about the first paragraph of Section 3.6, I don't see why what is written corresponds to what you said. The "gap" mentioned there is not a separation between an algorithm and the other: it literally refers to a gap in our knowledge, as explained above: "the best upper bound we knew was $O(1/\varepsilon)$, the best lower bound we knew was $\Omega(1/\sqrt{\varepsilon})$, so there is a gap between the two where the 'right' answer lies." – Clement C. Mar 09 '18 at 23:16
  • @ClementC. Also refer to this, http://awibisono.github.io/2016/06/20/accelerated-gradient-descent.html "But recall that the convergence rate for gradient descent under the same assumption is O(1/ϵk). This suggests that gradient descent is suboptimal, i.e., there is a gap with the optimal rate of O(1/ϵk2).", "And thus the gap remained, until Nesterov came to the rescue in 1983 and proposed his optimal method that achieves the optimal rate." – Olórin Mar 09 '18 at 23:21
  • The keyword here is "suggest". It's not a proven separation (afaik). Gap is there used in a different way, that of "separation" (not gap of knowledge). – Clement C. Mar 09 '18 at 23:23