Steepest Descent in elliptical error surface

Question

I am watching the Neural Network videos by Prof. Geoff Hinton. In there he talks about the problem with elliptical error surface.

In particular, he says, if the error surface is very elliptical, the direction of steepest descent is perpendicular to the direction towards the minimum.

My understanding of steepest descent was that it is perpendicular to the error surface and it points towards the direction of minimum.

Slides:

Link to timestamped videos:
https://youtu.be/tIovUOirJkE?list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9&t=224
https://youtu.be/4BZBog1Zx6c?list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9&t=66

Questions:
1) By definition, shouldn't steepest descent point in the direction of minimum?
2) Can you help me understand why is steepest descent perpendicular to the direction of minimum in case of elliptical error surfaces? But points towards minimum in case of circular error surface.

PS: responding to Amakelov's answer (modifying question, since I can't upload images in comment):

Let's assume this is my elongated elliptical error surface bowl. In this for steepest descent, I will have a large ycomponent, small x component and variable z component (depending how low in the error bowl I am).

So, if I look at the top down view, I will see a big movement in the y axis and a small movement in the x axis but the movement will still be in the direction of minima (and not perpendicular to it). What am I missing here?

Answer 1

1) No, and you can easily convince yourself by a mental experiment. Imagine you're literally standing on this elliptical ravine at the point on the figure. Remember that the ellipses denote locations with the same altitude. Then, when you're near the narrow end of the ravine (but not exactly at the narrow end!), the fastest way to descrease your altitude is not to walk towards the minimum (which would involve a lot of sideways motion that doesn't contribute to decreasing your altitude) but to walk directly down towards the long axis of the ellipse.

Formally, what we mean by 'steepest descent' in math is that, locally, the direction opposite to the gradient $\nabla f(x)$ provides the most decrease in altitude for a given horizontal distance $\delta$ you are willing to travel away from $x$, in the limit $\delta\to0$.

2) The steepest descent direction is not literally perpendicular to the direction of the minimum. Rather, in this extreme case it is almost perpendicular, and the reason is as above. Another way to look at it is to observe that the gradient is orthogonal to the level sets of your function, and since the level sets of the elliptical ravine near its narrow ends (but not exactly at the narrow ends!) are closely aligned with the direction toward the minimum, this leads to your gradient being almost orthogonal to the direction to the minimum.

Answer 2

Here is a figure that may help. I draw a long ellipse, which is intended to be one contour line, and a gradient vector that I drew by eye. The gradient vector is always perpendicular to the contour lines. It should be clear that the vector does not point toward the center of the ellipse. It is also not really close to perpendicular to the direction toward the center of the ellipse. I think the slides you cite exaggerate the perpendicularity. It does get worse if the ellipse is even longer than I have drawn. Added: To my eye, the gradient is closer to $45^\circ$ from the direction to the minimum than perpendicular. I think this is correct. If you minimize along the direction of the gradient you will cross the axis of the ellipse and stop at a point where your direction of travel is along the contour line. You stop and turn a right angle as I pointed out in my answer to your other question. Now you are again following the gradient and again about $45\circ$ from the direction to the minimum. You will zigzag across the axis. The good news is your total travel is only $\sqrt 2$ times as far as the direct route. The bad news is that each turn requires evaluating the gradient, which can be expensive. The point I picked is about the worst case. If you start closer to the end of the ellipse the gradient will be closer to the direction to the center. If you start farther from the end, the first step will be farther from the direction to the center but the next will be better.