How to rigorously robustness to outliers and prove that Least Absolute Deviation is more robust than Ordinary Least Square?

Question

I am an engineer who really love math, and recently watched an educational video "Fitting a line WITHOUT using least squares?" where at timestamp 7:10, the presenter demonstrates that Least Absolute Deviation (LAD) regression is more robust to outliers than Ordinary Least Squares (OLS) regression.

While the visual demonstration is compelling, I'm seeking a more rigorous mathematical understanding of this concept.

Background

From my understanding:

• OLS minimizes the sum of squared residuals.
• LAD minimizes the sum of absolute residuals.

The video shows empirically that when outliers are introduced, the LAD regression line remains relatively stable while the OLS line shifts significantly.

Questions

1. How is "robustness" formally defined in statistical theory? Is there a standard mathematical framework for measuring this property (For example: measure of robustness) ?

2. What is the rigorous proof that LAD is more robust to outliers than OLS? I'm particularly interested in understanding:

• The theoretical foundation for this claim and how to construct the proof.
• Whether this holds under all conditions or if there are specific assumptions required.

Any references to textbooks or papers that cover this would be greatly appreciated.

If the rigorousness is too complicated for an answer then something that engineer might understand is all so okay for an answer.

Answer 1

Let's try to understand what is really going on here. Forget about 2D and lines. Let's just consider the problem of choosing a point $z$ on the real line that would minimize the sum of deviations from some given finite set of points $z_1,\dots,z_n\in\mathbb R$ with or without squaring. If you do the corresponding simple algebra, you'll see that the least square method gives you the mean while the least absolute deviation gives you the median. And indeed, the median is less sensitive to changing the data; in fact, so much less that it just completely ignores all the data except one or two in the very middle of the dataset, so you can drive all the rest to $\pm\infty$ in any way you want without crossing the middle point or interval and it won't be felt at all. Is such super-stability good or bad? Well, that depends entirely on what you want from your choice of $z$. Also, the median is often not unique, so it is going to be not "the best point" but either "a best point", or "the set of best points" for LAD. Again, whether it is good or bad depends on what you are going to do with it and what you want to use it for.

The line story is qualitatively exactly the same and his "Wow, it barely moves at all!" should actually be "Hmmm... it doesn't move at all..." and if he bothered to move other $y$-values, he would immediately discover that as long as he doesn't cross the blue line in that motion, the line won't change no matter what he does. The relevant algebra becomes a bit nastier, but all morals of the above story are preserved verbatim.

At last, you don't really need any linear programming to find the (or, rather, an) LAD line, just as you don't really need it to find the median, though, of course, his way to reduce the task to a linear programming problem is absolutely correct and will produce a valid answer; it is just an overkill here.