Least Absolute Deviation (LAD) with Non Negative Constraint

Question

I would like to solve the following:

$$ \begin{align} \text{minimize} & \quad & \left\| A x - b \right\|_{1} \\ \text{subject to} & \quad & x \succeq 0 \end{align} $$

What kind of toolkit should we use to solve this problem?
I know we can turn this into a linear programming problem. Can that be done by just adding another constraint?

Answer 1

As suggested by @littleO, The use of Linear Operator and the ${L}_{1}$ Norm makes the problem a good fit for the Primal Dual Hybrid Gradient (PDHG) Method (Chambolle Pock Algorithm).

The objective can be written as:

$$ \arg \min_{\boldsymbol{x}} {\left\| \boldsymbol{A} \boldsymbol{x} - \boldsymbol{b} \right\|}_{1} + \delta_{\mathcal{C}} \left( \boldsymbol{x} \right) = f \left( \boldsymbol{A} \boldsymbol{x} \right) + g \left( \boldsymbol{x} \right) $$

Where $\delta_{\mathcal{C}} \left( \boldsymbol{x} \right) = \begin{cases} 0 & \text{if} \; \boldsymbol{x} \geq \boldsymbol{0} \\ \infty & \text{else} \end{cases}$.

The method requires the Proximal of the Convex Conjugate of $f \left( \cdot \right)$ (For some connection, see The Proximal Operator of the ${L}_{\infty}$ (Infinity Norm) and Legendre Transform (Convex Conjugate) of a Norm):

$$ \operatorname{prox}_{\sigma {f}^{\ast}} \left( \boldsymbol{y} \right) = \operatorname{proj}_{ {\left\| \cdot \right\|}_{\infty} \leq 1 } \left( \boldsymbol{y} - \sigma \boldsymbol{b} \right) = \operatorname{clamp}_{\left[ -1, 1 \right]} \left( \boldsymbol{y} - \sigma \boldsymbol{b} \right) $$

The Proximal Mapping of $g \left( \cdot \right)$ is the projection of the non negative values:

$$ \operatorname{prox}_{\tau g} \left( \boldsymbol{y} \right) = \max \left\{ \boldsymbol{y}, \boldsymbol{0} \right\} $$

The whole algorithm becomes:

• $\boldsymbol{y}^{\left( k + 1 \right)} = \operatorname{prox}_{\sigma {f}^{\ast}} \left( \boldsymbol{y}^{\left( k \right)} + \sigma \left( \boldsymbol{A} \bar{\boldsymbol{x}}^{\left( k \right)} - \boldsymbol{b} \right) \right) = \operatorname{clamp}_{\left[ -1, 1 \right]} \left( \boldsymbol{y}^{\left( k \right)} + \sigma \left( \boldsymbol{A} \bar{\boldsymbol{x}}^{\left( k \right)} - \boldsymbol{b} \right) \right)$.
• $\boldsymbol{x}^{\left( k + 1 \right)} = \operatorname{prox}_{\tau {g}} \left( \boldsymbol{x}^{\left( k \right)} - \tau \boldsymbol{A}^{\top} \boldsymbol{y}^{\left( k + 1 \right)} \right) = \max \left\{ \boldsymbol{x}^{\left( k \right)} - \tau \boldsymbol{A}^{\top} \boldsymbol{y}^{\left( k + 1 \right)}, \boldsymbol{0}\right\}$.
• $\bar{\boldsymbol{x}}^{\left( k \right)} = \boldsymbol{x}^{\left( k + 1 \right)} + \theta \left( \boldsymbol{x}^{\left( k + 1 \right)} - \boldsymbol{x}^{\left( k \right)} \right)$.

Where $\sigma \tau {\left\| \boldsymbol{A} \right\|}_{2}^{2} \leq 1$ and $\theta \in \left[ 0, 1 \right]$.

It is a much faster method than the Projected Sub Gradient method, though a little sensitive to the hyper parameters values.

The performance are verified with a DCP solver which generates the reference solution.

---

The code is available on my StackExchange GitHub Repository (Look at the Mathematics\Q1385984 folder).

Answer 2

The problem is given by:

$$ \begin{align} \text{minimize} & \quad & \left\| A x - b \right\|_{1} \\ \text{subject to} & \quad & x \succeq 0 \end{align} $$

The easiest (Yet one of the slowest) methods to solve this would be using the Projected Sub Gradient Method.

The Gradient of $ \left\| A x - b \right\|_{1} $ is given by $ {A}^{T} \operatorname{sgn} \left( A x - b \right) $.
The projection onto the non negative orthant is given by $ {x}_{+} $.

Here is the code:

vX = zeros([numCols, 1]);

for ii = 1:numIterations
    vX = vX - ((stepSize / sqrt(ii)) * mA.' * sign(mA * vX - vB));
    vX = max(vX, 0);
end

objVal = norm(mA * vX - vB, 1);

disp([' ']);
disp(['Projected Gradient Solution Summary']);
disp(['The Optimal Value Is Given By - ', num2str(objVal)]);
disp(['The Optimal Argument Is Given By - [ ', num2str(vX.'), ' ]']);
disp([' ']);

The full code (Including verification against CVX) can be found in my Mathematics Exchange Q1385984 GitHub Repository.

Remarks:

• You certainly can transfer this problem into another form (Like Linear Programming Problem) and solve it more efficiently. Yet this will require a solver.
• Other method based on $ \operatorname{Prox} $ operator or Primal - Dual Method are also applicable and efficient in this case.