Matrix derivative $(Ax-b)^T(Ax-b)$

Question

I am trying to find the minimum of $(Ax-b)^T(Ax-b)$ but I am not sure whether I am taking the derivative of this expression properly.

What I did is the following: \begin{align*} \frac{\delta}{\delta x_i}\left(\sum_i \sum_j (A_{ij}x_i-b_i)(A_{ij}x_j-b_j)\right)&= \sum_j A_{ij}(A_{ij}x_j-b_j) + \sum_i A_{ij}(A_{ij}x_j-b_i) \end{align*}

but I'm not quite sure if this is correct and what the derivate would then be. Any help is appreciated.

Answer 1

Perhaps some help to compute the partial derivatives would be appreciated. It is always best to be explicit when one is a bit confused with heavy notation. Write $A = (a_{ij})$, $x = (x_1,\dots,x_n)^{\top}$ and $b = (b_1,\dots, b_m)^{\top}$, assuming $A$ is an $m \times n$ matrix. Then the $i^{\text{th}}$ component of $Ax-b$ is $$ (Ax-b)_i = \left( \sum_{j=1}^n a_{ij} x_j \right) - b_i $$ so that $$ (Ax-b)^{\top} (Ax-b) = \sum_{i=1}^m (Ax-b)_i^2 = \sum_{i=1}^m \left( \left( \sum_{j=1}^n a_{ij} x_j \right) - b_i \right)^2. $$ Suppose you want to compute the derivative with respect to $x_k$, $1 \le k \le n$ (I choose $k$ because choosing $i$ or $j$ would be confusing with the preceding subscripts used). Then $$ \frac{\partial}{\partial x_k} (Ax-b)^{\top} (Ax-b) = \sum_{i=1}^m \frac{\partial}{\partial x_k} \left( \left( \sum_{j=1}^n a_{ij} x_j \right) - b_i \right)^2 = \sum_{i=1}^m 2 \left( \left( \sum_{j=1}^n a_{ij}x_j \right) - b_i \right) (a_{ik}). $$ In particular, we can let $A_k = (a_{1k},a_{2k},\dots,a_{mk})$ so that $$ \frac{\partial}{\partial x_k} (Ax-b)^{\top} (Ax-b) = 2 \langle A_k, Ax-b \rangle $$ where $\langle - , - \rangle$ denotes inner product. You can use convexity arguments to show that any critical point is a minimizer in this case ; although you can see that the minimizer will not always be unique, even when $m=n$ ; it suffices for $A$ to be singular for this to happen.

Hope that helps,

Answer 2

The derivation becomes a lot simpler if we take the derivative with respect to the entire $x$ in one go:

$$\frac{\delta}{\delta x}(Ax-b)^T(Ax-b) \ \ = \ \ 2(Ax-b)^T\frac{\delta}{\delta x}(Ax-b) \ \ = \ \ 2(Ax-b)^TA$$

This follows from the product rule:

$$\frac{\delta}{\delta x}uv \ \ = \ \ \frac{\delta u}{\delta x}v+u\frac{\delta v}{\delta x}$$

And that we can swap the order of the dot product:

$$\frac{\delta}{\delta x}u^Tu \ \ = \ \ \frac{\delta u^T}{\delta x}u+u^T\frac{\delta u}{\delta x} \ \ = \ \ (\frac{\delta u}{\delta x})^Tu+u^T\frac{\delta u}{\delta x} \ \ = \ \ 2u^T\frac{\delta u}{x}$$

I just learned matrix calculus from the Matrix Cookbook yesterday, and I love it :)

Answer 3

I assume you are trying to minimize $\langle A x -b, A x - b\rangle.$ This is always nonnegative, so if $A$ is nonsingular, then the minimum is $0.$ Otherwise, the $i$-th component of the gradient is

$$\langle A_i, Ax +b\rangle + \langle A x + b, A_i\rangle$$

Where $A_i$ is the $i$-th column of $A.$ What does this tell you?

Answer 4

Using the identities: $\frac{\partial x^{T} Bx}{\partial x} = (B+B^{T} )x $

$\frac{\partial x ^{T} a}{\partial x} = \frac{\partial x a ^{T}}{\partial x} = a$

$\frac{\partial a ^{T} x b}{\partial x} = a b ^{T}$

$\frac{\partial a ^{T} x ^{T} b}{\partial x} = ba ^{T}$

We have

$f(x) = (Ax-b)^{T}(Ax-b) $

$= x^{T} A^{T} Ax - x^{T} A^{T} b -b ^{T} Ax -b^{T} b$

Using above identities, we have

${f}'(x) = (A^{T} A+(A^{T} A)^{T})x - A^{T} b -(b^{T} A)^{T} $

$ = 2A^{T} Ax - 2A^{T} b$

Answer 5

I will include an answer using Einstein notation, because I work with high order tensors and this notation makes this kind of derivative very easy to obtain.

$$f(\mathbf{x}) = \left[A_{ij}\,x_j-b_{i}\right] \left[A_{ik}\,x_k-b_i\right] = A_{ij}\,A_{ik}\,x_j\,x_k - b_i\,A_{ij}\,x_j - b_i\,A_{ik}\,x_k + b_i\,b_i$$

$$f(\mathbf{x}) = A_{ij}\,A_{ik}\,x_j\,x_k - 2\,b_i\,A_{ij}\,x_j + b_i\,b_i$$

---

We have that $\frac{\partial x_i}{\partial x_j} = \delta_{ij}$, thus:

$$\frac{\partial f}{\partial x_p} = A_{ij}\,A_{ik}\,\delta_{jp}\,x_k + A_{ij}\,A_{ik}\,x_j\,\delta_{kp} - 2\,b_i\,A_{ij}\,\delta_{jp} = A_{ip}\,A_{ik}\,x_k + A_{ij}\,A_{ip}\,x_j - 2\,b_i\,A_{ip}$$

We can then transpose some terms to get:

$$\frac{\partial f}{\partial x_p} = A^T_{pi}\,A_{ik}\,x_k + A^T_{pi}\,A_{ij}\,x_j - 2\,A^T_{pi}\,b_i = 2\left[A^T_{pi}\,A_{ij}\,x_j - A^T_{pi}\,b_i\right]$$

In matrix notation we get:

$$\boxed{\frac{\partial f}{\partial \mathbf{x}} = 2\,\mathbf{A}^T\left[\mathbf{A}\,\mathbf{x}-\mathbf{b}\right]}$$

---

If $\mathbf{A}$ is invertible, we get:

$$\frac{\partial f}{\partial \mathbf{x}} = \mathbf{0} \Leftrightarrow \mathbf{x} = \mathbf{A}^{-1}\,\mathbf{b}$$

If $\mathbf{A}$ is rectangular and $\mathbf{H} = \mathbf{A}^T\mathbf{A}$ is invertible, considering $\mathbf{q} = \mathbf{A}^T\,\mathbf{b}$, we get:

$$\frac{\partial f}{\partial \mathbf{x}} = \mathbf{0} \Leftrightarrow \mathbf{x} = \mathbf{H}^{-1}\,\mathbf{q}$$

For singular matrices, there should be infinite solutions for the minimization problem.