When does the inverse of a covariance matrix exist?

Question

We know that a square matrix is a covariance matrix of some random vector if and only if it is symmetric and positive semi-definite (see Covariance matrix). We also know that every symmetric positive definite matrix is invertible (see Positive definite). It seems that the inverse of a covariance matrix sometimes does not exist.

Does the inverse of a covariance matrix exist if and only if the covariance matrix is positive definite? How can I intuitively understand the situation when the inverse of a covariance matrix does not exist (does it mean that some of the random variables of the random vector are equal to a constant almost surely)?

Any help will be much appreciated!

Answer 1

If the covariance matrix is not positive definite, we have some $a \in \mathbf R^n \setminus \{0\}$ with $\def\C{\mathop{\rm Cov}}\C(X)a = 0$. Hence \begin{align*} 0 &= a^t \C(X)a\\ &= \sum_{ij} a_j \C(X_i, X_j) a_i\\ &= \mathop{\rm Var}\left(\sum_i a_i X_i\right) \end{align*} So there is some linear combination of the $X_i$ which has zero variance and hence is constant, say equal to $\alpha$, almost surely. Letting $H := \{x \in \mathbf{R}^n: \sum_{i} a_i x_i = \alpha\}$, this means, as @drhab wrote $\mathbf P(X \in H) = 1$ for the hyperplane $H$.

Answer 2

As is nicely explained here What's the best way to think about the covariance matrix?

if $A$ is the covariance matrix of some random vector $X\in\mathbb{R}^n$, then for every fixed $\beta\in\mathbb{R}^n$, the variance of the inner product $\langle\beta,X\rangle$ is given by $\langle A\beta,\beta\rangle$. Now, if $A$ is not invertible, there exists a non-zero vector $\beta\neq 0$ such that $A\beta=0$, and so $\langle A\beta,\beta\rangle = 0$, which means that the variance of $\langle X,\beta\rangle$ is zero.

Proposition 1. If the covariance matrix of a random vector $X$ is not invertible then there exists a non-trivial linear combination of the components of $X$ whose variance is zero.

This is closely related to what drhab mentioned in a comment above - for if the variance of $\langle X,\beta\rangle$ is zero, then $X-a\beta$ is almost surely orthogonal to $\beta$, for some constant $a$.In fact an alternative but equivalent formulation to the proposition above is:

Proposition 2. If the covariance matrix of a random vector $X$ is not invertible then there exists $\beta\neq 0$ such that a translate of $X$ is orthogonal to $\beta$ with probability one.

Answer 3

The question actually has little to do with probability theory, the observation holds for any square matrix regardless of it's origin.

It's easy to prove by considering the eigenvalues of the matrix. If and only if all of them are non-zero is the matrix invertible. It follows from the characteristic equation $\det (A-\lambda I)=0$, if $\lambda = 0$ is a solution then and only then $\det(A-0I) = \det(A) = 0$.

Positive definite means that all eigenvalues are positive, but positive semi-definite means only that they are non-negative.

Answer 4

This are some thoughts. Let $x$ be a random vector whose entries are i.i.d. Let $A$ be any square matrix which is not full rank. Then the covariance matrix of the random vector $y=Ax$ is not invertible. To see this, note that $E[Axx^TA]=AE[xx^T]A^T$. Thus, regardless of the rank of $E[xx^T]$, covariance matrix of $y$ will not be invertible.

Answer 5

Here is a collection of facts related to covariance matrices. They are useful when the covariance matrix is singular.

Consider a $p$-dimensional random vector $\mathbf X$ with square-integrable components, let $\mathbf \mu$ denote its mean vector, let $\mathbf \Sigma$ denote its covariance matrix and let $k=\operatorname{rank} \mathbf \Sigma$.

If $\mathbf a \in \ker \mathbf \Sigma$, then $\mathbf a^\top \mathbf X$ has variance $0$, hence $\mathbf a^\top (\mathbf X - \mathbf \mu) = \mathbf 0$ a.s. By considering a basis of $\ker \mathbf \Sigma$, it follows that $\mathbf X\in \mathbf \mu + (\ker \mathbf \Sigma)^\perp$ a.s., and since $\mathbf \Sigma$ is symmetric we obtain $\mathbb P(\mathbf X\in \mathbf \mu + \operatorname{Im} \mathbf \Sigma) = 1$. Hence $\mathbf X$ lies with probability $1$ in $\mathbf \mu + \operatorname{Im} \mathbf \Sigma$, which is an affine subspace of dimension $k$.

---

It is possible to find a matrix $\mathbf A\in \mathbb R^{k\times p}$ such that the $k$-dimensional random vector $\mathbf A \mathbf X$ has an invertible covariance matrix.

Indeed, write $\lambda_1,\ldots,\lambda_k$ the positive eigenvalues of $\mathbf \Sigma$ and diagonalize it as $\mathbf \Sigma = \mathbf P \mathbf D \mathbf P^\top$ where $\mathbf D = \operatorname{diag}(\lambda_1,\ldots,\lambda_k,0,\ldots,0)$. Next, define $\mathbf A = [I_k|\mathbf 0_{k\times (p-k)}]\mathbf P^\top$ and note that the covariance matrix of $\mathbf A \mathbf X$ is $\mathbf A \mathbf \Sigma \mathbf A^\top = \operatorname{diag}(\lambda_1,\ldots,\lambda_k)$, which is positive definite.

---

It is even possible to find $k$ components of $\mathbf X$ such that the corresponding $k$-dimensional vector has an invertible covariance matrix.

Let $\mathbf \Delta = \operatorname{diag}(\lambda_1,\ldots,\lambda_k)$ and write $$\mathbf \Sigma = \mathbf P \mathbf D \mathbf P^\top = \left[\begin{array}{ c| c } \mathbf P_1 & \mathbf P_2 \end{array}\right] \left[\begin{array}{ c | c } \mathbf \Delta & \mathbf 0 \\ \hline \mathbf 0 & \mathbf 0 \end{array}\right] \left[\begin{array}{ c| c } \mathbf P_1 & \mathbf P_2 \end{array}\right]^\top = \mathbf P_1 \mathbf \Delta^{1/2} \mathbf \Delta^{1/2}\mathbf P_1^\top = \mathbf G^\top \mathbf G,$$ where $\mathbf G =\Delta^{1/2}\mathbf P_1^\top \in \mathbb R^{k\times p}$.

Note that $k = \operatorname{rank}\mathbf \Sigma = \operatorname{rank}\mathbf G^\top \mathbf G = \operatorname{rank}\mathbf G$, hence from the columns of $\mathbf G$ (say $\mathbf g_1,\ldots,\mathbf g_p$) we can extract a basis of $\mathbb R^k$, say $\mathbf g_{i_1},\ldots,\mathbf g_{i_k}$. Let $I = \{i_1,\ldots,i_k \}$ and $\mathbf G_I$ denote the matrix with columns $\mathbf g_{i_1},\ldots,\mathbf g_{i_k}$. Then $\mathbf G_I^\top \mathbf G_I$ is invertible, and it is the covariance matrix of the $k$-vector $(X_i)_{i\in I}$.

Next, assume without loss of generality that $I=\{1,\ldots,k\}$ and partition $\mathbf X = \left[\begin{array}{ c } \mathbf X_I \\ \hline\mathbf X_{I^c} \end{array}\right]$, $\mathbf \Sigma = \left[\begin{array}{ c | c } \mathbf \Sigma_{II} & \mathbf \Sigma_{II^c} \\ \hline \mathbf \Sigma_{I^cI} & \mathbf \Sigma_{I^cI^c} \end{array}\right] $, $\mathbf G = \left[\begin{array}{ c| c } \mathbf G_I & \mathbf G_{I^c} \end{array}\right]$.

Note that $\mathbf \Sigma_{II} = \mathbf G_I^\top \mathbf G_I$, $\mathbf \Sigma_{II^c} = \mathbf G_I^\top \mathbf G_{I^c}$, $\mathbf \Sigma_{I^cI} = \mathbf G_{I^c}^\top \mathbf G_{I}$, $\mathbf \Sigma_{I^cI^c} = \mathbf G_{I^c}^\top \mathbf G_{I^c}$. It follows from straightforward algebra that the Schur complement of $\mathbf \Sigma_{II}$ in $\mathbf \Sigma$ is $\mathbf 0$: $$\mathbf \Sigma_{I^cI^c} - \mathbf \Sigma_{I^cI}\mathbf\Sigma_{II}^{-1}\mathbf \Sigma_{II^c} =\mathbf G_{I^c}^\top \Big(\mathbf I - \mathbf G_I (\mathbf G_I^\top \mathbf G_I)^{-1} \mathbf G_I^\top \Big)\mathbf G_{I^c} =\mathbf 0.$$ It is well-known that $\mathbf \Sigma_{I^cI^c} - \mathbf \Sigma_{I^cI}\mathbf\Sigma_{II}^{-1}\mathbf \Sigma_{II^c}$ is the covariance matrix of $$\mathbf X_{I^c} - \mathbf \mu_{I^c} - \mathbf \Sigma_{I^cI} \mathbf \Sigma_{II}^{-1}(\mathbf X_{I}- \mathbf \mu_{I}).$$ This last quantity is in fact the residual in the best approximation of $\mathbf X_{I^c}$ by an affine function of $\mathbf X_{I}$. Since the residual has mean $\mathbf 0$ and covariance matrix $\mathbf 0$, it is $\mathbf 0$ almost surely. Therefore, $$\mathbb P\Big(\mathbf X_{I^c} = \mathbf \mu_{I^c} + \mathbf \Sigma_{I^cI} \mathbf \Sigma_{II}^{-1}(\mathbf X_{I}- \mathbf \mu_{I})\Big) = 1,$$ and $\mathbf X_{I^c}$ is an affine function of $\mathbf X_{I}$.