Let $A$ and $B$ in $M_n(\mathbb C)$ such that the rank of $AB-BA$ is $1$. Prove that $A$ and $B$ are simultaneously triangularisable.
This generalizes the classical case $AB = BA$.
By induction on $n$, it suffices to show that $A$ and $B$ have a common eigenvector. So, it would be sufficient to find a eigenspace of $A$ which is stable by $B$ since matrices are complex.
Do you have ideas for that? Thank you.
That follows from the book Simultaneous Triangularization by Radjavi and Rosenthal (page 8). The original proof is due to Thomas Laffey.
Let $\{y\}$ be a basis of $\mathrm{Im}(AB-BA)$. Let $\lambda\in\mathrm{Spec}(B)$. If $B=\lambda I$, then there is almost nothing to do. Otherwise $F=\ker(B-\lambda I)$, $G=\mathrm{Im}(B-\lambda I)$ are non-trivial $B$-invariant subspaces. If we show that $F$ or $G$ is $A$-invariant, then we are the kings of oil.
Assume that $F$ is not $A$-invariant. Then there is $x$ s.t. $(B-\lambda I)x=0$, $(B-\lambda I)Ax\not= 0$. We have $$A(B-\lambda I)x-(B-\lambda I)Ax=ABx-BAx=-(B-\lambda I)Ax\in\mathrm{Im}(AB-BA)\cap\mathrm{Im}(B-\lambda I)\setminus\{0\}.$$ Thus $y\in G$.
Let $z\in \mathbb{C}^n$. Then $A(B-\lambda I)z$ is in the form $(B-\lambda I)Az+\alpha y$. Therefore, $G$ is $A$-invariant and we are done. $\square$
The proof of @user91684 is a shadow of a larger phenomenon!
Let $\mathcal{C}$ be an abelian category.
Definition: We say that an object V is simple if for every other object W and any Hom, $f: V \to W$ we have that either $f$ is the zero map or that it is a monomorphism
Lemma: Let $g: V \to V$ be a map such that $Im(g)$ is simple. Then for any maps $f: U \to V, h: V \to W$ such that $h \circ g \circ f = 0$, we have that either $g \circ f = 0$ or $h \circ g = 0$.
Proof: Write $g = m \circ e$, where $m: Im (g) \to V$ is mono and $e: V \to Im(g)$ is epi. Then we have that by the hypothesis $(h \circ m) \circ (e \circ f) = 0$. Suppose that $h \circ m = 0$ then $h \circ g = 0$ and we're done. Else, since it's a non zero map from a simple object we have that $h \circ m$ is a mono. Thus from the hypothesis, $e \circ f = 0$.
Now we note that on setting $\mathcal{C}$ to be the category of vector spaces, $V = \mathbb{F}^n$, $g = AB - BA$, $f$ to be the inclusion map of $\ker(A)$ into $V$, and $h$ to be the map from $V$ onto $\mathrm{coker}(A)$, we have that from a tautological calculation.
$$ h \circ g \circ f = 0 $$
But our hypothesis is that the rank of $g$ is at most one, i.e it's image is a simple object in the category of vector spaces
Thus, we get that either $h \circ g = 0$ or that $g \circ f = 0$. The former is equivalent to $B(Im (A)) \subset Im(A)$ and the latter is equivalent to $B(\ker(A)) \subset \ker(A)$.
Now if $\mathbb{F}$ is algebraically closed we can plug into $A$ an $A-\lambda I_n$ where $\ker(A-\lambda I_n)$ is non zero. By induction on the dimension of the space that $A,B$ act on, we're done.
Lets prove that $A$ and $B$ have at least one common eigenvector. If $x\in \ker(A-\lambda I)$, we have $$ (AB-BA)x = (A-\lambda I)Bx. $$ For $x\in \ker (AB-BA)$ this imply $Bx\in \ker(A - \lambda I)$, which means that $\ker(A-\lambda I)$ is an invariant subspace for operator $B$, so there is an eigenvector $y$ of $B$ lying in it.
We win when at least one eigenvector of $A$ or $B$ lying in $\ker(AB-BA)$. Consider the case when all the eigenvectors of $A$ and $B$ are not in $\ker(AB-BA)$. From well-known identity $$ \dim\ker(AB-BA) = n - \mathrm{rank}\,(AB-BA) = n-1, $$ so eigenvectors of both $A$ and $B$ lying in the subspace of dimension $1$, which means they have common eigenvector.
Now, lets $v$ be a common eigenvector of $A$ and $B$ in some basis $\{e_1,\dots, e_n\}$. Then in basis $\{v,e_2,\dots, e_n\}$ matrices of $A$ and $B$ become $$ A = \begin{pmatrix} \lambda & X \\ 0 & A_1 \end{pmatrix}, \quad B = \begin{pmatrix} \mu & Y \\ 0 & B_1 \end{pmatrix}. $$ This yields $\mathrm{rank}(A_1B_1-B_1A_1) \leq \mathrm{rank}(AB-BA)$, so we have the same situation for $A_1$ and $B_1$. Repeating this reasoning $n$ times we obtain the upper-triangular forms of $A$ and $B$ in the same basis.