Here is a generic quartic equation using a different approach.
Consider a right $\triangle ABC$ with legs $BC=a$ and $AB=b$ (as shown in the diagram below).

Let $(x,y)$ be the coordinates of point $M$ lying inside $\triangle ABC$ such that it divides $\triangle ABC$ into three equal areas $PMQB$ (labelled 'grey'), $[MRCQ]$ (labelled 'blue') and $[ARMP]$ (labelled 'yellow').
$$\therefore [PMQB]=[MRCQ]=[ARMP]=\frac{1}{3}[\triangle ABC]\tag 1$$
From $[PMQB]=\frac{1}{3}[\triangle ABC]$,
$$xy=\dfrac13\left(\dfrac{1}{2}ab\right)\implies y=\dfrac{ab}{6x}\tag 2$$
From Eq.(1), total area of $\triangle ABC$,
$$[PMQB]+[MRCQ]+[ARMP]=[\triangle ABC]$$
$$xy+([\triangle APM]+[\triangle ARM])+([\triangle MQC]+[\triangle MRC])=\dfrac 12ab$$
$$xy+[\triangle APM]+[\triangle MQC]+[\triangle AMC]=\dfrac 12ab\ \ \ \quad \left(\because [\triangle ARM]+[\triangle MRC]=[\triangle AMC]\right)$$
$$xy+\dfrac12(b-y)x+\dfrac 12(a-x)y+\dfrac12(MR)\sqrt{a^2+b^2}=\dfrac 12ab$$
$$\implies MR=\dfrac{ab-bx-ay}{\sqrt{a^2+b^2}}\tag 3$$
Now from $\triangle APS\sim \triangle ABC$,
$$\dfrac{PS}{BC}=\dfrac{AP}{AB}\implies PS=\dfrac{a}{b}(b-y)$$
From Eq.(1),
$$[PMQB]+[MRCQ]=\frac{2}{3}[\triangle ABC]$$
$$[PSCB]+[\triangle MRS]=\frac{2}{3}[\triangle ABC]$$
$$\dfrac{1}{2}(PS+BC)(MQ)+\dfrac{1}{2}(MR)(RS)=\frac{2}{3}\left(\dfrac 12ab\right)$$
$$\dfrac{1}{2}\left(\dfrac{a}{b}(b-y)+a\right)y+\dfrac{1}{2}(MR)(MR\tan\alpha)=\dfrac 13ab$$
$$\dfrac{1}{2}\left(\dfrac{2ab-ay}{b}\right)y+\dfrac{1}{2}(MR)^2\dfrac{a}{b}=\dfrac 13ab$$
$$\dfrac{2aby-ay^2}{2b}+\dfrac{a}{2b}\left(\dfrac{ab-bx-ay}{\sqrt{a^2+b^2}}\right)^2=\dfrac 13ab\quad \quad \left(\text{substituting MR from}\ Eq.(3)\right)$$
$$\dfrac{2ab\cdot \dfrac{ab}{6x}-a\left(\dfrac{ab}{6x}\right)^2}{2b}+\dfrac{a}{2b}\left(\dfrac{ab-bx-a\cdot \dfrac{ab}{6x}}{\sqrt{a^2+b^2}}\right)^2=\dfrac 13ab\quad \quad \left(\text{substituting y from}\ Eq.(1)\right)$$
$$36x^4-72ax^3+24(a^2-b^2)x^2+12ab^2x-a^2b^2=0\tag 4$$
The above quartic equation (4) always gives a unique positive real value of $x$, and the corresponding value of $y$ from Eq.(2), such that the point $M(x,y)$ lies inside the right $\triangle ABC$ and hence it always satisfies the following condition
$$a, b, x, y, ab-bx-ay>0\iff x\in \left(\dfrac{(3-\sqrt{3})a}{6}, \dfrac{(3+\sqrt{3})a}{6}\right) $$
In general, for a right $\triangle ABC $ with its legs $BC=a$, and $AB=b$ coinciding with the +ve x and y-axes respectively, the coordinates of point $M$ inside triangle are given as
$$\boxed{\color{blue}{M(x,y)\equiv\left(x,\dfrac{ab}{6x}\right)}}$$
where $x$ is a root of the following quartic equation:
$$\boxed{\color{blue}{36x^4-72ax^3+24(a^2-b^2)x^2+12ab^2x-a^2b^2=0\quad \forall \ x\in \left(\dfrac{(3-\sqrt{3})a}{6}, \dfrac{(3+\sqrt{3})a}{6}\right) }}$$
Now, a per given values in the question, $a=BC=5$ and $b=AB=\sqrt{11^2-5^2}=4\sqrt{6}$, the quartic equation is as follows
$$36x^4-72(5)x^3+24(5^2-(4\sqrt6)^2)x^2+12(5)(4\sqrt6)^2x-(5)^2(4\sqrt6)^2=0$$
$$3x^4-30x^3-142x^2+480x-200=0\tag 5$$
$$(x-2)(3x^3-24x^2-190x+100)=0$$
Eq(5) is the same quartic equation as obtained by OP that leads to the same solution as given by @Intelligenti pauca.