This is not a proof, just a commentary on your attempt and a visual representation.
I am puzzled by this exercise which necessitates indeed to have a certain knowledge of conic curves (you say you have this knowledge) but also of techniques of minimization that look beyond the use of simple differentiation (and you say you don't know the technique of Lagrange multipliers).
Nevertheless, I show you what I have done graphically without saying it is a solution.

Fig. 1: Hyperbolas $(C'_m)$ for $m$ ranging from $30/6$ (the highest) to $49/6$, the lowest, which is the solution, just tangent to the arc of ellipse.
First of all, your transformation by squaring into :
$$(x-y)^2 + 3(x-2)(y-2) = 3$$
brings in a "spurious arc" of curve ; indeed, to the (blue) curve with equation
$$(C) \ : \ \ \ \ x+y+\sqrt{xy}=3,$$
you add the (cyan) curve $x+y-\sqrt{xy}=3$ which is "parasitic".
How are the red curves defined ? By parametric equations :
$$(C'_m) \ : \ \ \ \ \frac{1}{x}+\frac{10}{y}=m$$
which are branches of hyperbolas.
The question boils down to find the largest value of $m$ for which there is a point $(x,y)$ belonging to $C$ and $C'_m$. Indeed, this looks to happen for $m=49/6$. But a visual conviction is not a proof.
SAGE program which has generated Fig. 1 :
var('x y');
g=implicit_plot(x+y+sqrt(x*y)-3, (x,0,5), (y,0,5),color='blue');
h=implicit_plot(x+y-sqrt(x*y)-3, (x,0,5), (y,0,5),color='cyan');
g=g+h;
for m in range(30,49) :
h=implicit_plot((10/y+1/x-m/6),(x,0,5),(y,0,5),color='red');
g=g+h;
show(g);