Say I want to apply $\hat{A}=\left(x\frac{d}{dx}x\right)^2$ to $f(x)=\exp(-x)$. How do I do it exactly?
Also, in what order?
Do I first multiply by $x$ then get calculate derivative and repeat, or
do I first calculate derivative then multiply by $x$ then repeat?
$f(x)=\text{exp}(-x) \implies \frac{d}{dx}\left(xf(x)+x^2\frac{d}{dx}(f(x)\right)=\frac{d}{dx}\left(x \text{exp}(-x)-x^2 \text{exp}(-x)\right)$ and just calculate the RHS?
Let $\hat{A} = \left( x \dfrac{d}{dx} x \right)^2$. Let's introduce the temporary notation $\hat{B} = x \dfrac{d}{dx} x$, so that $\hat{A} = \hat{B}^2$. Then for any function $f(x)$ we would have
$$\hat{B} f(x) = x\dfrac{d}{dx}\left( xf(x) \right) $$
so $$\begin{aligned} \hat{A} f(x) &= \hat{B} \left( \hat{B} f(x) \right) \\ &= \hat{B} \left(x\dfrac{d}{dx}\left( xf(x) \right)\right) \\ &= x \dfrac{d}{dx} \left( x \left(x\dfrac{d}{dx}\left( xf(x) \right)\right) \right) \end{aligned}$$
Now just work your way from the inside out.
One has $A = (xDx)^2 = (xDx)(xDx) = xDx^2Dx$. Then, apply the operators from the right to the left, i.e. $$ \begin{align} Af &= (xDx^2Dx)f \\ &= xDx^2D(xf) \\ &= xDx^2(f + xf') \\ &= xD(x^2f + x^3f') \\ &= x(2xf + x^2f' + 3x^2f' + x^3f'') \\ &= 2x^2f + 4x^3f' + x^4f'' \end{align} $$
If you've had some group theory (just the beginnings of it, such as the first day or two in a class or text), then $\left(x\frac{d}{dx}x\right)^2$ is interpreted the same as $(abc)^2$ in a group where the group operation is not necessarily commutative (composition of operators is not commutative, but it is associative). Thus, $(abc)^2 = (abc)(abc) = abcabc,$ where the first equals is simply what the exponent of $2$ means and the second equals is because we are assuming associativity. (See this 12 September 2006 sci.math post for a lot of nuanced associativity issues.)
Even if you haven't seen any group theory, the following should still be clear. And googling for further examples (or look in physics quantum mechanics texts; and in quantum chemistry texts; and operator methods/algebra sections that often appear in ordinary differential equations texts and in undergraduate "math for physicists and engineers" texts) will help to see what to do when addition is thrown into the mix.
In your case:
$$ \left(x\frac{d}{dx}x\right)^2 \;\; = \;\; \left(x\frac{d}{dx}x\right) \left(x\frac{d}{dx}x\right) \;\; = \;\; \left(x \; \frac{d}{dx} \; x \; x \; \frac{d}{dx} \; x \right) $$
Since juxtaposition of these $6$ (not all distinct) operators means composition [1], and composition of functions is done from left to right [2] [3], carrying this out means doing the following steps in the order given:
Step 1: Multiply the input function by $x.$ Note that since ordinary (school) multiplication of functions is commutative, it doesn't matter whether you multiply on the right or on the left -- the product of the two functions (one is $x,$ the other is the input function) will be the same.
Step 2: Take the derivative of what you get after doing Step 1.
Step 3: Multiply by $x$ what you get after doing Step 2.
Step 4: Multiply by $x$ what you get after doing Step 3.
Step 5: Take the derivative of what you get after doing Step 4.
Step 6: Multiply by $x$ what you get after doing Step 5.
Once this is understood, you should be able to follow expositions where a sequence such as this is compressed notationally (e.g. Steps 3 and 4 can be replaced by a single step of multiplying by $x^2).$
[1] This is simply composition of functions that one sees in precalculus school math, except here the functions are called operators to distinguish them from their input functions.
[2] For some authors, it's right-to-left, so if something seems amiss, you'll want to double-check with the author's preference, which if not explicitly stated can be inferred by looking at some simple examples shown when it's first introduced.
[3] By the way, for various issues relating to function composition being done left-to-right instead of right-to-left, see the question/answers/comments to the Mathematics SE question History of $f \circ g$ and the Mathematics Educators SE question Why do we use functional composition in the order we do?.
In a more general sense, you use some sort of Functional Calculus
https://en.wikipedia.org/wiki/Functional_calculus
"....Which provides a justification for applying Mathematical Functions to Mathematical Operators... Now a branch of Functional Analysis..."
Edit: If our operator $T:= ( x \frac{d}{dx} x )^2$ were bounded , as if if acted on Test functions, we could apply the Continuous Calculus , where we define T in its Spectrum first and , informally, extend it densely through the Weirstrass Approximation theorem, to the whole space.
if not, if the operator was unbounded, as it would be if the action was on $ L^2(\mathbb R)$ we would use the Borel Functional Calculus to extend by using/ Integrating against, Projection-valued measures.