The original circle is the intersection of the unit-radius infinite cylinder
$$ x^2 + y^2 = 1 $$
and the $xy$ plane $z = 0$.
If $p = (x, y, z) = \begin{bmatrix} x \\ y \\ z \end{bmatrix} $
And $ Q = \begin{bmatrix} 1 && 0 && 0 \\ 0 && 1 && 0 \\ 0 && 0 && 0 \end{bmatrix}$
Then equivalent to the above, the unit circle in the $xy$ plane is the intersection of the cylinder
$ p^T Q p = 1 $
And the plane
$ e_3^T p = 0 $
where $e_3 = (0,0,1) = \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}$
Now you define an affine transformation on $p$ which produces an image $p'$ corresponding to $p$ as follows
$ p' = f(p) = A p + b $
Further, we project the image $p'$ onto a plane passing through the origin, and having a unit normal vector $n$. Let the projected $p'$ be $p''$ then
$p'' = [I_3 - {nn}^T] p' $
Define matrix $J = I_3 - {nn}^T$ ,then
$p'' = J p' $
Using the expression for $p'$ we get
$ p'' = J ( A p + b ) = (J \ A) \ p + (J \ b) = F p + g$
with $F = J \ A $ and $g = J \ b $.
It is well-known that matrix $J$ is rank-deficient, actually its rank is $2$. So, assuming matrix $A$ is full rank, then $J \ A $ will also be rank deficient, and have a rank of $2$. And because of that, we cannot invert matrix $F$, which we need to solve for $p$ in terms of $p''$.
However we can use vector parametric methods to find what we're looking for.
The original unit circle in the $xy$ plane can be written parametrically as
$ p(\theta) = u_1 \cos(\theta) + u_2 \sin(\theta) $
where $u_1 = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} $ and $u_2 = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix} $
Therefore,
$ p'(\theta) = A ( u_1 \cos(\theta) + u_2 \sin(\theta) ) + b $
And
$ p''(\theta) = F p + g = F (u_1 \cos(\theta) + u_2 \sin(\theta) ) + g $
Let's define $w_1 = F \ u_1 $ , $w_2 = F \ u_2 $
$ p''(\theta) = w_1 \cos(\theta) + w_2 \sin(\theta) + g $
Note that we know that $p''$ lies in the plane $ n \cdot (x, y, z) = 0 $
We can attach a reference frame to this plane, with its $x'y'$ plane coinciding with the plane itself, and its $z'$ axis along the normal $n$. If $q = (x', y', 0)$ is the coordinate of point on the projection plane with respect to the arbitrarily selected orientation of the $x'$ and $y'$ axes, then
$ p'' = R q $
Note that the third coordinate of $q$ is zero. The $3 \times 3$ $R$ is a rotation matrix and is given by
$ R = [ \hat{x'}, \hat{y'}, \hat{n} ] $
where $ \hat{x'} $ is the unit vector along the positive $x'$ axis expressed in world coordinates, and $\hat{y'}$ is the unit vector along the positive $y'$ axis expressed in world coordinates.
From this, it follows that
$ q(\theta) = R^{-1} p''(\theta) = R^{T} ( w_1 \cos(\theta) + w_2 \sin(\theta) + g ) $
Define $ y_1 = R^{T} w_1$ and $y_2 = R^T w_2 $ and $q_0 = R^T g $ , then
$ q(\theta) - q_0 = \begin{bmatrix} x' \\ y' \end{bmatrix} = \begin{bmatrix} y_1 && y_2 \end{bmatrix} \begin{bmatrix} \cos(\theta) \\ \sin(\theta) \end{bmatrix} $
Define
$Y = \begin{bmatrix} y_1 && y_2 \end{bmatrix} $ and $ u = \begin{bmatrix} \cos(\theta) \\ \sin(\theta) \end{bmatrix} $
then
$ q - q_0 = Y u \hspace{50pt}(1)$
Note that $q$ and $q_0$ have their third component identically zero.
Also, note that $Y$ is a $3 \times 2$ matrix, and $u$ is a $2 \times 1$ unit vector.
Pre-multiply both sides by $Y^T$, you get
$ Y^T q = (Y^T Y) u \hspace{50pt}(2)$
The matrix $(Y^T Y)$ is $2 \times 2$. Solving for $u$ we get
$ u = (Y^T Y)^{-1} Y^T (q - q_0) $
We know that $u$ is a unit vector, i.e. $u^T u = 1$ , therefore,
$ (q - q_0)^T \ Y (Y^T Y)^{-T} (Y^T Y)^{-1} Y^T \ (q - q_0) = 1 \hspace{50pt}{(3)} $
Recall that $q = \begin{bmatrix} x' \\ y' \\ 0 \end{bmatrix} $
Equation $(3)$ will give you the coefficients of $x'^2, y'^2, x'y', x', y'$ plus the constant term.
We can slightly modify $(3)$, by noting that matrix $Y$ can be written as
$ Y = R^T W $
where $W = \begin{bmatrix} w_1 && w_2 \end{bmatrix} $
Then $ Y^T Y = W^T R R^T W = W^T W $
Therefore, equation $(3)$ becomes
$ (q - q_0)^T \ R^T W (W^T W)^{-T} (W^T W)^{-1} W^T R \ (q - q_0) = 1 \hspace{30pt} (4) $
And recall that $q_0 = R^T g $.
A substantially different approach, starts with the equation
$ p''(\theta) = w_1 \cos(\theta) + w_2 \sin(\theta) + g \hspace{50pt}(5) $
which I copied from above.
I can now replace $ \theta $ with $(t - t_0)$ which $t$ is arbitrary and $t_0$ is a fixed value that is to be determined. With this, equation $(5)$ becomes
$ p''(t) = w_1 \cos( t - t_0) + w_2 \sin(t - t_0) + g \hspace{50pt}(6)$
Expand the trigonometric terms, to get
$ p''(t) = \cos(t) [ w_1 \cos(t_0) - w_2 \sin(t_0) ] + \sin(t) [ w_1 \sin(t_0) + w_2 \cos(t_0) ] + g \hspace{50pt} (7) $
By properly choosing $t_0$ we can make the vector $[ w_1 \cos(t_0) - w_2 \sin(t_0) ]$ orthogonal to the vector $[ w_1 \sin(t_0) + w_2 \cos(t_0) ]$, this requires that
$[ w_1 \cos(t_0) - w_2 \sin(t_0) ]\cdot [ w_1 \sin(t_0) + w_2 \cos(t_0) ] = 0 \hspace{50pt}(8)$
Expanding, gives us
$ (w_1 \cdot w_1 - w_2 \cdot w_2) ( \sin(t_0) \cos(t_0) ) + (w_1 \cdot w_2 ) ( \cos^2(t_0) - \sin^2(t_0) ) = 0 \hspace{50pt}(9) $
Using trigonometric identities, this in turn implies,
$ \dfrac{1}{2} \sin(2 t_0) (w_1 \cdot w_1 - w_2 \cdot w_2) + \cos(2 t_0) (w_1 \cdot w_2 ) = 0 \hspace{50pt}(10)$
And this finally gives us
$ t_0 = \dfrac{1}{2} \tan^{-1} \dfrac{ 2 (w_1 \cdot w_2 ) }{ w_2 \cdot w_2 - w_1 \cdot w_1 ) } \hspace{50pt}(11)$
This means that at this calculated $t_0$ the vectors
$ v_1 = w_1 \cos(t_0) - w_2 \sin(t_0) $
and
$ v_2 = w_1 \sin(t_0) + w_2 \cos(t_0) $
represent the major and minor semi-axes of the ellipse that is a result of transformation and projection.
Since $v_1$ is orthogonal to $v_2$ we can take these two directions to be the $x'$ and $y'$ axes direction.
So the equation of the ellipse now is
$ p''(t) = g + \cos(t) v_1 + \sin(t) v_2 \hspace{50pt}(12) $
where $v_1 \perp v_2$, therefore, taking the $x'$ coordinate to be along the $v_1$ direction and $y'$ to be along the $v_2$ direction, and further, if we let the origin of the coordinate system be at $g$ (which is known), then the quadratic polynomial describing the ellipse is
$ \dfrac{x'^2}{\| v_1 \|^2 } + \dfrac{ y'^2}{\| v_2 \|^2} = 1 \hspace{50pt}(13)$
To see this, look at equation $(12)$, we have
$ p''(t) - g = \cos(t) v_1 + \sin(t) v_2 \hspace{50pt}(12) $
Define the matrix
$ R = \begin{bmatrix} \dfrac{v_1}{\| v_1 \|} && \dfrac{v_2}{\| v_2 \| } && n \end{bmatrix} $
Now define the local coordinates vector $r = R^T (p'' - g) $
Then
$ r = \begin{bmatrix} x' \\ y' \\ 0 \end{bmatrix} = \begin{bmatrix} \| v_1 \| && 0 \\ 0 && \| v_2 \| \\ 0 && 0 \end{bmatrix} \begin{bmatrix} \cos(t) \\ \sin(t) \end{bmatrix} \hspace{50pt} (14) $
Disregarding the third row that is zero on both sides of the equation, and inverting, we get
$ \begin{bmatrix} \cos(t) \\ \sin(t) \end{bmatrix} = \begin{bmatrix} \dfrac{1}{\| v_1 \|} && 0 \\ 0 && \dfrac{1}{\| v_2 \|} \end{bmatrix} \begin{bmatrix} x' \\ y' \end{bmatrix} = \begin{bmatrix} \dfrac{x'}{\| v_1 \|} \\ \dfrac{y'}{\| v_2 \|} \end{bmatrix} $
Since $\cos^2(t) + \sin^2(t) = 1 $ , then
$ \dfrac{x'^2}{\| v_1 \|^2 } + \dfrac{ y'^2}{\| v_2 \|^2} = 1$
as expressed in equation $(13)$.