How to define orientation from the axioms of Euclidean geometry

Question

My question originates from this answer by Kulisty, where it is stated that the notion of orientation can be defined using only the axioms of neutral geometry. I am particularly interested in proving the transitivity of that orientation relation.

Since my context is Euclidean geometry (thus allowing for the use of congruence and parallelism), I interpret the concept in terms of free Euclidean vectors. Specifically:

Let $(l, m)$ be a pair of distinct, non-adjacent oriented half-lines issuing from a common origin $O$. This pair induces an half-plane $M$: it has the full line that contains the half-line $l$ as boundary and contains $m$ in its interior.

Now, given two such pairs $(l, m)$ and $(l', m')$, both originating from the same point $O$, I define the relation $\sim$ as follows: $$ (l, m) \sim (l', m') \quad \text{iff} \quad \left(l' \subset M \land l \nsubseteq M'\right) \lor \left(l' \nsubseteq M \land l \subset M'\right) $$ This relation is intended to capture the same notion of orientation as defined by Kulisty. The case where $l = l'$ (or $l=-l'$) is straightforward and omitted from the definition for brevity.

Now, since I am working in Euclidean geometry, I translate this into the language of vectors: to each pair $(l, m)$ I associate a pair of normalized orthogonal vectors $(\vec{l}, \vec{m})$. Then, the orientation relation above is equivalent to requiring that the determinant of the change-of-basis matrix from $(\vec{l},\vec{m})$ to $(\vec{l'}$, $\vec{m'})$ is positive.

However, I am looking for a proof of transitivity of the relation $\sim$ without appealing to the sign of the determinant.

Additionally, I am interested in extending this notion of orientation to 3-dimensional Euclidean space. In the plane, orientation corresponds to choosing a "direction" (e.g., counterclockwise), whereas in 3D it corresponds to choosing between left-handed and right-handed systems.

My questions are:

1. Can transitivity of the relation $\sim$ be proved purely in geometric (synthetic or vector-based) terms, avoiding any reference to the determinant sign?
2. Is there a natural way to define an analogous orientation relation between triples of non-coplanar vectors in 3D Euclidean space that corresponds to the usual right-hand rule?

(July 7th edit: added a reformulation)

Given a fixed coordinate system on the plane $E$, the pair of half-lines $(l, m)$ determines a linear function $\phi(x, y)$ such that $\phi(P) = 0$ for all points $P \in l$, and $\phi(P) > 0$ for all points $P \in m$. The relation $(l_1, m_1) \sim (l_2, m_2)$ can then be reformulated as $\phi_1(l_2) \cdot \phi_2(l_1) < 0$.

Answer 1

There is a reasonably straightforward way to do what you want in neutral synthetic planar geometry, once you have developed the theory of rigid motions. That theory is developed from the Hilbert axioms, in particular the axioms of congruence; you can find some of these ideas sketched out in Hartshorne's book. There is another point of view which I must say I very much prefer (apologies to David Hilbert) where one instead replaces the axioms of congruence with axioms of rigid motion.

---

Here's the short version: $(l,m) \sim (l',m')$ if and only if the unique rigid motion that takes $M$ to $M'$ and $l$ to $l'$ is not a reflection and is not a glide reflection.

---

Here's a longer version (followed by an explanation of the transitivity of the orientation relation). First, some theorems of rigid motions (also known as The axioms of rigid motions, from the other point of view):

1. The set of rigid motions is a subgroup of the group of permutations of the set of points of the plane (using the operation of composition).
2. Every rigid motion takes lines to lines, and rays to rays (ray $=$ half-line), and half-planes to half-planes.
3. Let me define a frame as a pair $(H,l)$ where $H$ is a half-plane and $l$ is a ray contained in the boundary line of $H$. The action of the group of rigid motions on the set of frames is faithful and transitive, meaning that for any two frames $(H,l)$ and $(H',l')$ there is a unique rigid motion $T$ such that $T(H)=H'$ and $T(l)=l'$.

Next, one develops a (partial) classification of rigid motions. As part of this classification one defines reflections and glide reflections (there are also rotations and translations, but I am avoiding those for reasons to be explained).

Here is a theorem of neutral geometry:

The group of rigid motions $\mathcal M$ has an index 2 subgroup $\mathcal M_+$ such that its coset $\mathcal M_- = \mathcal M - \mathcal M_+$ consists of the reflections and the glide reflections.

Let's give $\mathcal M_+$ a name: we shall call it the subgroup of orientation preserving rigid motions.

$\bigl($Let me add that in neutral geometry, $\mathcal M_+$ can also be described as the identity, the rotations, the translations, and "everything else, each of which can be described as a composition of two rotations". In Euclidean geometry, where one assumes the parallel postulate, "everything else" is unnecessary: the composition of two rotations is always the identity, a rotation, or a translation. In hyperbolic geometry, where one denies the parallel postulate, "everything else" is a parabolic transformation. Fortunately, this "non-neutral" issue is irrelevant to your question on orientation, although it caused me a little headache in getting this answer right.$\bigr)$

So, consider the set of "oriented angles" which is my shortand for an ordered pair $(l,m)$ of distinct, nonadjacent rays having the same endpoint; and I presume that by "nonadjacent" you mean that the two rays $l,m$ do not lie in the same line.

Consider two oriented angles $(l,m)$ and $(l',m')$, and let me denote their corresponding half-planes by $H$ and $H'$ according to your definition: $l$ is the boundary of $H$ and $m \subset H$; also $l'$ is the boundary of $H'$, and $m' \subset H'$. The relation $(l,m) \sim (l',m')$ is defined to mean that the unique rigid motion $T$ that takes the frame $(H,l)$ to the frame $(H',l')$ is contained in the orientation preserving subgroup $\mathcal M_+$; equivalently, $T$ is neither a reflection nor a glide reflector.

As a side effect, using the fact that $\mathcal M_+$ has index 2 in $\mathcal M$, it follows immediately that there are exactly two orientation classes.

---

Finally, to prove transitivity of the orientation relation, suppose that $(l,m) \sim (l',m')$ and that $(l',m') \sim (l'',m'')$. Let $T$ be the rigid motion taking $(H,l)$ to $(H',l')$ and let $T'$ be the rigid motion taking $(H',l')$ to $(H'',l'')$; it follows that $T' \circ T$ takes $(H,l)$ to $(H'',l'')$. Since $\mathcal M_+$ is a subgroup, it follows that $T' \circ T \in \mathcal M_+$, and therefore $(l,m) \sim (l'',m'')$.

Answer 2

The transitivity of this relation is proved similarly to transitivity of my relation.

Let me introduce the notation $l_1-l_2-l_3$ to denote the fact that the ray $l_2$ lies between rays $l_1$ and $l_3$ (rays $l_1$, $l_2$ and $l_3$ have the same origin here). This is a standard notion which can be defined within axiomatic approach. The following lemma seems obvious.

Lemma 1. If rays $l$, $k$ and $k'$ have the same origin, and $k$ and $k'$ lie on the same side of the line containing $l$, then either $l-k-k'$ or $l-k'-k$ or $k=k'$.

Below $M^*$ denotes the halfplane complementary to halfplane $M$ and $l^*$ denotes the ray complementary do ray $l$.

It is perhaps easier to define the orientation relation between pairs $(l,M)$, where $M$ is a halfplane with the boundary containing the ray $l$ (we consider only rays $l$ with one fixed origin).

Proof of transitivity. Assume that $(l_1,M_1)\sim(l_2,M_2)\sim(l_3,M_3)$. We must consider several cases. For example, assume $l_2\subset M_1$, $l_1\subset M_2^*$, $l_3\subset M_2$, $l_2\subset M_3^*$. Now we have three possibilities:

1. Rays $l_1$ and $l_3$ lie on on line. Then, since $l_1\subset M_2^*$ and $l_3\subset M_2$, we get $l_1=l_3^*$. Next, since $M_1$ and $M_3^*$ have a ray $l_2$ in common, we get $M_1=M_3^*$. This is one of the omitted cases from the definition of $\sim$.
2. $l_1\subset M_3$. Then, from this and $l_2^*\subset M_3$, by Lemma 1, we get $l_3-l_1-l_2^*$ or $l_3-l_2^*-l_1$. The first case is impossible, because rays $l_1$ and $l_3$ lie on opposite sides of the line containing $l_2$. Hence $l_3-l_2^*-l_1$ and since $l_2^*\subset M_1^*$, we get $l_3\subset M_1^*$ satisfying the definition of $(l_1,M_1)\sim (l_3,M_3)$.
3. $l_1\subset M_3^*$. Then, proceeding similarly as in case 2, we obtain $l_3\subset M_1$.

In general we must consider many, many cases is the similar way as above.

Possible answer to question 2. I believe this can be done. Suppose we have triples of noncoplanar rays $(l,m,n)$ and $(l',m',n')$ with the common origin $o$. Let $P$ and $P'$ be the planes determined by $(l,m)$ and $(l',m')$ respectiveley. If $P$ and $P'$ intersect along the line $K$, the we choose a ray $k$ included in $K$ with the origin $o$, and let $M$ and $M'$ be the halfplanes with the boundary $K$ such that the (planar) orientation $(k,M)$ is the same as the orientation $(l,m)$ and the orientation $(k,M')$ is the same as the orientation $(l',m')$. Then we define $$(l,m,n)\sim(l',m',n') :\iff (M\subset H'\wedge M'\subset H^*)\vee (M\subset H'^*\wedge M'\subset H),$$ where $H$ and $H'$ are halfspaces determined by rays $n$ and $n'$ respectively. The definition does not depend on one of the two possible choices of $k$. The case $P=P'$ is omitted for brevity.

I also believe this 3d definition can be extended to triples not with the same origin (as in the planar case).