An object has been approximated by a lot of triangles. Given the vertex positions of these triangles, how can I compute the normal of these vertices which pointing outside of the object.
I know the cross-product of two edges gives the normal, but how can I know it's pointing towards the outside of the object.
Facing the same problem, I found the general solution here:
It is for 2-D but generalizes to 3-D as well. Also, it does not suffer from non-convexity.
The first step is the same as PolyMesh suggested. All the normals have to face the same direction. This can be done by fixing the normal of one face, and then traversing the triangles using BFS search and setting their normals the same as their neighbour which has already been visited using BFS.
The second step, however, is not only easier, but also works for both convex and non-convex geometries. Evaluate the sum $$ I = \sum_{i=1}^{N_\text{triangles}} x_i n_{xi} \text{Area}_i, $$ where $x_i$ is the x coordinate of each triangle midpoint, $n_{xi}$ is the x entry of the normal vector at the triangle midpoint, and $\text{Area}$ is the area of the triangle. The normals point outward iff I > 0, and they point inward otherwise. The reason is that $\int_\text{Area} xn_xdA = \int_\text{Area} yn_ydA = \text{Volume}$ iff normals are pointing outwards. This property can be proved using the divergence theorem.
For a convex object, add up all the vertices to get a center representation. Take the dot product of the normal you obtain with the vector joining the vertex to this center. If this is negative the normal is pointing outward. (For non-convex object this won't work though)
You could always check out the code for what Blender does since it is open source. Though without looking at it, I would probably suggest making adjacent face normals aim the same direction as their neighbor. I'd do this starting with the first triangle and work outward from there. It doesn't really matter if they are facing in or out at this stage, you just want them all the same. This should work for any enclosed/water-tight mesh which I assume we are dealing with.
Then I would find a bounding box center or average of all points or some find-a-center algorithm. If the majority of normals are facing inward, then flip all of them. If the majority of normals are already facing outward, you don't have to do anything.
This is based on the idea that even in concave meshes, most normals will face outward. And as long as the normals are all facing the same direction as their edge sharing buddies, the fewer inward facing faces also do the right thing.