What does it mean to sample, in measure theoretic terms?

Question

Suppose I have some random variable $X$ defined on some probability space $(\Omega, \mathcal{F}, \mathbb{P})$. What does it mean, in measure theoretic terms, to draw a sample from $X$?

When $\Omega$ is finite, things make sense: we might say nature rolls its dice and draws an $\omega \in \Omega$ according to $\mathbb{P}$, and our sample is simply $X(\omega)$.

In uncountably infinite domains its not clear to me how this is defined. As we all know, every element $\omega \in \Omega$ has zero measure, only measurable subsets $A \subseteq \Omega$ have measure. But sampling implies getting elements of $\Omega$. How does one reconcile the idea of drawing a sample $x$ from some distribution with the fact that $P(X=x) = 0$?

And yet, I can call the rand function in my favourite programming language and it will sample from a continuous distribution. But of course it is not really continuous, it is a floating point approximation. Could it be that the discretisation is required to have a well-defined notion of sampling?

Answer 1

The simple answer is that in measure theory, we will talk about distributions of draws and not work with the individual draws themselves. There are ways to define uncountably many draws from a continuous distribution sensibly, but they are highly nontrivial.

Your favorite programming language avoids the problem by approximation.

Btw: It all depends on the measure, nothing prevents me from defining a probability measure on an uncountable domian that picks a certain element with probability $1$.

Answer 2

You don't even have to go to uncountable domains for this to occur. Consider a countably infinite sample space such as the rationals in interval $\left[0,1\right]$. If all outcomes are equally likely, the probability of any single rational is $0$.

But that's reasonable because it doesn't make sense to measure individual sample points when the possible outcomes are infinite and equally likely. This is really what you allude to in your last paragraph.

I think the paradox exists because this type of probability model doesn't really represent real world situations (either the countable or uncountable case) where an actual sample is taken. In a theoretically unbounded, countable sample space, such as $\mathbb{Z}^+$, in actuality there is normally either some upper bound on the possible outcomes or else the probabilities decrease as $\infty$ is approached. In either case, the lower-valued (more realistic) outcomes actually have a tiny but positive probability.

For the bounded case, say the reals in interval $\left[0,1\right]$, again any real world situation does not really match up. There is always some limit on the accuracy of measurement (number of decimal places, if you like) of each outcome. So you are really dealing, in terms of measurable sets, with intervals in $\left[0,1\right]$, however tiny they might be. For this not to be true, it implies that a number with $5$ billion decimal places is as equally likely as $0$ or $0.1$ or $0.345$, which would be difficult to justify.

That's why it makes sense to work with $\sigma$-fields of subsets of $\Omega$. The need for $\sigma$-fields is more obvious with infinite sample spaces than with finite ones.

I don't know if that helps at all.