0

Suppose there are 10000 students in a university. I perform an experiment where I randomly select 50 of the students at some university and ask each student how many hours did each of these students study in the past week - and then I take the average of these measurements. I then repeat this experiment 10 times.

Suppose this produces the following results:

  • Experiment 1: min = 3 hrs, mean = 5.3 hrs, max = 10 hrs, 90% CI : (4.1, 6.2)
  • Experiment 2: min = 2.1 hrs, mean = 5.5 hrs, max = 8.7 hrs, 90% CI : (4.9, 6.1)
  • Experiment 3: min = 1.5 hrs, mean = 2.5 hrs, max = 3 hrs, 90% CI : (1.7, 2.9)
  • Experiment 4: min = 3.9 hrs, mean = 5.9 hrs, max = 11 hrs, 90% CI : (4.40, 7.1)
  • Experiment 5: min = 1.0 hrs, mean = 5.8 hrs, max = 16.4 hrs, 90% CI : (4.4, 6.7)
  • Experiment 6: min = 4.45 hrs, mean = 5.9 hrs, max = 13 hrs, 90% CI : (4.8, 6.5)
  • Experiment 7: min = 1.0 hrs, mean = 5.3 hrs, max = 17 hrs, 90% CI : (4.1, 5.5)
  • Experiment 8: min = 3.5 hrs, mean = 5.1 hrs, max = 11 hrs, 90% CI : (4.6, 6.2)
  • Experiment 9: min = 3 hrs, mean = 5.21 hrs, max = 9 hrs, 90% CI : (4.5, 5.5)
  • Experiment 10: min = 1.6 hrs, mean = 5.3 hrs, max = 19 hrs, 90% CI : (4.0, 7.8)

Suppose we have some magic way of knowing that the true population (i.e. all 10000 students) average is exactly 5 hours.

There is a lot of debate on the correct interpretation of Confidence Intervals (https://link.springer.com/article/10.3758/s13423-015-0947-8). My question relates to this concept.

For instance, suppose I was only able to perform one of these experiments and interviewed the exact same subset of students from "Experiment 7". If I were to say "there is a 0.9 probability that the average student in this university studies somewhere between 4.1 hours and 5.5 hours" - this statement would NOT be considered as correct.

In my opinion, the above statement is "somewhat correct" - below is my logic.

  • Let's assume that the results of these 10 experiments all representative of the population. By this I mean, in 9 out of 10 of these experiments, the true population mean is contained within the 90% Confidence Interval calculated using the data from each of these experiments.

  • This means, in any experiment I perform - I have a 0.9 probability of coming across a dataset in which the true population average is contained within the respective confidence interval corresponding to the data from that experiment.

  • This also means, in any experiment I perform - I have a 0.1 probability of coming across a dataset in which the true population average is NOT contained within the respective confidence interval corresponding to the data from that experiment.

  • Therefore, provided there are no major design and procedural flaws in my experiment - what stops me from believing that "I have a 0.9 probability of encountering a sample such that the true population average is contained within a 90% Confidence Interval calculated using this particular sample"?

  • In other words, what stops me from believing that - I am 90% confident that the sample I collected contains the true population average within its 90% Confidence Interval?

Can someone please help me understand why I am wrong?

Thanks!

stats_noob
  • 4,375

1 Answers1

2

If you perform an experiment, for any true mean, there is indeed $90\%$ probability that true mean will be in $90\%$ confidence interval. But it is not the same as after performing the experiment the true mean is in the interval with $90\%$ probability.

The difference between those two statements is that in the first, we have a random variable - our confidence interval. While in the second, we don't have any - the true mean isn't (in frequentist approach) a random variable, so question "what is the probability that true mean is in $[4.9, 5.1]$" is meaningless.

It can be more intuitive when we have some intuition about possible parameter values. For example, as xkcd suggests: assume that our survey for some reason in $1\%$ of cases ignores answers and declares that every student spent $10^{10}$ years studying. Then we can declare anything as confidence interval if that happened, and usual $0.9/0.99$ confidence interval if not.

With such "test", we will still get true mean in confidence interval in $90\%$ of cases, but I don't think after getting interval $[10^{10}, 10^{10}]$ to hope that true mean is indeed $10^{10}$ with probability $90\%$, even if such hope appears just in $1\%$ of experiments.

mihaild
  • 18,164