Derivation of Stirling approximation from CLT

Question

In Casella Berger exercise 5.35, we have a sequence of sample means from the exponential $exp(1)$. Then, we can derive:

$$P\Bigg(\frac{\overline{X}_n - 1}{1/\sqrt{n}} \leq x\Bigg) \rightarrow P(Z \leq x)$$

where $Z \sim n(0, 1)$.

Now, the author suggests to show the Stirling approximation by taking the derivative on both sides. In short, he suggests:

$$\frac{d}{dx}P\Bigg(\frac{\overline{X}_n - 1}{1/\sqrt{n}} \leq x\Bigg) \approx \frac{d}{dx}P(Z \leq x)$$

from where the Stirling approximation follows.

Why are we justified doing this? Wiki article on weak convergence says:

"In general, convergence in distribution does not imply that the sequence of corresponding probability density functions will also converge."

Why is taking the derivative justified according to the author?

If it is allowed, can be it shown in some simple way without measure theory? Or link to the relevant theorems.

Answer 1

Let $f_n$ be the pdf of $Z_n=\sqrt{n}(\sum_i X_i/n-1)$ and $f$ be the pdf of a standard Gaussian. Then the exercise leads you to

$$f_n(0)=\frac{1}{n!}\left(\frac{n}{e}\right)^n \sqrt{n}$$

As you mention, from there we cannot simply use convergence in distribution to conclude that $f_n(0)\to f(0)=1/\sqrt{2\pi}$. Here is an argument based on characteristic functions to bypass this issue.

---

Let $\Phi_n(t)=e^{-it\sqrt{n}}(1-it/\sqrt{n})^{-n}$ be the characteristic function of $Z_n$ (note that it converges pointwise to $e^{-t^2/2}$, either by the CLT or by direct computation). Then,

$$|\Phi_n(t)|= \left| 1-\frac{it}{\sqrt{n}}\right|^{-n}=\left(1+\frac{t^2}{n}\right)^{-n/2}=\exp\left(-\frac{n}{2}\ln\left(1+\frac{t^2}{n}\right)\right)$$

Now it is just a simple exercise in real analysis to show that $|\Phi_n(t)|$ is uniformly dominated by an integrable function (treat the cases $t^2\le n$ and $t^2>n$ separately). Thus, from an inversion theorem and dominated convergence,

$$f_n(0)=\frac{1}{2\pi}\int_{-\infty}^{\infty} \Phi_n(t)\,dt\to 1/\sqrt{2\pi}.$$

Answer 2

With notation as above, it's simpler to evaluate $E(Z_n^+)$. By the standard CLT, this converges to $E(Z^+)=1/\sqrt{2\pi}$. No need for densities. Here $$x^+=\max(x,0).$$

The fact that $x^+$ is $relu(x)$ in machine learning makes this solution even more appealing. See https://mathdatasciencebook.com

To derive the moment-generating version of the standard CLT, assume $X$ is a random variable with mean zero, variance one, and finite moment-generating function $M(z)=E(e^{zX})$ for all complex $z$. If $Y$ is Poisson with mean $1$, these assumptions hold for $X=Y-1$.

Define $\epsilon_n(z)$ by the Taylor expansion $$M\left(\dfrac{z}{\sqrt{n}}\right) = 1 + \dfrac{z^2}{2n}(1+\epsilon_n(z))$$

Then derive the estimate $$\left|\sum_{k=0}^n \frac{z^k}{k!} - \left(1+\frac{z}n\right)^n\right| \le \frac{1}{n}\cdot \dfrac{|z|^2}2\cdot e^{|z|}$$ using the high-school binomial theorem.

Now replace $z$ by $z^2(1+\epsilon_n)/2$ in the estimate. This results in the moment-generating version of the standard CLT.

Finally, use the fourier transform to go from the moment-generating version of the standard CLT to the convergence of $E((S_n-n)^+/\sqrt{n})$.

Answer 3

I include a sketch of the "elementary" proof of the local central limit theorem required to prove Stirling's formula using this method. PDF version here.

---

---

---

I assume the reader already knows/[believes in] the proof of the standard CLT. See e.g. TT's 275A course notes using Lindeburg exchange. It is a substantial proof, but I think presentable in roughly 1-1.5 hours of lecture, so not too long. Not much prerequisite material is needed, besides the basic definitions of probability theory, the "weak$^*$ definition" of convergence in distribution (Def. 2 in TT, which is equivalent to the c.d.f. definition by Prop. 4), and Taylor expansion.

The local (discrete) CLT proof in TT Notes 5 is trickier, using Fourier/characteristic function methods. I would like to use this article to explain Burgess Davis' elementary proof of a local discrete CLT, in the context of a simple application of it towards a probabilistic proof of Stirling's formula for the factorial. $\newcommand{\V}[1]{\text{Var}\left[#1\right]} \newcommand{\bb}{\mathbb} \newcommand{\defeq}{:=} \newcommand{\d}{\;\textrm{d}} \newcommand{\lilo}{o} \newcommand{\eps}{\epsilon}\newcommand{\av}[1]{\left|{#1}\right|} \newcommand{\xto}{\xrightarrow}$

Proof of Stirling from local Central Limit Theorem (lCLT)

Let $X_1, X_2,\ldots$ be i.i.d. Poisson(1) r.v.'s (i.e. valued in $\bb N$, where $\bb P[X=k] = f_X(k) \defeq \frac{e^{-1}}{k!}$). Recall (or calculate for yourself) $\bb E[X_i] = 1, \V{X_i}=1$.

Let $S_n \defeq X_1 + \ldots + X_n$. A probabilistic argument tells us that $S_n \sim \text{Poisson}(n)$, so in ptclr., $\bb E[S_n] = n, \V{S_n}=n$.

CLT tells us $\frac{S_n-n}{\sqrt n} \to_d \text{Normal}(0,1)$, or equivalently (for all $t\in \bb R$) $$\lim_{n\to\infty} \bb P\left[ \frac{S_n-n}{\sqrt n} \leq t\right] = \Phi(t) \defeq \int_{-\infty}^t \phi(x) \defeq \int_{-\infty}^t \frac{1}{\sqrt{2\pi}} e^{-x^2/2} \d x.$$ For the sake of relating to the writeups of McDonald and/or Davis, I introduce the notation $a_n \defeq n$, $b_n \defeq \sqrt n$ so that $\frac{S_n-a_n}{b_n} \defeq \frac{S_n-n}{\sqrt n}$.

Intuitively, this says that $$\bb P[S_n \leq \underbrace{b_n t + a_n}_{\defeq k}] \approx \Phi(\underbrace{t}_{=\frac{k-a_n}{b_n}})$$ so taking the "derivative in $k$" (use FTC2 on $\Phi$) heuristically gives $\bb P[S_n = k] \approx \frac{1}{b_n} \phi(\frac{k-a_n}{b_n})$.

Indeed, we $\color{red}{\text{aim to show}}$ the following "local discrete CLT" $$\sup_{k\in \bb Z} \av{\underbrace{b_n}_{\defeq \sqrt n} \cdot \; \underbrace{\bb P[S_n=k]}_{=\frac{e^{-n}n^k}{k!}} \; - \; \phi \Big(\underbrace{\frac{k-a_n}{b_n}}_{\defeq \frac{k-n}{\sqrt n}}\Big)} \xto{n\to\infty} 0$$ In particular, taking $k=n$, this implies that $$\av{\sqrt n \cdot \frac{e^{-n} n^n}{n!} - \frac{1}{\sqrt{2\pi}}} \xto{n\to\infty} 0,$$ or equivalently $\frac{\sqrt{2\pi n} \cdot (\frac ne)^n}{n!} = 1+\lilo_{n\to\infty}(1) \iff n! = \sqrt{2\pi n} \cdot (\frac ne)^n \cdot (1+ \lilo_{n\to\infty}(1))$.

Now, we prove the above "local discrete CLT", following McDonald.

Define variation "norm" on functions $f: \bb Z \to [0,\infty)$ by $\|f\|_V \defeq \sup_{k\in \bb Z} |f(k)-f(k+1)|$. Define $\underline{f}$ as the function $k \mapsto \min(f(k), f(k+1))$ --- observe that $\underline{f}\leq f$ pointwise on $\bb Z$. The "$f$" we deal with will be p.m.f.'s of our random variables $X_i$ ($f_{X_i}$ doesn't depend on $i$ because we assumed i.i.d.), $S_n$.

Lemma: Variation norm condition to discrete LCLT [Lemma 3.1 in McDonald, Lemma 1 in Davis]

If $a_n, b_n\geq 0$ are as above ($\frac{S_n-a_n}{b_n}$ satisfy CLT, etc.) and $b_n \to \infty$ (as $n\to\infty$), and $$\sup_n \left[b_n^2 \cdot \|f_{S_n}\|_V\right] < \infty, \quad {(\text{e.g. } \sup_n \big[n \cdot \sup_k |\tfrac{e^{-n}n^k}{k!} - \tfrac{e^{-n}n^{k+1}}{(k+1)!}|\big] < \infty)}$$ then the LCLT result holds: $$\sup_{k\in \bb Z} \av{b_n \cdot\bb P[S_n=k] \; - \; \phi \Big(\frac{k-a_n}{b_n}\Big)} \xto{n\to\infty} 0$$

Proof:

define $\tilde \phi(\frac{k-a_n}{b_n}) \defeq b_n \cdot \big(\Phi(\frac{(k+1)-a_n}{b_n}) - \Phi(\frac{k-a_n}{b_n}) \big)$ (named $\tilde\phi$ because the RHS is a Newton quotient of $\Phi$, hence close to $\Phi'=\phi$).

In Davis' notation, ${e_{k,n} \defeq \frac{k-a_n}{b_n}, \tilde \phi(\frac{k-a_n}{b_n}) = b_n \cdot h_{k,n}}$.

Davis (1) notes $$\sup_{k\in \bb Z} |\tilde \phi(\tfrac{k-a_n}{b_n}) - \phi(\tfrac{k-a_n}{b_n})| \xto{n\to\infty} 0.$$ (this only needs $b_n\to\infty$; proof by mean value theorem or FTC).

Consider deviation $\Delta_n(k) \defeq b_n \cdot \bb P[S_n=k] - \tilde \phi(\frac{k-a_n}{b_n})$. Recall we $\color{red}{\text{WTS $\|\Delta_n(k)\|_{\ell^\infty_k(\bb Z)} \xto{n\to\infty} 0$}}$

---

We consider successive difference $|\Delta_n(k+1) - \Delta_n(k)|$. Triangle inequality splits into 2 terms, and $\|\bullet\|_V$ assumption bounds the $\bb P[\bullet]$ term, but we need something for the $\tilde\phi$ term.

For any fixed $k\in \bb Z$, Davis' (6) claims $$ \sup_{n\in \bb Z} \left[b_n \cdot |\tilde\phi(\tfrac{(k+1)-a_n}{b_n}) - \tilde\phi(\tfrac{k-a_n}{b_n})|\right] < \infty.$$ (intuition is that it is like Newton quotient for $\phi$). More rigorously, it is 2nd order Newton quotient for $\Phi$, so 2nd order MVT (or FTC) will give a bound in terms of $\Phi''$, which is uniformly bounded on $\bb R$.

Putting all the bounds of this textblock together, we get $$b_n \cdot\; |\Delta_n(k+1) - \Delta_n(k)| \leq b_n^2 \|f_{S_n}\|_V + b_n |\tilde\phi(\tfrac{(k+1)-a_n}{b_n}) - \tilde\phi(\tfrac{k-a_n}{b_n})|,$$ which is uniformly (in $n\in \bb Z$) bounded $\leq C < \infty$.

Thus, $|\Delta_n(j) - \Delta_n(k)| \leq C |j-k|\cdot \frac{1}{b_n}$, for all $n,j,k\in \bb Z$ (cf. Davis' (7)).

---

Fix $\eps > 0$. Fix $k\in \bb Z$.

We show that $|\Delta_n(k)|> \epsilon$ can happen for only finitely many $n$.

Well, suppose we do have an $n^\star$ s.t. $\Delta_{n^\star}(k)> \epsilon$ (the $<-\eps$ case is handled similarly). Then by Davis' (7), $\Delta_{n^\star}(j)> \frac \eps 2$ for all $j$ s.t. $C |j-k| \leq \frac \eps 2 \cdot b_{n^\star}$. Define $J$ to be this range of $j$. The number of such $j$, i.e. $|J|$, is $\geq \frac{\eps}{2C} \cdot b_{n^\star}$.

Summing $\sum_{j\in J} \Delta_{n^\star}(j)$ over this (size $\sim b_{n^\star}$ ) range of $j$ yields a value $> (\frac{\eps}{2C} b_{n^\star}) \cdot (\frac \eps 2)$.

But $$\sum_{j\in J} \Delta_n(j) = b_n \left(\sum_{j\in J} \bb P[S_n=j] - \sum_{j\in J} \tfrac{1}{b_n} \tilde \phi \big(\tfrac{j-a_n}{b_n}\big)\right)$$ $J$ is long enough ($|J| \sim b_n$ ) that the rightmost sum is essentially a Riemann sum converging to $\int_\text{an interval} \phi(x) \d x$, so within the regime of CLT.

So, if there is an $n^\star$ as above, then (dividing by $b_{n^\star}$ everywhere)

$$\left(\sum_{j\in J} \bb P[S_{n^\star}=j] - \sum_{j\in J} \tfrac{1}{b_{n^\star}} \tilde \phi \big(\tfrac{j-a_{n^\star}}{b_{n^\star}}\big)\right) > \frac{\eps^2}{4C} > 0$$ which CLT tells us can only be true for finitely many $n^\star$. $\blacksquare$

Ok, now just need some way to check the weird $\|\bullet \|_V$ assumption to apply the Lemma to our Stirling example.

McDonald pg. 10-11 takes it from here, using the idea of Bernoulli decomposition, and one simple Lemma on successive difference of normalized binomial coefficients $\binom nk / 2^n$ (on the same row of Pascal's triangle).