What are differences and relationship between shannon entropy and fisher information?

Question

When I first got into information theory, information was measured or based on shannon entropy or in other words, most books I read before were talked about shannon entropy. Today someone told me there is another information called fisher information. I got confused a lot. I tried to google them. Here are links, fisher information: https://en.wikipedia.org/wiki/Fisher_information and shannon entropy goes here https://en.wikipedia.org/wiki/Entropy_(information_theory).

What are differences and relationship between shannon entropy and fisher information? Why do two kinds of information exist there?

Currently, my idea is that it seems fisher information is a statistical view while shannon entropy goes probability view.

Any comments or anwsers are welcome. Thanks.

Answer 1

Fisher information is related to the asymptotic variability of a maximum likelihood estimator. The idea being that higher Fisher Information is associated with lower estimation error.

Shannon Information is totally different, and refers to the content of the message or distribution, not its variability. Higher entropy distributions are assumed to convey more information because must be transmitted with more bits.

However, there is a relationship between Fisher Information and Relative Entropy/KL Divergence, as discussed on Wikipedia.

Answer 2

The Fisher information is related to differential entropy through a normal perturbation, and in that case, entropy can be considered as an integral of Fisher information.

---

The following theorem (de Bruijn’s identity) is from Section 17.7 of Elements of Information Theory.

Let $X$ be any random variable with a finite variance with a density f (x). Let $Z \sim N(0,1)$ be an independent normally distributed random variable with zero mean and unit variance. Then $$ \frac{\partial}{\partial t}h_e(X + \sqrt{t}Z) =\frac{1}{2}J(X +\sqrt{t}Z)$$ where $h_e$ is the differential entropy with base e, and $J(Y)= \int_{-\infty}^{+\infty}f(y) (\frac{\frac{\partial}{\partial y}f(y)}{f(y)})^2dy$ denotes the Fisher information at $\theta=0$ for the location family of densities $f_{\theta}(y)=f(y-\theta)$.
In particular, if the limit exists as $t \to 0$, we have $$ \frac{\partial}{\partial t}h_e(X + \sqrt{t}Z)|_{t=0} =\frac{1}{2}J(X)$$

---

-The relationship is also discussed in this paper: Entropy and the Central Limit Theorem

-The following answer may also be relevant: Fisher information and the "surface area of the typical set"

Answer 3

Ref: "Introduction to Mathematical Statistics" by Hogg, McKean, Craig.

Consider a sample ${ X _1, \ldots, X _n }$ ${ \sim f(x \vert \theta) . }$

Note that the average log likelihood variable

$${ \ell (\theta, \mathbf{X}) = \frac{1}{n} \sum _{i = 1} ^{n} \log f(X _i \vert \theta) }$$

appears in maximum likelihood estimation.

Note that the derivative

$${ \partial _{\theta} \ell(\theta, \mathbf{X}) = \frac{1}{n} \sum _{i = 1} ^{n} \frac{\partial _{\theta} f(X _i \vert \theta)}{f(X _i \vert \theta) } }$$

is zero at ${ \theta }$ equal to the maximum likelihood estimator ${ \hat{\theta}(X _1, \ldots, X _n) . }$

Hence the derivative appears in the proof of asymptotic normality of the maximum likelihood estimator.

Q) What are the distributions of ${ \ell (\theta, \mathbf{X}) }$ and ${ \partial _{\theta} \ell (\theta, \mathbf{X}) }$?

Note that

$${ \ell (\theta, \mathbf{X}) }$$

is approximately a normal random variable with mean

$${ \boxed{ \mathbb{E}[\ell(\theta, \mathbf{X})] = - \text{Ent} (f( \cdot \vert \theta)) } . }$$

Note that

$${ \partial _{\theta} \ell(\theta, \mathbf{X}) }$$

is approximately a normal random variable with mean

$${ \mathbb{E} [\partial _{\theta} \ell(\theta, \mathbf{X})] = 0 }$$

and variance

$${ \text{var}(\partial _{\theta} \ell(\theta, \mathbf{X})) = \frac{1}{n} \text{var}( \partial _{\theta} \log f(X _i \vert \theta)) }$$

that is

$${ \boxed{ \text{var}(\partial _{\theta} \ell(\theta, \mathbf{X})) = \frac{1}{n} I(\theta) }. }$$

Hence both entropy and Fisher information appear naturally in the context of maximum likelihood estimation.

Remark:

Note that

$${ \text{Ent}(f(\cdot \vert \theta)) }$$

is a measure of randomness of ${ f(\cdot \vert \theta) . }$

Note that

$${ I(\theta) }$$

is a measure of how quickly ${ D _{\text{KL}} (f(\cdot \vert \theta) \, \lVert \, f(\cdot \vert \theta + h)) }$ increases with change ${ h . }$ Link to Stackexchange post: Link.