8

Binary entropy is given by $$H_{\mathrm b}(p) = -p \log_2 p - (1 - p) \log_2 (1 - p), \hspace{6 mm} p \le \frac{1}{2}$$ How can I prove that $$H_{\mathrm b}(p) \le 2 \sqrt{p(1-p)}$$

Michael Hardy
  • 1

3 Answers3

7

Given $I=(0,1)$, for any $x\in I$ let $g(x)=-x\log x$ and $f(x)=g(x)+g(1-x)$.

Obviously $f\in C^{\infty}(I)$, $\,f(x)=f(1-x)$, $\,f>0$ and $$ \lim_{x\to 0^+}f(x)=\lim_{x\to 1^-}f(x) = 0. $$ We have $f'(x)=\log(1-x)-\log(x)$, hence:

$$\begin{eqnarray*} \frac{d}{dx}\frac{f(x)^2}{x(1-x)}&=&f(x)\cdot\frac{-x\log x+(1-x)\log(1-x)}{x^2(1-x)^2}\\&=&\frac{g(x)^2-g(1-x)^2}{x^2(1-x^2)}.\tag{1}\end{eqnarray*}$$ Since $g(x)>g(1-x)$ over $J=\left(0,\frac{1}{2}\right)$ (the proof of this fact comes later), the function $\frac{f(x)^2}{x(1-x)}$is increasing over $J$, hence:

$$ f(x)\leq 2\cdot f\left(\frac{1}{2}\right)\cdot\sqrt{x(1-x)}=\color{blue}{\left(2\log 2\right)}\sqrt{x(1-x)} \tag{2}$$

follows. Back to the missing part: obviously $g(x)-g(1-x)$ vanishes at $0,\frac{1}{2},1$. In order to prove that these points are the only zeros, it is enough to check that $h(x)=g(x)-g(1-x)$ is concave over $J$. Pretty easy:

$$ \frac{d^2}{dx^2}h(x) = \frac{2x-1}{x(1-x)}.\tag{3} $$ The proof is finished by noticing that $f(p)=\log(2)\cdot H_{\mathrm b}(p)$.

An very tight approximation for the binary entropy function is given by:

$$ H_{\mathrm{b}}(p) \approx \left(4p(1-p)\right)^{\frac{3}{4}}.\tag{4}$$

It does not hold as an upper bound or a lower bound, the the absolute value of the difference between the RHS and the LHS is always less than $8\cdot 10^{-3}$.

Jack D'Aurizio
  • 372,937
  • 42
  • 421
  • 888
  • Can you please explain how you got step $(2)$ from $(1)$? –  Sep 12 '15 at 14:43
  • 2
    @guest123456: since $\frac{f(x)^2}{x(1-x)}$ is increasing over $\left(0,\frac{1}{2}\right)$, we have: $$ \frac{f(x)^2}{x(1-x)}\leq \frac{f(1/2)^2}{1/4} = 4,f(1/2)^2 = 4 \log^2 2$$ and rearranging $f(x) \leq 2\log(2)\sqrt{x(1-x)}$ follows. – Jack D'Aurizio Sep 12 '15 at 14:52
  • Thank you very much sir, awesome explanation. It was too difficult for me to solve in examination hall :) –  Sep 12 '15 at 14:54
  • 2
    The error is that the problem specifies binary logarithm, so there should not be a $\ln(2)$ in your answer. Meaning your comment on the question is also not correct. – user21820 Sep 12 '15 at 14:54
  • 1
    @user21820: thank you, I removed it and I am fixing the answer. Anyway, the inequality about my $f(x)$ is right. We just have to scale it by a factor $\log(2)$ to prove the wanted bound on the binary entropy. – Jack D'Aurizio Sep 12 '15 at 14:59
  • Where does the approximation $(4)$ come from? – Martin R Sep 15 '21 at 14:51
  • @MartinR: just trial and error. H is symmetric and concave, and for the exponent 3/4 we have a very good fit. – Jack D'Aurizio Sep 15 '21 at 15:13
  • If you ask systematically for an upper bound of the form $H_b(x) \leq c \cdot (4 x(1-x))^{q}$ with some constant $c$, and then ask for the largest exponent $q$ where such bound holds, you get $q = 1 / \ln(4) \simeq 0.721$ (with $c=1$), so the fit $q = 0.75$ is close. Reference: "Bounds for entropy and divergence for distributions over a two-element set" by Topsøe, Flemming. Pdf: http://www.emis.de/journals/JIPAM/images/044_00_JIPAM/044_00.pdf – Andreas Sep 16 '21 at 10:19
2

You may use the inequality $\ln(x)\le \sqrt{x-1}$ for any $x\ge 1$. Precisely speaking, note that \begin{equation} \begin{aligned} H(p)&=p\ln\frac{1}{p} +(1-p)\ln\frac{1}{1-p} \\ &\le p\sqrt{\frac{1}{p}-1} +(1-p)\sqrt{\frac{1}{1-p}-1}\\ &= 2\sqrt{p(1-p)} \end{aligned} \end{equation} Here the logarithm is to base natural exp..

Chun SO
  • 59
0

This inequality is actually not trivial. Here is a proof sketch. We know that there are three equality cases, so to get rid of the ones at the ends we take $\frac{H(p)}{2\sqrt{p(1-p)}}$. By differentiating with respect to $p$ and slowly analyzing the result (elementary algebra plus the inequality $x-\frac{1}{2}x^2 \le \ln(1+x) \le x$ for $x \ge 0$) it is possible to show that it is decreasing for $p > \frac{1}{2}$, which by symmetry means that it is maximum when $p = \frac{1}{2}$. It is not that hard but not elegant.

user21820
  • 61,632
  • Seems a lot of calculation, and as you said not elegant. But I am OK with it. Actually this problem was given in my Information Theory exam today and was marked as a difficult problem. So maybe there isn't any simple and elegant proof. –  Sep 12 '15 at 13:21
  • @guest123456: I came up with this inequality last year when trying to find a reasonable bound for the binary entropy function, and spent a good deal of time finding a way to prove it. Should be unrelated to your exam question haha.. – user21820 Sep 12 '15 at 13:27
  • This is not an answer but just the sketch of it. Details are not so hard to be provided. – Jack D'Aurizio Sep 12 '15 at 14:26
  • @JackD'Aurizio: Sure it's not meant to be a full answer since I don't find joy in posting what I consider inelegant computations. =) – user21820 Sep 12 '15 at 14:53
  • @user21820: it is an actual idea that is missing (slowly analyzing the result means nothing without details: what do you want to exploit? Convexity? Log-convexity? Cauchy-Schwarz? Something else?) not just computations. It is a perfect answer under the syntactical point of view, but quite an empty answer under the semantical point of view. – Jack D'Aurizio Sep 12 '15 at 14:57
  • @JackD'Aurizio: Okay I specified what is needed. The problem is that I do not have a typed version and I'm not feeling so hardworking to type it all out today but really I didn't use anything except that basic inequality and $4(1-\ln(2)) > 1$. – user21820 Sep 12 '15 at 15:10
  • 1
    @user21820: ok, I am fine with that, I'll stop nitpicking :) – Jack D'Aurizio Sep 12 '15 at 15:12
  • updated answer with sketch of proof, along the lines of the original answer, check if you want – Nikos M. Sep 14 '15 at 21:00