0

Differential entropy of a single random variable $X$ is

$$h(X) = -\int f(x_i) \ln f(x_i) \hspace{1mm} dx_i$$ The weighted differential entropy of $N$ different variables, each with their own differential entropy and uniquely assigned weight, contained in vectors $\boldsymbol w$ and $\boldsymbol h$, is the weighted sum $$\boldsymbol w ^\top \boldsymbol{h} =-\Sigma_{i=1}^N w_i \int f(x_i) \ln f(x_i) \hspace{1mm} dx_i$$

In the case of a single Gaussian variable, differential entropy has a closed-form analytical solution, being a function of its variance, $\sigma_i^2$: $$h = \frac{1}{2} \ln\left(2\pi e\sigma_i^2\right)$$

What then is a simplified form of weighted Gaussian entropy? Here is a start to my guess, which I am asking help to simplify, but at the same time open to analytical solutions for that compactly express the same quantity using multivariate input instead, i.e. the covariance matrix rather than individual $\sigma_i$'s. \begin{align}&= \Sigma_{i=1}^N \left\{ w_i \times \frac{1}{2} \ln\left(2\pi e\sigma_i^2\right)\right\}_{i=1}^N\\ &= ?\end{align}

develarist
  • 1,604
  • 1
    you should write $f_{i}(x)$ rather than $f(x_{i})$. And also, there is no exponential in the entropy. – Tobsn Nov 13 '20 at 17:22
  • Apart from that you can rewrite it e.g. as $\ln\left(\prod_{i=1}^{n}(2\pi\sigma_{i})^{w_{i}/2}\right)$ which is not looking more pleasent, though. – Tobsn Nov 13 '20 at 17:30
  • or if that's more pleasing as $\frac{1}{2}\left(\ln(2\pi)|w|+\sum w_{i}\ln(\sigma_{i})\right)$ where $|w|$ denotes the sum of the weights. – Tobsn Nov 13 '20 at 17:39
  • Norwich (2003) shows that there is an exponential in Gaussian entropy. I just made the mistake of exponentiating $\sigma$ when in fact it only has to be multiplied by $e$: https://www.biopsychology.org/norwich/isp/chap8.pdf – develarist Nov 14 '20 at 00:31
  • Then you get for instance (since I also forgot the square on sigma) $\frac{|w|}{2}(\ln(2\pi)+1)+\sum w_{i}\ln(\sigma_{i})$. – Tobsn Nov 14 '20 at 07:26
  • 1
    I don't understand the question. You can certainly rewrite it in vectorized form as $\frac{1}{2}w^T \ln(2\pi e\sigma^2)$ where the log is applied elementwise. Alternatively, you can write $\frac{1}{2}v^T \ln(2\pi e\Sigma)v$ where $v=\sqrt{w}$ and $\Sigma$ is a diagonal matrix with $\sigma^2$ on the diagonal, assuming the weights are nonnegative. The last one extends to the case where $\Sigma$ is not diagonal, but does it make sense? Maybe. I don't have time to find out. – Kiuhnm Nov 14 '20 at 21:18

0 Answers0