Many faces of entropy

With this definition, the KL divergence $D(\pi \Vert \mu )$ from $\mu$ to $\pi$ equals minus entropy of $\pi$ under $\mu$,

$$\begin{array}{}\text{(1)}& D(\pi \Vert \mu )=-H_{\mu }(\pi ).\end{array}$$

The differential entropy $h(\pi )$ is the entropy of $\pi$ under the Lebesgue measure $\lambda$,

The joint entropy $h(X,Y)$ equals the entropy of the joint distribution ${\mu }_{XY}$ over $X$ and $Y$ under the Lebesgue measure,

The mutual information $I(X;Y)$ equals minus entropy of the joint distribution ${\mu }_{XY}$ under the product measure ${\mu }_X\times {\mu }_Y$,

The conditional entropy $h(Y|X)$ equals the entropy of ${\mu }_{XY}$ under the product measure ${\mu }_X\times \lambda$,

Such unification was actually the motivation of Kullback and Leibler’s seminal paper on information and sufficiency.

*Variation of measure¶

Formula $\text{(1)}$ shows that we can as well take the KL as the basis for expressing all other entropies. I find it easier to think in terms of distance (or divergence) rather than entropy, therefore I prefer to work with the KL. Let’s denote $KL(\pi \Vert \mu )$ by $D_{\mu }(\pi )$ to use the same notation as for the entropy $H_{\mu }(\pi )$, then $$D_{\mu }(\pi )={\mathbb{E}}_{\mu }\left[f\left(\frac{d\pi }{d\mu }\right)\right]$$ with the generator function $f(x)=x\log x-(x-1)$ corresponding to the KL divergence (see f-divergence for details).

Let us see how $D_{\mu }(\pi )$ changes when we vary its arguments. Taking partial functional derivatives, we find $$\begin{array}{rl}\frac{\delta D_{\mu }(\pi )}{\delta \pi }(dx)& =f^{\prime }\left(\frac{\pi (dx)}{\mu (dx)}\right),\\ \frac{\delta D_{\mu }(\pi )}{\delta \mu }(dx)& =f\left(\frac{\pi (dx)}{\mu (dx)}\right)-\frac{\pi (dx)}{\mu (dx)}{f}^{\prime }\left(\frac{\pi (dx)}{\mu (dx)}\right).\end{array}$$ Note that since the arguments of $D$ are measures, its partial derivatives take the same argument as measures do, i.e., they take measurable subsets (which I denoted by $dx$) as input.

If we denote the Radon-Nikodym derivative by ${\pi }_{\mu }$, we can also write $$\begin{array}{rl}\frac{\delta D_{\mu }(\pi )}{\delta {\pi }_{\mu }}(x)& =f^{\prime }({\pi }_{\mu }(x)),\\ \frac{\delta D_{\mu }(\pi )}{\delta {\mu }_{\pi }}(x)& =f({\pi }_{\mu }(x))-{\pi }_{\mu }(x)f^{\prime }({\pi }_{\mu }(x)).\end{array}$$ For $f$ corresponding to the KL divergence, these formulas are particularly simple $$\begin{array}{rl}\frac{\delta D_{\mu }(\pi )}{\delta {\pi }_{\mu }}(x)& =\log ({\pi }_{\mu }(x)),\\ \frac{\delta D_{\mu }(\pi )}{\delta {\mu }_{\pi }}(x)& =-\frac{1}{{\mu }_{\pi }(x)}+1.\end{array}$$ The fact that the derivative with respect to ${\pi }_{\mu }$ equals the logarithm is the defining property of the KL divergence. Basically, the KL generator $f$ is the antiderivative of the logarithm, and that is where all the magic stems from. If you minimize a KL (or maximize entropy), you end up with a logarithm after differentiating the objective, which leads you to an exponential family. And all because $f^{\prime }=\log$. However, no such beautiful magic emerges from the other partial derivative.