# The $\alpha$-Gaussian

13 Oct 2018It would be a nice story to tell if the maximum $\alpha$-entropy distribution turned out to be the $\alpha$-Gaussian. But it wouldn’t deserve a post on its own, would it?

## Motivation

Do you remember the post where we computed the $\alpha$-divergence between two Gaussian distributions on the real line? Formula (2) derived there had a weird property to only apply if a certain condition (3) on variances is satisfied. What it means, for example, is that one cannot freely pick any two Gaussians and ask what the $2$-divergence between them is, because the $2$-divergence is only defined if the variance of the Gaussian in the numerator is smaller than twice the variance of the Gaussian in the denominator. Back then I could not explain this strange phenomenon; this post is an attempt to gain a better understanding of it.

### The $\alpha$-divergence

The $\alpha$-divergence is a generalization of the KL divergence
obtained by a smooth deformation of the logarithm function.
More concretely, let’s define a new function
$\log_\alpha \colon \mathbb{R} \to \mathbb{R}$ where $\alpha \in \mathbb{R}$,
called the *$\alpha$-logarithm*,
\begin{equation}
\log_{\alpha}(x):=\frac{x^{\alpha-1}-1}{\alpha-1}.
\end{equation}
In the limit $\alpha \to 1$,
this function coincides with the familiar natural logarithm.
Note that $\alpha$ is just a real-valued parameter and not the base;
no confusion with the base should arise
because only the natural logarithm is used here.

The *$\alpha$-divergence* is defined in a similar way as the KL divergence,
\begin{equation}
\text{KL}_{\alpha}(p\|q) := \frac{1}{\alpha} \int_{-\infty}^{\infty}
p(x)\log_{\alpha}\left(\frac{p(x)}{q(x)}\right)dx,
\end{equation}
with the natural logarithm $\log(x)$ replaced by $\alpha^{-1}\log_\alpha(x)$.
For more details on the definition of the $\alpha$-divergence,
see the great article by Andrzej Cichocki and Shun-ichi Amari in Entropy 2010
Families of Alpha- Beta- and Gamma- Divergences:
Flexible and Robust Measures of Similarities.

### The $\alpha$-exponential function

The KL divergence works so beautifully with exponential families because
the logarithm is the inverse of the exponential function.
However, if the logarithm gets replaced by the $\alpha$-logarithm,
such forward-inverse relation no longer holds.
That is why calculating the $\alpha$-divergence between Gaussians
is so cumbersome.
Thus, we are naturally led to introducing the inverse of the $\alpha$-logarithm,
called the *$\alpha$-exponential function*,
\begin{equation}
\exp_{\alpha}(y) := \sqrt[\alpha-1]{1+(\alpha-1)y}.
\end{equation}
The key property which is retained can be stated as
$\exp_{\alpha}(\log_{\alpha}(x)) = x.$
On the other hand, other nice properties are lost,
e.g., $\exp_{\alpha}(x+y) \neq \exp_{\alpha}(x) \exp_{\alpha}(y)$.
Moreover, the domain of the $\alpha$-exponential function depends on $\alpha$;
namely, condition $1+(\alpha-1)y\geq0$ must be fulfilled
for $\exp_{\alpha}(y)$ to return a real number.
The domain restriction property can be seen as a weakness
but also as a strength: it enables definition of
finitely-supported $\alpha$-exponential families.

## The $\alpha$-Gaussian distribution

By replacing the exponential function
in the probability density of the normal distribution
with the $\alpha$-exponential function,
we obtain the *$\alpha$-Gaussian probability density function*
\begin{equation}
N_{\alpha}\left(x|\mu,\sigma^{2}\right):=\frac{1}{c_{\alpha}}
\text{e}_{\alpha}^{-\frac{1}{2}\frac{\left(x-\mu\right)^{2}}{\sigma^{2}}}
\end{equation}
where the normalization constant is given by
\begin{equation}
c_{\alpha}:=\begin{cases}
\sqrt{\frac{2\pi\sigma^{2}}{1-\alpha}}
\frac{\Gamma\left(\frac{1}{2}\frac{1+\alpha}{1-\alpha}\right)}
{\Gamma\left(\frac{1}{1-\alpha}\right)},
& \alpha\in(-1,1],\quad x\in\mathbb{R}, \newline
\frac{\sqrt{2\pi\sigma^{2}}}{\left(\alpha-1\right)^{\frac{3}{2}}}
\frac{\Gamma\left(\frac{1}{\alpha-1}\right)}
{\Gamma\left(\frac{1}{2}+\frac{\alpha}{\alpha-1}\right)},
& \alpha>1,\quad x\in\left(\mu-\sqrt{\frac{2}{\alpha-1}},
\mu+\sqrt{\frac{2}{\alpha-1}}\right).
\end{cases}
\end{equation}
Curiously, the support of the $\alpha$-Gaussian density depends on $\alpha$:
for $\alpha > 1$, the support is finite;
for $\alpha \in (-1, 1]$, it is infinite;
and for $\alpha \leq -1$, the support is still infinite but the density
cannot be normalized.

An intuitive way to understand the effect of $\alpha$
is to start at $\alpha \to \infty$
and see how decreasing $\alpha$ towards $\alpha \to -\infty$
changes the probability density function.
For very large $\alpha \to \infty$,
the density $N_\alpha\left(x|\mu,\sigma^2\right)$
is entirely concentrated around the mean $\mu$;
the variance $\sigma^2$ controls the shape of the density
but does not affect its support.
When we decrease $\alpha$ towards $\alpha = 1$, the density starts spreading
out, growing towards the normal density, which it matches in the limit
$\alpha \to 1$.
If we keep decreasing $\alpha$, the tails of the $\alpha$-Gaussian density
function become heavier, enveloping the Gaussian tails;
at $\alpha \to -1$, the tails become so heavy that the probability mass
becomes infinite.
From that point on, decreasing $\alpha$ further makes the function look
more and more like a constant function,
matching it in the limit $\alpha \to -\infty$.
Thus, *by changing $\alpha$ we control the tails of the distribution.*

For completeness, we can obtain the cumulative distribution function \begin{equation} F_{\alpha}(x):=\frac{c_{\alpha}}{2}+x\,_{2}F_{1} \left( \frac{1}{2},\frac{1}{1-\alpha};\frac{3}{2}; -\frac{\left(1-\alpha\right)}{2}x^{2} \right) \end{equation} where $\,_{2}F_{1}$ is the hypergeometric function. One could use it, for example, to sample from the $\alpha$-Gaussian distribution via inverse transform sampling; however, inverting it is not straightforward.

## $\text{KL}_\alpha$ between $\alpha$-Gaussians

The motivation behind introducing the $\alpha$-Gaussian distribution was to simplify the computation of the $\alpha$-divergence. Unfortunately, since the $\alpha$-exponential function is not a homomorphism between addition and multiplication, one cannot nicely transform the ratio of densities into a single exponential which could then be removed by a logarithm, the way it is done in the KL divergence. Therefore, computing $\text{KL}_\alpha$ between $\alpha$-Gaussians is actually not easier than computing $\text{KL}_\alpha$ between usual Gaussians. Moreover, even computing the normal $\text{KL}$ divergence between $\alpha$-Gaussians analytically appears to be infeasible.

## Max $\alpha$-entropy distribution

Getting back to the question posed in the very beginning. Is the $\alpha$-Gaussian the maximum $\alpha$-entropy distribution, similar to how the Gaussian distribution is the maximum entropy distribution? The answer is unfortunately no, and furthermore, it seems to be intractable to analytically derive any $\alpha$-exponential family from the maximum $\alpha$-entropy principle. The reason lies again in the fact that even though we can write down the density in the form $p_\alpha(x) = \text{e}_{\alpha}^{-\eta-\lambda x-\nu x^{2}}$, we can neither normalize it nor evaluate the moments because $\alpha$-exponentiation doesn’t transform addition into multiplication.

## Conclusion

Alas, the $\alpha$-Gaussian distribution didn’t help us to simplify the computation of the $\alpha$-divergence. However, we have seen that by varying $\alpha$ one can generated either heavy-tailed distributions or light-tailed finitely-supported distribution. The fact that the $2$-Gaussian is finitely-supported gives a hint on why the $2$-KL between Gaussians is finite only on a subset of pairs of distributions but does not provide a direct explanation. Estimating the parameters of the $\alpha$-Gaussian distribution from data using maximum likelihood is probably intractable because the $\alpha$-logarithm does not turn a product into a sum. Sampling also does not appear to be straightforward. With all these disadvantages, the $\alpha$-Gaussian distribution is unlikely to become a useful tool in the statistical toolbox, but is rather doomed to remain a bizarre animal in the information geometry zoo.