Smoothing and differentiation

08 Oct 2017

The problem addressed in this post came up during a discussion with Samuele Tosatto. The motivation was to understand the often used reparameterization trick (presented here with Gaussians for concreteness)

$$\nabla_\theta \mathbb{E}_{x \sim N(0, 1)} \left[ f(\theta + \sigma x) \right] = \mathbb{E}_{x \sim N(0, 1)} \left[ \frac{x}{\sigma} f(\theta + \sigma x) \right]$$

from the point of view of smoothing and differentiation operators. A particularly curious observation can be made if the statement above is written in the language of operators, using $S$ for smoothing (with the Gaussian kernel) and $D$ for differentiation with respect to $\theta$. Exploiting commutativity of operators $S$ and $D$, one obtains

$$S \left[ D f_\theta \right] = S \left[ \frac{x}{\sigma} f_\theta \right],$$

which says that differentiation acts by multiplication under the smoothing operator—a property somewhat reminiscent of the Fourier transform. This and other related ideas are explored in a bit more depth in this pdf attachment, although many open questions remain.