Many Small Proofs and Derivations

Last updated: March 10, 2023

Two Proofs About the Kullback–Leibler Divergence

This post shows two different ways to prove that \(D_\mathrm{KL}(p || q) \ge 0\) given distributions \(p(x), q(x)\).

Three Discrete Sampling Methods

This post describes and implements three methods which can be used to sample from any discrete probability distribution.

Why Value Averaging (Probably) Doesn't Work

Value averaging (VA) is a proposed improvement over dollar cost averaging (DCA). Contrary to DCA, which accumulates a fixed cost each period regardless of the underlying market price, VA attempts to accumulate a fixed value each period. In practice this means VA accumulates more shares when the market price is lower than some target and accumulates less, or even sells off, shares when the price is higher.

Probabilistic Numerics

I recently came across the topic of probabilistic numerics. The essence of the idea is actually very simple, as I’ll try to illustrate in this post.

Deriving the Normal Equations

Suppose we are given a design matrix \(\mathbf{X} \in \mathbb{R}^{N \times K}\) and targets vector \(\mathbf{y} \in \mathbb{R}^N\). We want to find parameter vector \(\mathbf{w} \in \mathbb{R}^K\) which minimizes the residual sum of squares, i.e.

Linear Sandwich

Creating a Linear sandwich by stacking a bunch of Linear layers on top of each other just results in another linear transformation. Below, we show that this is indeed the case.

Deriving the Evidence Lower Bound (ELBO)

Suppose we have a joint distribution \(p(\mathbf{x}, \mathbf{z})\) where \(\mathbf{z}\) are latent variables. Then the Evidence Lower Bound (ELBO) for \(p(\mathbf{x})\) can be derived as:

Mental spew

“What is the meaning of life?” the interviewer asked matter-of-factly.

Caught off guard, his mind raced. Rapidly burning through myriad possibilities, he grasped for something insightful to say. Almost immediately he became acutely aware of each passing moment. He sat in silence for some time, mining his consciousness while the deafening silence grew heavier and heavier. No longer able to bear it, he spewed out a half-constructed thought, “Well you see, …”