Previously, we looked at how random variables interact in multivariate scenarios, measuring their relationships in terms of direction, magnitude, and intensity, alongside a quick look at parameter estimation. In this article, we’re going to unpack a fascinating nuance: how evaluating the differences between distributions—rather than just calculating distances—acts as the building block for parameter estimation.
Measuring Distribution Disparity
When we try to identify the parameter of a distribution we cannot do so directly. We instead use an alternative distribution with known parameter to approximate the target distribution. We measure the gap or variation between these two distributions and iteratively adjust the known parameter of the reference distribution to replicate the behaviour of the target distribution and its parameter. This gives rise to the concept of measuring differences between probability measures and Kullback-Leibler (KL) Divergence is one of the most convenient methods to achieve this. It is defined as the following.
For discrete case,
$$ KL(P_{\theta_{ref}},P_{\theta_{target}})=\sum_x P_{\theta_{ref}}(x)\log\left(\frac{P_{\theta_{ref}}(x)}{P_{\theta_{target}}(x)}\right) $$
For continuous case,
$$ KL(f_{\theta_{ref}},f_{\theta_{target}})=\int_x f_{\theta_{ref}}(x)\log\left(\frac{f_{\theta_{ref}}(x)}{f_{\theta_{target}}(x)}\right)dx $$
KL divergence is used extensively in Generative models where the objective is to generate synthetic data close to the real data distributions. It is also used heavily in the Reinforcement Learning with Human Feedback (RLHF) mechanism for adjusting the output of generative models based on user feedback.
Hunting for Optimality
The ultimate objective is to estimate the unknown parameter of any distribution and technically this is termed as Maximum Likelihood Estimation. We basically start with both target and reference distributions, use KL divergence to uncover the most plausible value of the unknown parameter so that we can analyse the target distribution better.
Now in reality, we cannot have probabilities at each observation; so we start with samples from that distribution and take the average or expectation to calculate average disparity.
$$ KL(P_{\theta_{ref}},P_{\theta_{target}})=E_{\theta_{ref}}\left[\log\left(\frac{P_{\theta_{ref}}(x)}{P_{\theta_{target}}(x)}\right)\right] $$
Using logarithmic property,
$$ KL(P_{\theta_{ref}},P_{\theta_{target}})=E_{\theta_{ref}}[\log(P_{\theta_{ref}}(x))]-E_{\theta_{ref}}[\log(P_{\theta_{target}}(x))] $$
Since the first part of the above equation completely depends on reference distribution, it does not depend on target distribution and its parameter and so, is constant.
$$ KL(P_{\theta_{ref}},P_{\theta_{target}})=Constant-E_{\theta_{ref}}[\log(P_{\theta_{target}}(x))] $$
Now as per Law of Large Numbers, expectation can be approximated using sample average.
$$ KL(P_{\theta_{ref}},P_{\theta_{target}})=Constant-\frac{1}{n}\sum_{i=1}^{n}\log(P_{\theta_{target}}(x_i)) $$
We want to minimize the KL divergence. So,
$$ \min KL(P_{\theta_{ref}},P_{\theta_{target}})=\min_{\theta_{target}}\left(-\frac{1}{n}\sum_{i=1}^{n}\log(P_{\theta_{target}}(x_i))\right) $$
Since minimizing negative of a function is same as maximizing the positive of that function, it becomes
$$ \min KL(P_{\theta_{ref}},P_{\theta_{target}})=\max_{\theta_{target}}\left(\frac{1}{n}\sum_{i=1}^{n}\log(P_{\theta_{target}}(x_i))\right) $$
Now using logarithmic property,
$$ \min KL(P_{\theta_{ref}},P_{\theta_{target}})=\max_{\theta_{target}}\log\left(\prod_{i=1}^{n}P_{\theta_{target}}(x_i)\right) $$
After removing the logarithmic factor we finally reach to the maximum likelihood principle.
$$ \min KL(P_{\theta_{ref}},P_{\theta_{target}})=\max_{\theta_{target}}\prod_{i=1}^{n}P_{\theta_{target}}(x_i) $$
The Maximum Likelihood Principle
We started with a measure of disparity between two probability distributions defined by their own parameters and gradually transformed the optimization objective of minimizing the divergence into a likelihood maximization problem.
The key insight is - it is evident that starting with initial KL divergence if we continue to minimize the gap, we will finally find out the initially unknown parameter of the target distribution that is highly likely to be the true parameter. Therefore, the parameter value that minimizes KL divergence also maximizes the likelihood of the observed data.
Maximum Likelihood Estimation therefore provides a principled mechanism for identifying the parameter values that best explain the observed data. This concept forms the mathematical backbone of a large number of machine learning algorithms, statistical models, generative AI systems and modern parameter estimation techniques.