The search for a universal mutual information estimator just hit a wall — and that wall was built by better testing.
Researchers published a benchmarking framework that evaluates mutual information (MI) estimators, the statistical tools used to measure how much two variables share information with each other, against far more demanding conditions than the field has used before. Existing benchmarks lean on simple, low-dimensional synthetic data; this new suite pulls in real-world image data and a broader class of dependency structures drawn from copula theory — a mathematical framework for describing relationships between variables independently of their individual distributions. Three families of estimators were put through the tests: non-parametric methods, discriminative approaches, and generative models.
The finding matters because MI estimation quietly underpins a wide range of machine learning tasks — feature selection, representation learning, and causal inference among them. If the benchmarks used to validate these estimators are too easy, researchers have been shipping methods that look solid on paper but degrade on anything resembling production data. This study's results show each estimator class can systematically beat the others under specific conditions, which means no one has been getting the full picture.
The code is open-source, which at least means the community can now stress-test its own assumptions — something the field probably should have done before the benchmarks became canonical.