A research team has shipped a faster, leaner way to discover conditional functional dependencies inside large datasets.
Conditional functional dependencies (CFDs) are rules that describe when one column in a table should determine another — but only under specific conditions. Think of them as precision integrity checks: instead of "city always determines zip code," a CFD might say "city determines zip code only for US addresses." Finding these rules automatically is useful for data cleaning, catching bad imports, and auditing pipelines. The problem is that the search is computationally brutal. The researchers built ParCFDFinder on top of an existing algorithm called CFDFinder, adding parallelization and a set of engineering improvements. They integrated the result into Desbordante, an open-source C++ data profiler with a Python interface.
The numbers are hard to argue with: up to 318 times faster and up to 23 times less memory compared with the Java-based Metanome tool that currently dominates this niche. Average gains — 118 times faster, 14 times less memory — are less dramatic but still significant enough to shift what counts as feasible. Datasets with hundreds of thousands of rows, previously impractical to profile on a laptop, now fall within reach.
Data profiling tools rarely get this kind of attention, which says something about how underserved the space is — and how much quiet pain exists in data engineering teams who currently skip automated dependency checks because they simply take too long.