An open-source math knowledge graph is using a committee of large language models to decide what actually counts as mathematics.
Mathswitch pulls concept records from Wikidata, Wikipedia, MathWorld, nLab, ProofWiki, and other sources, then links entries that describe the same idea. Because its concept set is built by querying Wikidata's collaboratively edited graph, the imported data is messy: some items are not mathematical at all, and others sit in a gray area. Researchers tested a voting ensemble — multiple LLM judges that each cast a classification vote — to filter out that noise. They used Wikidata items with known MathWorld identifiers as a positive control, then checked what happened to classifications when database identifiers were stripped from the context.
The interesting part is not the accuracy number but the error taxonomy. When the judges disagreed with MathWorld, the disagreements clustered into three buckets: degenerate descriptions (entries too sparse to classify reliably), narrow scope bias (models that read "mathematics" too strictly), and editorial-scope mismatches (cases where Mathswitch and MathWorld simply draw the boundary differently). Each failure mode points to a different fix, which is a more useful output than a single benchmark score.
LLM-as-judge setups are everywhere right now, but most are evaluated on text quality or factual recall — using an ensemble to curate a structured knowledge graph is a less common application, and the failure-mode breakdown is the kind of analysis that tends to get skipped in shinier papers.