A new benchmarking method suggests that years of graph neural network research may have been comparing models across a coin flip, not a fair test.
Researchers built a technique called HP that changes how datasets get split into training, validation, and test sets before benchmarking graph neural networks (GNNs). The standard approach borrows k-fold cross-validation from ordinary machine learning, sometimes stratifying folds by class label to keep the label mix even. But graphs are not rows in a spreadsheet: nodes are connected, and how much a node's neighbors share its label, its local homophily, varies across the graph and directly shapes how message-passing models behave. HP stratifies folds by homophily first and class label second, so no single fold accidentally loads up on nodes that are easy or hard for a GNN to classify.
Across 15 datasets and 7 GNN architectures, HP produced far more consistent accuracy rankings than random k-fold splitting, and it won on 13 of the 15 datasets while still preserving class balance. That matters because published claims that one architecture beats another by a point or two could just be measuring which random split a paper happened to use, not a real algorithmic edge.
It's a small fix, but it points at a bigger problem: benchmark leaderboards move faster than the statistical hygiene needed to trust them.