A new gradient-selection method skips the easy examples during neural network training and mostly gets away with it.
Researchers introduce K-ABENA, a framework that excludes low-loss observations from the backward pass during each training iteration. The key word is "compensated": earlier variants, along with established methods like OHEM and SBP, contain a selection bias that the authors prove mathematically prevents convergence at a true minimum. Their fixed version applies Horvitz-Thompson inverse-probability reweighting to correct for that bias. On real datasets — Breast Cancer, Digits, Wine, Diabetes — the compensated estimator is statistically indistinguishable from full-batch SGD (paired permutation tests, p >= 0.5) while cutting 28-54% of per-epoch gradient computation. At 0.17% class imbalance, full-batch SGD reaches AUC 0.9998; the compensated estimator hits 0.9991 at 28.4% compute savings, a small but real gap the authors do not hide. Uncompensated variants collapse to AUC 0.53-0.62 under the same conditions.
The proof that uncompensated loss-based selection — a category covering several popular techniques — cannot converge correctly is the sharper contribution here. It reframes a class of existing training shortcuts as quietly broken rather than merely suboptimal. For teams running large training jobs where gradient computation dominates cost, a 28-54% reduction per epoch compounds fast.
All experiments were CPU-scale, run on NumPy and scikit-learn. That limits how far the results transfer to GPU-heavy workloads, and the authors say so — which, in a field that routinely oversells benchmark conditions, is worth noting.