Two obscure tweaks to neural network training - fractional calculus and fractal math - only boost performance when paired carefully, not automatically.
Researchers tested fractional optimizers, a family of training methods that extend standard gradient descent using fractional derivatives and memory of past updates, against fractal activation functions, which use self-similar Weierstrass- and Blancmange-type curves instead of the usual smooth activation shapes. They ran the combinations on two math benchmark surfaces, Ackley and Himmelblau, including versions distorted with Weierstrass-style noise, and then on feed-forward networks classifying ten different datasets. The study compared standard optimizers, regularization-style fractional methods, and both explicit and adaptive memory-based fractional optimizers. No single combination won across the board.
The results argue against treating either technique as a drop-in upgrade. Regularization-style fractional scaling worked well only with specific fractal activations, and the memory-heavy Grunwald-Letnikov approach only showed its worth on the distorted benchmark surfaces, not the clean ones. That is a useful data point for anyone chasing smarter optimizers: fractional math and fractal activations are context-dependent tools, not universal replacements for the defaults most networks still use.
Most of deep learning's recent gains have come from scale and data, not exotic calculus - this paper is a reminder that clever math still has to earn its keep on the benchmark table.