Researchers have proposed a new reinforcement learning algorithm designed to stop deep Q-networks from systematically overvaluing bad decisions.
Deep Q-Networks learn by bootstrapping — using their own estimates to update themselves — and that loop tends to inflate value predictions over time. Prior fixes split into two camps: ensemble methods that average or minimize across multiple networks, and multi-step return methods that look further ahead before updating. Ensemble Elastic DQN (EEDQN) combines both. It replaces the computationally expensive state-similarity test from an earlier variant, Elastic Step DQN, with a simpler Q-value difference rule that decides on the fly how many steps to look ahead. It then applies different aggregation depending on the horizon: short returns use the ensemble mean, longer returns use the more conservative ensemble minimum.
The distinction matters because uniform conservatism tends to undertrain the network, trading one bias for another. By applying pessimism only where overestimation risk is highest — longer, noisier return horizons — EEDQN tries to stay honest without becoming uselessly timid. Tested across five MinAtar environments, it posted the highest final return in four of the five against a field that included Double DQN, Averaged DQN, and MaxMin DQN.
The caveat buried in the ablation results is the most interesting part: the best degree of conservatism was environment-dependent, meaning the adaptive logic does not fully solve the tuning problem — it just pushes it one level up. That is a common pattern in ensemble RL research, and EEDQN is no exception.