A research team has proposed a way to run full Bayesian reinforcement learning without needing a likelihood function — a requirement that blocks most existing methods from working in practice.
The paper introduces Likelihood-Free Iterated Batch Importance Sampling (LF-IBIS), an algorithm that combines Approximate Bayesian Computation with Iterated Batch Importance Sampling. The approach lets an agent update its beliefs online as new environment interactions arrive, producing approximate posterior distributions over both environment parameters and optimal policies. The researchers tested it on response-adaptive randomization in clinical trials — a domain where closed-form posteriors exist and can validate the approximation — then pushed it into settings where no closed form is available.
The practical gap this targets is real. In most deployable RL scenarios, the underlying environment dynamics are too complex to express as a tractable likelihood, which forces practitioners to either simplify their models or abandon Bayesian approaches altogether. By sidestepping that requirement, LF-IBIS keeps uncertainty quantification intact — including over the exploration-exploitation trade-off, which is where poor calibration tends to cost the most.
Bayesian RL has been a research staple for years, but the likelihood bottleneck has kept it largely confined to toy problems. Whether LF-IBIS scales to the messy, high-dimensional environments that make likelihood-free methods necessary in the first place is the question the paper leaves open.