A new theoretical paper argues that teaching an AI agent to "get to know you" runs into hard limits no matter how much you tinker with its scaffolding.
The work, posted to arXiv as arXiv:2609.36892 on September 30, 2026, studies "personal agents": AI assistants meant to adapt to an individual user's preferences without retraining the underlying language model. Instead of changing the model itself, these systems evolve their "harness," the surrounding layer that manages memory, context, and tool use. The researchers built a preference-oriented benchmark to test how harness architecture, harness scale, and self-evolution algorithms shape an agent's ability to personalize, then formalized harness evolution as a learning problem, attributing its failures to three familiar sources: approximation error, generalization error, and optimization error.
That framing matters because most personalization pitches assume an agent simply keeps getting better at knowing you the longer you use it. This paper says that assumption has a ceiling: some limits come from what a harness can represent at all, others from how little interaction data any single user actually generates for the system to learn from. It's a useful check on "an AI that learns you" products built on top of a frozen base model.
What the paper does not do is show where today's real agents actually sit relative to those limits - the analysis is theoretical, without benchmark scores attached to specific products.