Unreal Engine gives developers two ways to build a game feature: write C++ code, or wire it together visually in Blueprint. A new benchmark suggests AI agents are far better at the first than the second.
Researchers built CraftBench-UE, a testing harness that runs coding agents inside an isolated Unreal Engine instance, rebuilds their submitted work in a clean project, and checks it with deterministic build, asset, and runtime tests instead of an LLM judge. The benchmark covers 70 tasks split across C++ source, Blueprint assets, and editor scripting, evaluated against seven models under two different editor-tool setups. On 10 tasks that specify identical gameplay but require either C++ or Blueprint as the deliverable, C++ completion rates beat Blueprint by 30.0 and 42.9 percentage points depending on setup. Even Blueprint submissions that arrived on time and passed asset checks failed the actual runtime gameplay tests 42.2% to 50.0% of the time.
That gap matters because Blueprint is Unreal's pitch to non-programmers, and it is increasingly the interface AI coding tools get pointed at for game development. The benchmark shows that passing surface-level checks, like whether an asset exists or a project compiles, does not mean the gameplay actually works, which is exactly the gap that makes agent output risky to trust without dedicated testing.
It is another data point in a pattern that keeps showing up in coding-agent evaluations: code that runs is not the same as code that does what was asked.