A new benchmark catches large language models fumbling words that didn't exist a few years ago.
Researchers built KoNeoBench, a dataset of 1,785 Korean neologisms pulled from online news since 2020 and vetted through expert lexicographic review. Each entry includes usage examples, a breakdown of how the word was formed, and a dictionary-style definition. The team designed four tasks around this data and tested a batch of recent LLMs against a human baseline. Across the board, the models struggled to trace a neologism back to its source components, sort it into the right semantic category, or write an accurate definition.
Most language benchmarks freeze vocabulary in place, which quietly assumes language stops changing the moment a model finishes training. That assumption breaks harder in Korean than in English, since new Korean words are typically built by fusing content words with functional morphemes rather than just mashing two nouns together. KoNeoBench is one of the more concrete demonstrations that English-centric benchmark design doesn't transfer cleanly to languages with different word-formation rules.
The gap isn't really about Korean. It's a reminder that any benchmark is a snapshot, and the internet keeps inventing words faster than anyone can retrain on them.