Researchers have released a dataset that catalogs thousands of real Arabic conversations with voice assistants, including the ones that went badly.
The dataset, called WASIL, contains 8,529 in-the-wild spoken interaction turns, paired with audio, automatic speech recognition (ASR) hypotheses, assistant responses, and explicit like/dislike feedback from users. About 14.2 percent of those turns were marked as dislikes. A separate 2,000-turn test set covers Modern Standard Arabic and four major dialects. The team also built low-cost gold transcripts by cross-checking multiple ASR systems, and labeled each turn by answerability, splitting answerable requests from ambiguous ones, unsupported ones, and plain noise.
That labeling is the point. Most voice assistant complaints get blamed on the underlying language model, but a chunk of them are really transcription failures feeding bad input to a model that then does exactly what it was told. By separating ASR errors from genuine ambiguity or dumb requests, WASIL lets researchers measure how much of an assistant's failure is really a listening problem versus a reasoning problem. That distinction is harder to pin down for Arabic, where dialect variation stacks on top of the usual speech-recognition noise most English-centric benchmarks never have to deal with.
It is also a reminder of how little of this evaluation infrastructure exists outside English and a handful of other major languages. A benchmark is not a product fix, but you cannot fix what you cannot measure, and right now most Arabic-language assistants are being judged, if at all, on vibes.