Researchers want AI to judge not just what you say in a job interview, but how your face moves while you say it.
A team has proposed a framework that feeds an LLM two streams of data from asynchronous video interviews: the candidate's transcribed text responses, and sequences of facial action units — the discrete muscle movements that underlie expressions like a furrowed brow or a lip press. The system converts those AU sequences into plain-text descriptions, merges them with the spoken-word transcript inside the LLM, then runs the combined output through a lightweight regression layer to produce continuous scores across personality traits. Tested on the AVI-6 benchmark, the approach beat most existing baselines on prediction error and correlation with human raters.
The appeal, from a research standpoint, is filling the gap that text-only models leave open. Candidates in recorded interviews can craft careful answers, but non-verbal cues have long been treated as a separate, harder-to-process signal. Packaging AU data as text lets a single LLM handle both channels without separate vision encoders, keeping the architecture compact and the outputs interpretable.
The obvious tension here is that automated personality scoring in hiring is already contested territory — regulators in several jurisdictions have moved to restrict or require disclosure of AI-based candidate assessment tools, and the validity of personality tests in predicting job performance remains debated in the psychology literature. A system that is more accurate at reading faces is not automatically a system that should be used.