Researchers built an AI interviewer that reads how much you know and pitches its next question accordingly.
The system, called an Evidence-Traceable Dynamic Interviewer Architecture, runs entirely on a locally hosted large language model rather than a cloud API. A five-module, prompt-driven pipeline keeps a persistent record of the conversation so it can profile a participant's expertise as the interview unfolds, then generate questions, responses, and transition messages pitched to that level. In a study with 246 participants, the expertise-profiling module matched independently reported expertise 78.9% of the time, with a weighted Cohen's K of 0.80. A separate module for generating follow-up questions showed a strong link between participant expertise and question complexity, and participants rated the interviews highly for relevance, engagement, and satisfaction.
Most chatbot-style interviewers, whether used for user research, job screening, or classroom quizzing, run the same script regardless of who is answering, which wastes an expert's time and overwhelms a novice. Running the model locally also keeps interview transcripts off a third-party API, a detail that matters for research involving sensitive or identifiable responses. The real pitch here is the "evidence-traceable" part: the system keeps a state record showing why it judged someone a given expertise level, the kind of auditability ad hoc prompting with a chatbot does not give you.
An 80% agreement rate with self-reported expertise is respectable, but self-report is a soft yardstick, and getting institutional review boards comfortable with an LLM interviewing human subjects unsupervised is a separate problem this paper does not solve.