Auditing AI outputs is the wrong unit of analysis, argues a new paper — the conversation that produced them is what actually reveals accountability.
Researchers have proposed LLMography, a framework that treats Human-AI conversation logs the way bibliography treats sources: as structured, citable records of how a piece of work came to exist. A prototype tool analyzes chat traces and generates reports across seven KPIs — Prompt Quality Score, Human Direction Score, AI Dependency Level, Auditability Score, Final Output Traceability, Privacy Risk Level, and a recommended LLMography label. In a preliminary run on 19 anonymized engineering-student audit reports, most interactions were classified as Human-AI co-produced. Average scores came in at 86.8/100 for Human Direction, 81.9/100 for Prompt Quality, 72.8/100 for Auditability, and 77.1/100 for Final Output Traceability. The paper also applies its own framework to itself, landing the classification "human-originated, human-directed, AI-assisted co-production."
The practical gap this targets is real: organizations deploying AI in education or engineering workflows currently have no standard way to distinguish a human who used AI as a spell-checker from one who outsourced the entire task. A conversation-level trace would make that distinction auditable rather than assumed. The framework also introduces a Privacy Risk Level score, quietly flagging that detailed interaction logs carry their own exposure.
The proposal is academic for now — 19 student reports is a thin evidence base, and the hardest adoption problem (getting people to share unedited AI conversation histories) goes largely unaddressed. It reads more as a call to reframe the debate than a deployable standard, but that reframe is the right one.