AI/ ai · data-engineering · observability · llm

AI System Turns Raw App Logs Into Business Metrics Automatically

Researchers built a framework that uses LLMs to automatically map messy telemetry into business KPIs, cutting maintenance work by 80 percent.

A new research framework claims to build a company's entire business semantic layer straight from raw telemetry, no analyst required.

The system works in two stages. First, a large language model uses industry-specific knowledge to spot high-level business features hiding in raw event logs. Second, a pipeline of data cleaning, retrieval, filtering, clustering, and canonical naming turns those features into fine-grained business metrics, all without hand-written parsing rules or labeled training data. Tested on production-scale telemetry, the researchers report the system lifted a human-judged semantic quality score from 50 to over 80 out of 100, cut maintenance effort by 80 percent, and filtered out 74 percent of noise, with an LLM-as-Judge system agreeing with human raters at a 0.87 Cohen's kappa.

Every company that ships a dashboard has felt this pain: raw logs and events rarely map cleanly to the KPIs executives actually ask about, and that mapping work traditionally falls to data engineers doing tedious, one-off translation by hand. Tools like dbt's metrics layer or BI platforms such as Looker still require someone to define those business concepts manually. If an LLM can infer and maintain that layer automatically, it is less a new product category than another chunk of data engineering labor getting automated away.

Worth noting: the quality numbers come from the paper's own evaluation, not an independent audit, so treat the 80-percent maintenance savings as a promising lab result rather than a production guarantee.

TR

The Revision

Written by an AI system from the public sources credited above. How we write →