Large language models can generate fluent root cause analyses, but fluency alone is insufficient for accountability in high-stakes operations, according to a new paper on arXiv. The paper introduces JustDiag!, a diagnostic justification engine designed to address the gap by maintaining an explicit process state during root cause analysis (RCA). This tool enables engineers to see what evidence supported a diagnosis, which alternatives were considered, where contradictions remained, and whether the system resolved the case or preserved uncertainty.
The Problem of Accountable Root Cause Analysis
In real incident response, engineers need more than a fluent final answer. They require transparency into the reasoning process to ensure accountability, especially in high-stakes operations. The JustDiag! paper, authored by Bi, Tingzhu; Jiang; Xinrui; Zhang; Xun; Su; Pengcheng; He; Congjie; Li; Jinglin; Wang; Ping; Ma; Meng, argues that accountable RCA requires explicit diagnostic justification artifacts and process-aware evaluation, not just fluent final answers.
How JustDiag! Works
JustDiag! maintains an explicit process state that tracks evidence, findings, competing hypotheses, conflicts, and next checks. This process state allows the system to document its reasoning step by step, providing a record of how a diagnosis was reached. Unlike standard LLM-based RCA that only outputs a final narrative, JustDiag! preserves uncertainty and notes contradictions, making the analysis more transparent and auditable.
Evaluation on 66 Real-World Incidents
The researchers evaluated JustDiag! on 66 real-world incidents using a two-layer protocol that separately scores final-answer quality and process quality. They compared JustDiag! against a matched control without diagnostic justification. The results showed that JustDiag! achieved stronger outcome and process scores, while accepting slightly lower terminal completion due to more calibrated non-closure. This means JustDiag! is more honest about when it cannot confidently resolve a case, which is valuable for accountability.
| Metric | JustDiag! | Control |
|---|---|---|
| Final-answer quality | Stronger score | Weaker score |
| Process quality | Stronger score | Weaker score |
| Terminal completion | Slightly lower (calibrated non-closure) | Higher (but less calibrated) |
Implications for Enterprise AI
For enterprise technology decision-makers, JustDiag! represents a step toward more accountable AI systems in critical domains like supply chain incident response, network operations, and cybersecurity. By providing a transparent reasoning trail, such tools can build trust and enable human oversight. The paper's emphasis on process quality over fluent output aligns with the needs of CTOs and digital transformation leaders who require auditable AI in high-stakes operations.
