Large language models (LLMs) exhibit representational and syntactic biases that are difficult to evaluate due to the stochastic nature of text generation. Standard auditing methods rely on single-output inspection or static automated metrics, which obscure underlying probability distributions and fail to capture biases hidden in lower-probability generation branches. According to a new arXiv preprint (2606.19344) by Pelossi et al., researchers have developed TreeTracer, a visual analytics tool designed to evaluate LLM bias through aggregated comparison.
The TreeTracer Approach
TreeTracer employs a systematic perturbation analysis pipeline. It replaces ontology-defined terms in each input prompt, aggregates hundreds of stochastic generations into a syntax-aligned hierarchical structure, and performs classification-aware node merging with an auxiliary language model. The resulting structure is visualized through a custom Sankey diagram. By juxtaposing two ontology-driven trees, the workspace enables direct comparison between semantic contexts and supports systematic bias detection.
To avoid misinterpretation of bias presence, the system applies contrastive inference to compute and directly display counterfactual token probabilities across contexts. This reduces the risk of misinterpreting the presence of bias, according to the paper.
Validation and Case Studies
The researchers validated TreeTracer through case studies comparing an unaligned baseline model GPT-2 XL against the constitutionally aligned Apertus models. The visual aggregation successfully exposed hidden representational harms, such as:
- Counterfactual pronoun suppression
- Conversational marginalization of individuals
A preliminary user study confirmed that the aggregated comparative interface reduces cognitive load and effectively supports analysts in detecting systemic biases, the paper reports.
| Feature | GPT-2 XL (Unaligned) | Apertus (Constitutionally Aligned) |
|---|---|---|
| Pronoun suppression | Detected | Not detected |
| Conversational marginalization | Detected | Reduced |
| Bias detection clarity (user study) | Lower | Higher |
Implications for Enterprise AI Governance
For enterprise technology leaders, tools like TreeTracer offer a method to audit LLM outputs for hidden biases that could affect customer-facing applications, hiring tools, or supply chain communications. The ability to visualize probability distributions across multiple generation paths provides a more comprehensive evaluation than single-output checks. While the initial validation was on research models, the methodology can be applied to any transformer-based LLM, making it relevant for enterprise deployment decisions.
The paper (arXiv:2606.19344) is authored by Pelossi, Matteo, Sevastjanova, Rita, Spinner, Thilo, and El-Assady, Mennatallah, and is available under a CC BY 4.0 license.
