Recent advances in Large Language Models (LLMs) have substantially transformed Automated Essay Scoring (AES), yet the internal mechanisms underlying LLM-based scoring remain poorly understood. In a new study, researchers led by Zuo et al. systematically analyze the hidden representations of eight LLMs across two English essay datasets (ASAP++, CSEE) and one Portuguese dataset (ENEM). Using linear probing, cross-prompt generalization, dimensionality reduction, and neuron-level analyses, they find consistent evidence that essay quality information is encoded in a linearly accessible form within LLM representations.
Probing Essay Quality in LLM Representations
The study provides evidence that representations of essay quality emerge progressively across layers, remain robust across prompting strategies, and partially transfer across essay prompts despite differences in scoring rubrics. Nonlinear probes provide only marginal and inconsistent improvements over linear probes, suggesting that most essay quality information is already linearly decodable. The researchers further identify individual "essay scoring neurons" whose activations strongly correlate with essay scores and whose behavior is sensitive to targeted intervention. Moreover, the layer-wise distribution of these neurons systematically shifts with essay length, with longer essays relying more heavily on deeper layers.
Key Findings
- Linearly decodable: Essay quality is linearly accessible in hidden representations.
- Progressive emergence: Quality information emerges across layers, becoming more pronounced in deeper layers.
- Robustness: Patterns hold across different prompting strategies.
- Cross-prompt transfer: Quality information partially transfers between essay prompts even with different rubrics.
- Essay scoring neurons: Individual neurons strongly correlate with scores; intervention alters scoring.
- Length dependence: Distribution shifts with essay length; longer essays use deeper layers more.
Methodology and Datasets
The study employed eight LLMs and three datasets:
| Dataset | Language | Description |
|---|---|---|
| ASAP++ | English | Extended Automated Student Assessment Prize dataset |
| CSEE | English | CSEE dataset |
| ENEM | Portuguese | Brazilian national exam dataset |
Techniques included linear probing, cross-prompt generalization, dimensionality reduction, and neuron-level analyses.
Implications for Interpretable AI
These findings provide evidence that LLMs encode structured representations related to essay quality, offering new insights into the interpretability of LLM-based AES systems. Understanding where and how essay quality information is represented can guide improvements in model design and scoring reliability. The approach of probing hidden representations may also be applicable to other domains requiring quality assessment, though further research is needed. The study underscores that LLMs do not simply output scores but build internal representations of text quality, a step toward more transparent AI evaluation systems.