Topic
psychometrics
New JE-IRT Framework Reveals Multidimensional Abilities of Large Language Models
Standard LLM evaluation compresses diverse abilities into single scores. JE-IRT, a geometric item-response framework, embeds both LLMs and questions in a shared space, where direction encodes semantics and norm encodes difficulty. The approach reveals topical specialization, explains out-of-distribution behavior, and uncovers cross-subject ability directions like an arithmetic axis, offering a more interpretable lens for model evaluation.
Psychometric Datasheet Reveals 'Dark Current' Bias in LLM-as-a-Judge Evaluation Systems
Researchers introduce a Judge Datasheet protocol to measure biases in LLM-as-a-judge systems, including dark current under vacuum inputs and positional false preference. A case study of three open-weight models reveals stark differences in measurement reliability, with implications for enterprise AI evaluation.