item response theory

New JE-IRT Framework Reveals Multidimensional Abilities of Large Language Models

Standard LLM evaluation compresses diverse abilities into single scores. JE-IRT, a geometric item-response framework, embeds both LLMs and questions in a shared space, where direction encodes semantics and norm encodes difficulty. The approach reveals topical specialization, explains out-of-distribution behavior, and uncovers cross-subject ability directions like an arithmetic axis, offering a more interpretable lens for model evaluation.