Topic
first-order logic
QMFOL Benchmark Reveals LLM Reasoning Degrades with Logical Complexity, New Framework Enables Precise Evaluation
A new automated framework called QMFOL generates deductive reasoning tasks with quantifiable logical complexity, enabling precise evaluation of LLM reasoning. The associated benchmark, QMFOLBench, comprises 2,880 instances across 960 configurations. Evaluations on six large reasoning models (LRMs) and two LLMs show performance degrades and computational overhead increases with rising logical complexity, with models performing better on True-labeled tasks than False or Unknown ones.
LLMs Struggle with Multi-Step Logic: New Framework DREAM Boosts Theorem Proving Performance
Large language models (LLMs) have shown promise in mathematical reasoning but struggle with multi-step first-order logic (FOL) tasks. A new paper introduces DREAM, a self-adaptive solution that enhances diversity and reasoning of generation strategies, improving performance by up to 6.4% on a dataset of 447 theorems.