Topic
test case generation
QMFOL Benchmark Reveals LLM Reasoning Degrades with Logical Complexity, New Framework Enables Precise Evaluation
A new automated framework called QMFOL generates deductive reasoning tasks with quantifiable logical complexity, enabling precise evaluation of LLM reasoning. The associated benchmark, QMFOLBench, comprises 2,880 instances across 960 configurations. Evaluations on six large reasoning models (LRMs) and two LLMs show performance degrades and computational overhead increases with rising logical complexity, with models performing better on True-labeled tasks than False or Unknown ones.
AI-Driven Test Case Generation from Natural Language: Survey Reveals Six Quality Gaps and Research Roadmap
A systematic review of 21 primary studies on AI-driven test case generation from natural language requirements reveals that no existing approach simultaneously satisfies six key quality dimensions: automation, ambiguity handling, domain applicability, traceability, evaluation thoroughness, and hallucination control. The survey synthesizes three evolutionary eras and proposes four actionable research guidelines targeting hallucination, traceability, complexity sensitivity, and compliance.