Topic
calibration
Fine-Tuning LLMs for Vulnerability Detection Fails to Improve Security Reasoning, Study Finds
A new study introduces CWE-Trace, a framework for evaluating LLM vulnerability detection using Linux kernel samples. It finds that fine-tuning and data contamination do not improve security reasoning; detection accuracy remains near chance, and models lack genuine comprehension.
New Research Proposes Adversarial Reweighting to Calibrate Mixture-of-Experts Models Under Distribution Shift
A new study examines how mixture-of-experts (MoE) models behave under distribution shift and proposes an adversarial reweighting approach to maintain calibration. The method improves the accuracy-calibration tradeoff across model classes, prediction tasks, and distribution shifts.
WorkflowPerturb Benchmark Offers Calibrated Stress Tests for Multi-Agent Workflow Metrics
WorkflowPerturb introduces 4,973 golden workflows and 44,757 perturbed variants with three perturbation types at severity levels 10%, 30%, and 50%, enabling calibrated interpretation of workflow evaluation metrics for change management in multi-agent systems.