Topic
reproducibility
New AIBOM-Driven Framework Automates Advisory Generation for Agentic AI Cybersecurity
Researchers present a reproducible framework that automates the generation of CSAF VEX advisories for agentic AI by combining static SBOM/AIBOM artefacts with runtime telemetry, cryptographically signing them, and validating via deterministic replay. The evaluation uses approximately 10,000 component entries from synthetic workloads of 50 to 5,000 components, incorporating OSV, GitHub Advisory, KEV, and EPSS datasets.
AgentBeats Proposes Open Standard for Reproducible AI Agent Evaluation Across Benchmarks
A new research paper introduces AgentBeats, a framework for open, standardized, and reproducible AI agent assessment. The approach uses judge agents and protocols A2A and MCP to unify evaluation, demonstrated through a five-month competition with 298 judge agents and 467 subject agents.
RecourseBench: Modular Framework Promises Reproducible Evaluation of AI Recourse Methods
A new framework called RecourseBench aims to standardize and validate algorithmic recourse methods—counterfactual explanations that show individuals how to reverse an AI's decision. It decomposes the evaluation pipeline into five decoupled layers and integrates 28 state-of-the-art methods, with automated tests to verify reproducibility.