Topic
evaluation
Risk-Aware LLM Agents for Geospatial Data Retrieval: New Framework Passes Adversarial Tests
Researchers present a risk-aware LLM agent framework for retrieving remote sensing data from cloud-based geospatial catalogues using natural language queries. The system integrates Guardrail, General-QA, and Recommender-Analyst agents to convert user intent into structured API calls. Preliminary adversarial evaluation shows prompt-level safety instructions improve robustness, though rare high-impact failures persist.
New OSGuard Benchmark Evaluates Safety of Computer-Use Agents for Enterprise AI Deployment
Researchers introduce OSGuard, a benchmark suite for evaluating safety in computer-use agents. It includes action-level guardrail decisions and a risk-augmented execution suite to detect unsafe completions that satisfy nominal task objectives. Early tests show current multimodal guardrails perform well on isolated action judgments but reveal gaps in end-to-end safety.
RecourseBench: Modular Framework Promises Reproducible Evaluation of AI Recourse Methods
A new framework called RecourseBench aims to standardize and validate algorithmic recourse methods—counterfactual explanations that show individuals how to reverse an AI's decision. It decomposes the evaluation pipeline into five decoupled layers and integrates 28 state-of-the-art methods, with automated tests to verify reproducibility.
Metric Match: New Subset Selection Method Improves LLM Judge Reliability Evaluation, Cuts Annotation Costs by 32.5%
Researchers developed Metric Match, a subset selection method that reduces costly human annotations needed to evaluate LLM judge reliability. The approach achieves a 0.838 win-rate over random selection, cuts estimation error by 18.7%, and reduces annotation needs by 32.5%. A medical case study showed $1,041.67 in savings.