Enterprise software teams building multi-agent LLM systems that generate structured workflows from natural-language requests face a persistent problem: routine updates — swapping a model, refactoring an agent prompt, or simply re-running the same input — can produce workflows that differ from previously validated references. Without a principled way to assess whether a change is safe to ship, engineers rely on metric scores that are often poorly calibrated. According to a paper published on arXiv, "metric scores are poorly calibrated, and a numeric change rarely communicates the severity of the underlying degradation."
To address this, researchers introduced WorkflowPerturb, a controlled benchmark designed to study workflow evaluation metrics by applying realistic, graded perturbations to golden workflows. The benchmark contains 4,973 golden workflows and 44,757 perturbed variants across three perturbation types: Missing Steps, Compressed Steps, and Description Changes. Each perturbation is applied at severity levels of 10%, 30%, and 50%.
The team benchmarked multiple metric families and analyzed their sensitivity and calibration using expected score trajectories and residuals. Their results "characterize systematic differences across metric families and support severity-aware interpretation of workflow evaluation scores in change-management settings."
The Perturbation Design
The three perturbation types simulate common real-world changes:
- Missing Steps: Omission of one or more workflow steps at varying severity.
- Compressed Steps: Combining multiple steps into a single, more abstract step.
- Description Changes: Modifying the natural-language description of steps while preserving structure.
| Perturbation Type | Description | Severity Levels |
|---|---|---|
| Missing Steps | Omitted workflow steps | 10%, 30%, 50% |
| Compressed Steps | Steps merged into fewer | 10%, 30%, 50% |
| Description Changes | Altered text of steps | 10%, 30%, 50% |
Implications for Enterprise Automation
Multi-agent LLM systems are already deployed in production for cloud automation, DevOps, and enterprise process orchestration, according to the paper. WorkflowPerturb provides a methodology to calibrate metric interpretation, enabling teams to distinguish between acceptable variation and true degradation.
"Operating such systems exposes a recurring change-management problem," the paper states. The benchmark aims to give engineers a severity-aware framework to evaluate whether a modification is safe.
For supply chain and logistics organizations that automate procurement, order processing, or customs documentation workflows using LLM-based agents, the ability to compare workflow outputs against a reference under controlled perturbations could reduce deployment risk. While the current dataset is generic, the perturbation methodology could be adapted to domain-specific workflows.
The dataset will be released upon acceptance of the paper, according to the researchers.
