Building explicit skill libraries is a promising approach to make computer-using agents more inspectable, but manually authoring such libraries is labor-intensive and difficult to scale. In a new study published on arXiv, researchers Hao, Yuexing, Li, and Xiaomin investigate whether skill libraries can be automatically mined from agent interaction data and whether those mined skills improve downstream policy performance. According to the paper, the answer is a qualified yes on readability but a clear no on transfer.
The team developed a three-stage pipeline that first segments GUI trajectories into meaningful steps, then clusters those segments into candidate skills, and finally trains a skill-aware policy using the resulting annotations. The pipeline uses a boundary detector to partition trajectories, an orderless segment representation to describe each segment, and an offline reward model to guide policy learning.
Readability results were promising. When evaluated against InteraSkill Workflows labels, five of the eight mined clusters achieved a purity of at least 0.95, indicating that the clusters align closely with human-defined skill categories. The researchers note that "the mined clusters are readable on the source benchmark" — meaning the automated process can capture human-interpretable skill structure from raw interaction data.
Downstream performance tells a different story. The same clusters did not translate into meaningful policy improvements. When a GRPO (a gradient-based policy optimization method) was applied using the mined skill annotations, the IW skill-step accuracy improved only marginally, from 18.5% to 20.5%. On the BrowseComp+ benchmark, accuracy remained essentially unchanged. Moreover, the skill-aware policy underperformed trivial frequency priors on key source-domain metrics, suggesting that the mined skill representations do not provide a useful signal for learning better policies outside the training environment.
The authors present the work as a diagnostic study. They state: "trajectory mining can expose inspectable skill structure, but the current boundary detector, orderless segment representation, and offline reward model are insufficient for reliable cross-domain policy improvement."
| Metric | Baseline | After GRPO with mined skills |
|---|---|---|
| IW skill-step accuracy | 18.5% | 20.5% |
| BrowseComp+ accuracy | unchanged | unchanged |
| Source-domain frequency prior outperformance | — | underperformed |
The findings highlight a central challenge for the field of automated skill generation: while unsupervised trajectory mining can produce clusters that match expert labels, bridging the gap to improved agent behavior remains elusive. For enterprise technology leaders evaluating AI agents for automation in domains such as supply chain or logistics, the study serves as a cautionary tale — inspectability and readability of learned skills do not automatically translate to better operational outcomes. The current bottleneck lies in the representations and reward models used to connect mined skills to policy learning.
Future work, the researchers suggest, would need to refine the segment representation, improve the boundary detector, and develop reward models that better capture the semantics of skill execution across different tasks. Until those components advance, automated skill mining will remain a diagnostic tool rather than a production-ready solution.
