The most plausible near-term role of medical large language models (LLMs) is to assist rather than replace physicians, yet current evaluations often test isolated capabilities—clinical knowledge, electronic health record (EHR) system interaction, or patient communication—separately. According to a paper published on arXiv, physician assistance instead requires coordinating these capabilities within the same interaction, where physicians issue underspecified requests, patients describe symptoms ambiguously, and EHR systems demand precise tool use.
To address this gap, the researchers introduced PhysAssistBench, a benchmark for interactive doctor-patient-EHR assistance. The benchmark is built from real MIMIC-IV clinical cases and uses a scalable pipeline to construct agentic patients: interactive, record-grounded agents that turn static EHR records into multi-turn clinical scenarios while preserving clinical factuality.
The Need for Coordinated Assistance
The paper argues that current medical LLM evaluations do not capture the complexity of real-world physician support. In practice, a physician may issue an underspecified request, a patient may describe symptoms ambiguously, and the EHR system requires precise tool use. Coordinating these elements within a single interaction is essential for effective assistance, but existing benchmarks do not test this.
Benchmark Construction: From MIMIC-IV to Agentic Patients
PhysAssistBench uses a scalable pipeline to transform static EHR records from the MIMIC-IV database into dynamic, agent-based patient simulations. The benchmark provides a curated bilingual evaluation set of 1,296 manually reviewed and physician-validated turns, ensuring both clinical factuality and linguistic diversity. The agentic patients are designed to interact naturally with physician queries, responding in ways that mirror real patient behavior.
Evaluation Results: Current LLMs Fall Short
Experiments with leading LLMs showed that current models remain unreliable in this interactive setting, according to the paper. The study identifies a key bottleneck for clinical LLMs: reliable assistance requires coordination across knowledge, communication, and systems, not isolated gains in any single area. This finding suggests that even state-of-the-art models struggle when forced to integrate multiple competencies simultaneously.
Implications for Enterprise AI
While PhysAssistBench is specific to healthcare, its core finding—that coordination across knowledge, communication, and systems is the limiting factor—is directly relevant to enterprise AI deployments in complex, multi-step workflows such as supply chain management or trade finance. Any AI assistant tasked with interacting across diverse systems (e.g., ERP, TMS, customs portals) and human stakeholders must demonstrate similar coordination capabilities. The PhysAssistBench methodology could inspire analogous benchmarks for other domains.