Many scientific datasets are burdened with incomplete or noncompliant metadata, limiting their findability, interoperability, and reuse. Even when community standards for metadata reporting exist, they often lack machine-actionable representations. A new system from a team of researchers tackles this problem by coupling a large language model (LLM) with real-time access to authoritative biomedical terminology services, moving beyond static text prompts.
The research, described in a paper on arXiv, presents an LLM-based agent that retrieves canonically correct standards on demand by querying standard reporting guidelines and ontology services during inference. This contrasts with prior approaches that feed ontology constraints as static text, relying solely on the model's training knowledge. The system is evaluated on 839 legacy metadata records from the Human BioMolecular Atlas Program (HuBMAP), using an expert-curated gold standard for exact-match assessment.
Background: The Metadata Standardization Challenge
Scientific metadata are often recorded in ad hoc formats, making them noncompliant with community reporting guidelines. To produce FAIR (Findable, Accessible, Interoperable, Reusable) datasets, metadata standards must be encoded as machine-actionable templates with rich field specifications and precise value constraints. Previous work showed that LLMs guided by field names and ontology constraints improve standardization, but those constraints were provided as static text prompts, which limited accuracy.
The Ontology-Constrained LLM Agent Approach
The proposed system adds a real-time tool access layer to the LLM. Instead of embedding ontology constraints in the prompt, the agent queries standard reporting guidelines and authoritative biomedical terminology services (such as ontology repositories) at inference time to retrieve the correct canonical values. This ensures the model uses up-to-date, authoritative knowledge rather than its potentially outdated or incomplete training data.
The researchers tested the system on both ontology-constrained fields (where values must come from a controlled vocabulary) and non-ontology-constrained fields (free text). In both cases, augmenting the LLM with real-time tool access consistently improved prediction accuracy over using the LLM alone.
Evaluation Results from HuBMAP
| Feature | LLM Alone (Static Prompts) | LLM + Real-Time Tool Access |
|---|---|---|
| Ontology-constrained fields | Baseline accuracy | Consistently higher accuracy |
| Non-ontology-constrained fields | Baseline accuracy | Consistently higher accuracy |
| Data source | Trained knowledge | Query of live ontology services |
The evaluation was performed on 839 legacy metadata records from HuBMAP, with an expert-curated gold standard providing exact-match ground truth. The consistent improvement across both field types demonstrates a practical path toward automated standardization of legacy biomedical metadata.
The approach is domain-agnostic in principle: any field with a standard reporting guideline could be served by a similar real-time query system, making it relevant to other scientific or enterprise metadata challenges.
Implications for Data-Driven Organizations
For organizations managing large volumes of legacy data, manual metadata curation is slow and error-prone. This LLM-agent approach offers a scalable alternative that reduces reliance on static knowledge and increases standardization accuracy. The system's architecture—querying external authoritative services in real time—can be adapted to other contexts where metadata standards exist but are not machine-actionable. The researchers plan to extend the work to additional ontologies and metadata domains, potentially supporting broader adoption across the biomedical community and other industries.
