Validating changes in low-level C firmware is expensive because unit tests (UTs) are fragile under strict build constraints, where missing headers, unresolved symbols, and dependency mismatches frequently prevent compilation and linking. A new study published on arXiv introduces an automated workflow that leverages large language models (LLMs) to generate unit tests for the Open-Source Silicon Initialization Library (openSIL) firmware codebase maintained by Advanced Micro Devices (AMD). The goal is to reduce manual debugging effort while improving test coverage.
The workflow combines automated generation of test scaffolds, library-aware creation or reuse of stubs, mocks, and fakes, and an iterative compile-dispatch repair loop driven by build logs and line-coverage feedback. According to the paper, this multi-agent pipeline addresses the common failure modes in constrained firmware environments where traditional test creation is highly manual and error-prone.
Quantitative Results
The researchers evaluated the approach across multiple metrics, including compilation success, repair iterations, dispatch success, and line coverage. Across 76 functions under test, the workflow generated compilable unit tests for 73 functions, a compilation success rate of approximately 96%.
In a configuration without line-coverage guidance or retrieval-augmented generation (RAG), mean line coverage reached 73.9%. On a 48-function subset evaluated under two enhanced configurations, the results were significantly higher:
| Configuration | Functions Evaluated | Mean Line Coverage |
|---|---|---|
| No line-coverage guidance or RAG | 76 functions (73 compiled) | 73.9% |
| Line-coverage guidance alone | 48-function subset | 98.8% |
| Line-coverage guidance + vector-database retrieval | 48-function subset | 94.7% |
The study shows that adding line-coverage guidance boosts coverage substantially, while combining it with vector-database retrieval yields 94.7% coverage. The paper reports that the automated generation-and-repair pipelines can "substantially improve UT creation efficiency and coverage for constrained firmware environments while reducing manual debugging effort."
Implications for Enterprise Software Quality
For enterprise technology decision-makers, this research demonstrates how generative AI can be applied to critical, low-level code validation—a domain often considered too rigid for automation. Firmware bugs in supply chain hardware (e.g., server chips, networking gear) can cause costly delays. By automating unit test generation, companies can accelerate validation cycles and improve reliability without scaling manual engineering resources.
Technical Approach
The workflow employs an LLM-guided multi-agent pipeline. It first generates test scaffolds and then uses library-aware techniques to create or reuse stubs, mocks, and fakes to satisfy build dependencies. The iterative repair loop uses compiler error logs to refine the tests, and line-coverage feedback guides subsequent generations. The study also experimented with retrieval-augmented generation using a vector database, though the coverage was slightly lower (94.7%) than with line-coverage guidance alone (98.8%). The authors note that time, cost, and token usage were tracked as secondary measures, but specific figures are not detailed in the source.
The research was conducted on the openSIL firmware codebase, which is maintained by AMD and is used in silicon initialization. This focus on real-world firmware makes the findings directly applicable to embedded systems and IoT hardware that underpin modern supply chains.
