Financial analysts in regulated industries face a double bind: they need to trust AI-generated answers before acting on them, but many institutions cannot send sensitive client data to external model providers. Existing chart-QA agents are accuracy-focused and opaque, and most assume proprietary API access. According to a paper on arXiv by Narayanan, Aravind, Raza, and Shaina, none of the current systems combine auditability with on-premise deployability without significant accuracy compromise. The researchers present AgentFinVQA, a multi-agent pipeline designed to solve this problem.
How the Multi-Agent Pipeline Works
AgentFinVQA decomposes each query into five discrete steps: planning, optical character recognition (OCR), legend grounding, visual inspection, and verification. Every step is recorded in a traceable Model Evaluation Packet (MEP) per sample. This design gives practitioners full insight into how an answer was derived, enabling trust and compliance in regulated settings.
The pipeline supports two backbone models: the proprietary Gemini-3 Flash and the open-weights Qwen3.6-27B-FP8, which can be served locally to maintain data residency.
Quantified Accuracy Gains
On the FinMME benchmark, AgentFinVQA delivered statistically significant improvements. The table below shows the performance against a zero-shot baseline using the same backbone.
| Backbone Model | Zero-Shot Baseline | AgentFinVQA | Improvement | Statistical Significance |
|---|---|---|---|---|
| Gemini-3 Flash | 63.56% | 71.24% | +7.68 pp | McNemar p ≈ 1.1×10⁻¹⁶ |
| Qwen3.6-27B-FP8 (local) | Not explicitly stated | +4.84 pp over baseline | +4.84 pp | Not stated |
According to the paper, the open-weights system keeps most of the accuracy gains while enabling full data residency, making it a practical choice for institutions that cannot use cloud APIs.
Built-in Verifier and Error Analysis
AgentFinVQA includes a verifier module whose verdict serves as a useful confidence signal. Answers that the verifier confirms achieve 68.2% exact accuracy, compared to only 55.6% for revised (potentially corrected) answers. This allows human reviewers to prioritise low-confidence cases.
Error analysis revealed that three categories account for nearly two-thirds of all failures:
- Question misunderstanding
- Legend confusion
- Extraction error
The researchers note that these same categories are the least likely to be detected by the verifier, pointing to clear directions for future improvement.
On-Premise Deployment for Data Sovereignty
The ability to run AgentFinVQA with open-weights models on local infrastructure eliminates the need to send client data to external API providers. For financial institutions, trade finance desks, and other regulated entities, this directly addresses data privacy and sovereignty requirements. The researchers have released the code to support reproducible evaluation and further development.
The paper demonstrates that auditable, on-premise financial chart QA is practical. While the immediate application focuses on financial charts, the same architecture could extend to any domain that requires trustworthy, explainable analysis of visual data—such as customs documentation or supply chain compliance forms.
