Traditional reinforcement learning from verifiable rewards (RLVR) relies on a single binary verification signal, ignoring the rich structured feedback available in formal reasoning environments. Researchers have demonstrated that the Lean proof assistant can serve as a symbolic process oracle, providing both outcome-level and fine-grained tactic-level verified feedback during training. This approach, detailed in a recent arXiv preprint, processes proof attempts into sequences of tactics, with Lean's type-theoretic elaboration identifying both locally sound steps and the earliest failing step. The resulting dense, verifier-grounded credit signals enable more precise training of language models for theorem proving.
The Lean Proof Assistant as a Process Oracle
Lean, an interactive theorem prover, underpins this method. According to the paper, “Proof attempts are parsed into tactic sequences, and Lean's elaboration marks both locally sound steps and the earliest failing step.” This yields dense, verifier-grounded credit signals rooted in type theory. Unlike binary correct/incorrect rewards, this tactic-level feedback provides a process-based reward that is both sound and dense. The researchers note that symbolic proof assistants are not only evaluative verifiers but can also act as process-level reward oracles during training.
Symbolic proof assistants are not only verifiers at evaluation time, but can also act as process-level reward oracles during training.
Training Methodology with GRPO-Style Objectives
The training objective incorporates structured rewards into a GRPO-style reinforcement learning framework. The paper introduces first-error propagation and first-token credit methods that balance outcome- and process-level advantages. This allows the model to learn from partial successes rather than only final outcomes. The experiments leverage two theorem-proving systems: STP-Lean and DeepSeek-Prover-V1.5.
Experimental Results and Benchmarks
The study reports that tactic-level supervision outperforms outcome-only baselines in most settings. Benchmarks include MiniF2F and ProofNet, standard datasets for formal mathematical reasoning. The paper states: "Experiments with STP-Lean and DeepSeek-Prover-V1.5 show that tactic-level supervision outperforms outcome-only baselines in most settings, delivering improvements on benchmarks such as MiniF2F and ProofNet."
| Metric | Outcome-Only Baseline | Tactic-Level Supervision |
|---|---|---|
| Performance on MiniF2F | Baseline | Improved |
| Performance on ProofNet | Baseline | Improved |
| Training Signal Density | Binary (correct/incorrect) | Dense (per-tactic) |
| Feedback Granularity | Outcome-level | Process-level |
Broader Implications for AI Reliability
Beyond empirical gains, the paper highlights a broader perspective: symbolic proof assistants can act as process-level reward oracles during training. This opens a path toward reinforcement learning frameworks that combine the scalability of language models with the reliability of symbolic verification for formal reasoning. For enterprise technology leaders, this research points toward more trustworthy AI systems in domains requiring formal verification, such as contract analysis, compliance checking, and supply chain logic. The ability to provide dense, verifier-grounded feedback during training could significantly improve AI performance in tasks where correctness is critical.