Continuous actor-critic reinforcement learning methods often produce policies with erratic, high-frequency oscillations that render them unsuitable for physical deployment, according to a new paper published on arXiv. The research team — Lee, Jeong Woon, Kwak, Kyoleen, Kim, Daeho, and Hwang, Hyoseok — argues that existing approaches combat this by directly regularizing the policy's output, which treats the symptom rather than the cause. Their work offers a theoretical foundation and a practical framework called PAVE (Policy-Aware Value-field Equalization) that targets the root of the problem.
The Root Cause of Policy Non-Smoothness
The authors establish that policy non-smoothness is fundamentally governed by the differential geometry of the critic. Using implicit differentiation on the actor-critic objective, they prove that the sensitivity of the optimal policy is bounded by the ratio of the Q-function's mixed-partial derivative (noise sensitivity) to its action-space curvature (signal distinctness). This insight shifts focus from the actor to the critic as the source of oscillation.
PAVE: A Critic-Centric Regularization Framework
To empirically validate their theoretical insight, the researchers introduce PAVE (Policy-Aware Value-field Equalization), a critic-centric regularization framework. PAVE treats the critic as a scalar field and acts to stabilize its induced action-gradient field. The method works by minimizing Q-gradient volatility while preserving local curvature. Importantly, PAVE achieves this without modifying the actor network — only the critic is regularized.
Experimental Results
Experimental results reported in the paper demonstrate that PAVE achieves smoothness comparable to policy-side smoothness regularization methods, while maintaining competitive task performance. This suggests that addressing the critic's geometry can yield smooth policies without the need for direct actor regularization, potentially simplifying training pipelines and improving deployment readiness.
Implications for Physical Deployment
The ability to produce smooth, low-oscillation policies is critical for systems that interact with the physical world, such as robots, autonomous vehicles, and industrial controllers. By stabilizing the learning signal at its source — the critic's gradient field — PAVE offers a path to more reliable and safer reinforcement learning applications.