Constraint-aware reinforcement learning for energy-efficient and regulation-compliant wastewater treatment

Abstract

This paper addresses the sustainable and regulation-compliant control of wastewater treatment plants (WWTPs) characterized by nonlinear process dynamics, external disturbances, and multiple, often competing, effluent-quality constraints (e.g., biochemical oxygen demand, ammonia, nitrate, and phosphorus limits). We propose an Adaptive Lagrangian Soft Actor–Critic (AL-SAC) algorithm that incorporates a Lagrangian relaxation term into the maximum-entropy SAC objective in order to impose hard bounds on these key effluent variables. Lagrange multipliers are updated online based on instantaneous constraint violations, eliminating manual penalty tuning and enhancing convergence stability. The proposed AL-SAC controller is evaluated on the Benchmark Simulation Model No. 1 (BSM1) and provides up to a 23.9% reduction in energy consumption (i.e., aeration and pumping), improves the effluent quality index (EQI) by 2.92%, and significantly reduces the period over which total nitrogen (TN) and soluble ammonium nitrogen (SNH) concentrations exceed regulatory thresholds. These results demonstrate AL-SAC’s promise for energy-efficient and fully compliant WWTP operation.