System-Aware Reinforcement Learning for Optimized Implicit Imbalance Participation in Belgium

Abstract

The increasing integration of renewable energy sources into electrical grids has disrupted the balance between production and consumption. To address this challenges, transmission system operators such as the Belgian one have introduced imbalance tariffs that penalize harmful energy deviations. Although the imbalance settlement mechanism allows balance responsible parties to dynamically adjust their energy positions, it also exposes them to significant risks. In fact, Belgian imbalance prices are determined retrospectively at the end of each settlement block, meaning that energy deviations occur under uncertain pricing conditions. Reinforcement learning (RL) offers a promising solution for navigating this uncertainty thanks to its ability to manage stochastic environments and deliver long-term rewards. However, achieving profitable participation in imbalance settlement requires more than just handling price volatility; it also demands a deep understanding of the grid dynamics. This paper examines how enriching an RL agent's observation space with grid-related data can enhance its awareness of system dynamics and improve decision-making. We specifically focus on the agent's performance during unstable periods - i.e., quarters where last-minute deviations in system imbalance heavily influence prices - by introducing a related metric. Using the soft actor-critic algorithm, we control a simulated battery energy storage system participating in the Belgian imbalance settlement, leveraging historical data spanning three years. Our findings indicate that, compared to a system-agnostic RL agent (i.e., an agent that does not have grid-related values in the observation space), the system-aware agents develop more effective policies, particularly during unstable quarters.