Risk-sensitive Reinforcement Learning-based Strategies for Dutch Implicit Balancing

Abstract

Adopting renewable energy sources (RES) can pave the way toward reaching net-zero carbon emissions. However, the intermittent nature of RES can pose significant challenges to balance responsible parties (BRPs) and transmission system operators (TSOs) in maintaining the balance of the electricity grid. BRPs can assist TSOs in balancing the grid by occasionally deviating from their nomination to help reduce the system imbalance, which is called implicit balancing. In this paper, we propose data-driven implicit balancing strategies for BRPs in the Dutch imbalance settlement mechanism. Dutch implicit balancing is challenging due to the Dutch imbalance pricing calculation, which is a combination of single and dual pricing methods. To cope with this challenge, a risk management perspective is incorporated into the proposed method through distributional reinforcement learning. Distributional reinforcement learning agents are trained to manage a BRP's battery in the presence of wind farm generation stochasticity to reduce its imbalance cost. Dutch imbalance data of 2024 are used to assess the performance of the learned implicit strategies. The proposed method is benchmarked against deterministic model predictive control and a rule-based controller. The results show that both risk-neutral and risk-averse agents improve daily profit by 29.3% and 20.7 %, respectively, compared to the rule-based controller. Moreover, the risk-averse agent decreases the average portfolio deviation during dual pricing situations by 19.2% compared to the risk-neutral agent, resulting in a lower imbalance cost for the BRP in these situations.