Reinforcement Learning for BESS

Overview

In my Master’s project at HTWG Konstanz, I investigated the use of Deep Reinforcement Learning (DRL) for the optimal control of a Battery Energy Storage System (BESS) under uncertain electricity price and demand conditions.
→ Download Report (PDF)
→ View Code on GitHub

A custom simulation environment was developed in Python using Gymnasium, modeling battery dynamics including State of Charge (SoC), State of Health (SoH), and battery degradation while incorporating forecasted electricity prices and demand data together with their associated uncertainties.

The project focused on two energy management scenarios:

• Energy Arbitrage (buying and selling electricity based on market prices)
• Peak Shaving (reducing load peaks using battery storage)

In addition to a rule-based baseline controller, multiple RL agents (DQN, TD3, and QR-DQN) were implemented and evaluated under uncertain operating conditions.

Methods & Tools

• Simulation Environment: Python, Gymnasium
• RL Framework: Stable-Baselines3
• RL Algorithms: DQN, TD3, QR-DQN
• Data Processing: Pandas, NumPy
• Visualization & Evaluation: Matplotlib, TensorBoard
• Battery Modeling: SoC/SoH dynamics, cycle aging
• Environment: JupyterLab, Conda

Results

• RL-based approaches achieved higher revenues in the arbitrage scenario compared to the rule-based controller.
• QR-DQN achieved the best overall performance due to its robustness against uncertainty in forecasted data.
• In the peak-shaving scenario, DQN and QR-DQN reduced load peaks more effectively than the rule-based approach and TD3.
• The project demonstrates how DRL can improve decision-making for energy storage systems operating under uncertain market conditions.