Comment by Christopher Lee, Digital & Data Associate at Energy Systems Catapult.

Unlike Machine Learning (ML) and Artificial Intelligence (AI), which have become ubiquitous terms used by the general public, Reinforcement Learning (RL) is far from a mainstay in everyday vocabulary.

RL is a branch of ML that focuses on optimal decision making through trial-and-error; it can, for example, reduce energy consumption or lower the carbon emissions of a system. In our latest report, Prospects for Reinforcement Learning, we provide an overview of RL and highlight the landscape for near-term and future RL applications in the energy sector through a series of industry and academic use cases.

Drawing inspiration from other sectors

RL gained recognition for beating world champions in the game of Go, with Google DeepMind’s AlphaGo. More recently, RL helped propel ChatGPT into the mainstream by optimising the base language model, making it better align with user intent from human feedback.

Regardless of your views on the societal implications of ChatGPT or the obscure value of playing games, these breakthroughs in RL should serve as inspirational proof-of-concepts for the potential of RL in tackling real-world challenges, such as climate change.

Drawing from these inspirations is an important part of societal advancement. Taking the space industry as an example, the technologies developed have a spinoff effect into commercial products that have transformed our daily lives, such as GPS navigation and memory foam.

The landscape for RL applications in the energy sector

RL is an immature technology in comparison to other ML methods; as such, its penetration into productionised industry applications—especially in the energy sector—are scarce. Nonetheless, we see some distinct use cases for the near-term and future, which is illustrated in the figure below. Like the space industry, inspiration from AlphaGo and ChatGPT can influence various applications within the energy sector at varying maturity, which we dive into below:

Near-term applications

Energy companies face similar general business problems to other sectors. It’s easy to get caught up in the exciting ‘energy specific’ uses of data science and ML and overlook how learnings from other sectors can indirectly benefit energy.

A recent report from Energy Systems Catapult on the data science skills gap in the energy sector highlighted how various energy companies are applying data science. The most common was forecasting—which is the archetypal application of ML in energy.

Similarly, an academic literature review of AI techniques in energy applications showed RL was predominantly applied to the control of energy management systems—the archetypal equivalent for RL in energy. However, if we kept a narrow view of ML or RL in the energy sector, we would miss the benefits from using natural language processing (NLP) to understand and remedy customer complaints within consumer affairs teams in these energy companies, as mentioned in the data science skills gap report.

A similar argument could be made for ChatGPT. Though the main value of ChatGPT comes from the generative AI component (i.e., generating text responses), the RL from human feedback (RLHF) component plays a critical role in optimising/fine-tuning the generative AI model to align with the users’ intent.

ChatGPT is likely to transform all industries, but the use of RLHF for fine-tuning could also enable energy companies to tailor models more precisely to their specific customer base, provided that sufficient and relevant data is collected. Additionally, other areas where energy companies can potentially draw inspiration from include recommender systems and financial trading. Drawing from the learnings of other sectors is crucial as the energy sector typically lags when it comes to data science and ML.

RL has already started to directly impact the energy sector through industrial/commercial control systems, such as DeepMind’s commercial cooling systems and Carbon Re’s cement production. Though it is still early days as most implementations of RL are pilot projects or not yet fully autonomous. However, the use of RL in ChatGPT showcases how RL can have a profound impact on industries and society today, giving hope to the pilot projects and research in the energy sector.

Future applications

Domestic energy management and large-scale energy systems applications tend to remain in the research phase, as there are stricter safety, regulatory and security hurdles to overcome. Interestingly, methods used in AlphaGo and ChatGPT have been adopted in energy research, which set two paradigms.

Namely, AlphaGo focused on removing human influence to achieve superhuman performance in gameplay, whereas ChatGPT did the opposite by incorporating human feedback to improve language understanding. Both approaches exploit several of the main advantages of RL—e.g., handling uncertainty, complex systems, and human feedback—and have been explored in energy research:

• AlphaGo: As the energy system becomes increasingly complex, with the uncertainty of intermittent renewables, it is becoming more difficult for human operators to manage the grid. Similar techniques to AlphaGo were employed in scheduling power generator dispatching and running a power network to surpass human operator performance.
• ChatGPT: As we continue to drive down energy demand, we potentially can adversely affect user comfort. In the context of building heating, a user can potentially use an app to indicate their thermal comfort and the RL agent could learn this perceived comfort from the human feedback, like ChatGPT.

The energy sector’s transformation may pose a challenge for traditional methods. However, RL offers a flexible and powerful alternative that can potentially keep up with the sector’s growing complexity.

As RL continues to evolve, drawing inspiration from other sectors and applications will be crucial to realise its full potential in the energy sector.