For almost three decades, the UK energy sector has been pursuing the promise that improved data sharing will unlock a more efficient, fair and sustainable energy system. Yet we’ve remained trapped in a series of never-ending pilot projects. To break free, we must learn from repeated hype cycles and end our fixation on analysing barriers to innovation.
From early digital infrastructure like ElectraLink in the late 1990s, through consumer data access via project midata in the 2010s, to today’s visions of digital twins, the energy sector has seen a parade of white papers, consultations and pilot projects.
But fundamental challenges remain:
The repeated failures and reboots may be symptomatic of something more structural: the way hype circulates through energy innovation, creating cycles of mobilisation, disappointment, realignment and concealment.
In a recent ‘Researcher in Residence’ fellowship, hosted by Energy Systems Catapult and funded by Innovation Launchpad Network+, I set out to trace the hype surrounding energy data sharing. Here’s what I discovered.
Efforts to open up data in the UK energy system have been tied-up with market liberalisation since the 1990s, when organisations like Elexon and ElectraLink were established. They were initially designed to help utilities share information about their customers but were not intended as open or interoperable platforms.
This meant that in its first several years of operation, ElectraLink’s Data Transfer System had limited third-party access and did not allow for real-time data transfer via APIs.
Meanwhile, Elexon’s Balancing Mechanism, which is used to balance electricity supply and demand in each half-hour trading period, has had its governance model and remit repeatedly questioned by Ofgem. There were even accusations of unfair market advantage for some electricity generators. Ultimately, the system’s lack of transparency on data access has prevented regulators, journalists and researchers from fully understanding its inner logic.
By the 2010s, new priorities emerged. The government-run midata programme was supposed to empower consumers by giving them access to their energy use data in machine-readable formats. Machine-readable formats were crucial because they would help consumers switch providers and integrate them into digital services – so the argument went. But the initiative stalled because of poor quality data, a lack of incentives to share data and patchy participation among suppliers. Ofgem ultimately paused the programme in 2020.
More recently, the focus has shifted from retail data to system-wide operational data. This has been brought about by the rise of distributed, renewable energy supply and flexibility markets which have made access to grid and operational data a central issue.
In 2019, Energy Data Taskforce gave us the landmark ‘Strategy for a Modern Digitalised Energy System’. It proposed a principle of ‘presumed open’ for all non-sensitive energy data.
The government largely endorsed these recommendations, mandating Open Data Triage, a unified process for assessing security of data, metadata and associated software scripts. It also ordered the publication of aggregated smart meter consumption data – requiring DNOs to make this data as accessible as possible. It’s a good start for sharing insights on residential consumer data but more work is needed.
Despite these efforts, structural issues have persisted:
In 2024, Ofgem launched a new phase of activity, consulting on plans for a national Data Sharing Infrastructure (DSI). This technical architecture and governance mechanism should provide interoperability and trust across the energy system:
My research shows how the DSI has been linked to the development of energy system ‘digital twins’. Energy Systems Catapult defines them as “digital models which integrate two-way data flow between the model and physical object or system”.
When the Digital Twin Hub was established in 2020, the initial hype around digital twins mobilised people and resources and led to a rash of pilot projects. But the hype soon faded to disappointment as people struggled to access the necessary data.
In an interview I conducted for my research, I was told: “There’s a long trail of open datasets. However, without smart meter data and high-quality building asset registers, you’re just looking at low value use cases.”
The challenge of data sharing was quietly reframed. We moved away from defining digital twins as models to understanding them as data sharing infrastructures.
Despite this change, industry and academia are still stuck discussing data accuracy, trust in outputs, and model uncertainty. Instead, we need action on the specific issues related to data sharing.
I recommend:
This moment is thick with hype. Phrases like ‘seamless data sharing’ and ‘together we can all innovate’ echo earlier waves of reform. Those studying the hype surrounding science and technology have long argued that such future-facing narratives do not only serve as communication tools, but also as performative scripts that justify investment, reshape institutional priorities and marginalise more cautious voices. Hype is not inherently bad but the quiet realignment of priorities we are currently witnessing could prevent us learning the lessons of failure from past data sharing initiatives.
I hope I’ve convinced you that the UK’s repeated struggles with energy data sharing reveals the limits of repeatedly analysing ‘barriers to innovation’.
The UK energy sector has been here before. But perhaps this time, the lessons of history – the false starts and broken feedback loops – can guide a different outcome. One that breaks the sector free from a series of never-ending pilot projects.
As we look to the future of UK energy Data Sharing Infrastructure, we should carry two things with us: an ambition to build systems that are open, fair, and future ready. And a humility to learn from the past.
Dr Ola Michalec joins Energy Systems Catapult’s Value in Energy Data series to discuss the development of digital twins in the energy industry
Watch the video