Flexibility first: How EDF is using half-hourly data to reshape the energy industry

MHHS is disruptive, but it comes with opportunities, says EDF’s Alex Read

Regulation is often seen by industry as a burden, but for the fleet of foot it can be a positive driver of innovation. A case in point is the UK energy sector, where Net Zero and flexible consumption are explicit policy goals, and Market-wide Half-Hourly Settlement (MHHS) is a key plank in their delivery.

"MHHS is disruptive, but it comes with opportunities - particularly around flexibility, forecasting accuracy and the tariffs we can offer customers," Alex Read, senior enterprise product manager, data, at EDF UK, explained. "We're using regulatory change as an opportunity to be more innovative and drive innovation."

MHHS is an industry-wide programme led by energy regulator Ofgem. It aims to provide a more accurate picture of electricity consumption across the UK based on half-hourly data provided by modern meters.

From a technology perspective, MHHS is a big data challenge. The move from coarse-grained, ad hoc measurements to millions of readings processed at 30-minute intervals has meant an explosion in data volumes and analytical complexity.

But meeting these requirements has allowed EDF to improve its forecasting and analysis across all areas of the business, including retail, wholesale, finance and regulatory reporting.

"The obvious change is that we now need to collect and analyse data every 30 minutes - but it also fundamentally changes how we forecast, price and hedge energy," Read told Computing.

"Reducing forecasting error by even one or two percent in a £10 billion market is not a small number."

EDF combines weather data from the Met Office, pricing data and other metrics from across the industry, and brings these together with half-hourly meter data.

It's not just about matching supply and demand. Combining datasets allows the company to reduce operational risk and offer new services to retail and wholesale customers.

Data platforms

As data has become ever more central, so have the platforms that sit behind it. EDF UK recently moved its 5.5 million customer accounts onto Kraken, the platform spun off from Octopus last year, which "is fast becoming a major industry player in the utilities market," according to Read.

Critical data from Kraken is now hosted on Snowflake's Data Cloud, which also supports all retail, wholesale, finance and regulatory reporting.

EDF UK has been a Snowflake customer for the last four years, adopting it after its previous platforms were showing signs of strain under increasing data volumes and analytical requirements, Read explained.

"All of our critical data from Kraken is hosted on Snowflake, and all of our analytical and reporting products are built from there."

Data marketplace

Like other data-centric organisations, energy companies are becoming part of wider data hubs, rather than just data owners. For forecasting, EDF pulls in data from the Met Office and others via the Snowflake Marketplace. It also shares data internally between retail and wholesale divisions, and externally with auditors and regulators.

While it helps that many organisations are also on the Snowflake Marketplace, Read is aware of the need to retain operational independence.

"Snowflake is a heavily integrated platform, and we also host on AWS. From an ETL perspective, we've got Matillion and dbt. We've also got integrated data catalogues as well.

"So while Snowflake is the heart of our data platform, it also integrates with other capabilities that do a lot of the heavy lifting. That gives us confidence that we've got a robust platform."

New services

From a consumer perspective, half-hourly data and near real-time analytics allow suppliers to design tariffs that encourage behaviour that reduces peak-time pressure on the grid and rewards customers for it.

For example, EDF's Sunday Saver Challenge rewards customers with up to 16 hours of free electricity on Sundays if they shift weekday energy use away from periods of peak demand.

Another new tariff is FreePhase, which gives customers the option of moving heavy workloads (for example, charging an EV or doing the laundry) to cheaper periods, with occasional free electricity when the wholesale price is negative.

"Having the ability to understand consumption and a customer's ability to change their consumption patterns on a half-hourly basis, according to what the price is coming from the market, gives us the ability to really tailor flexibility-type services for the industry and our customers," said Read.

Customers' interactions with their accounts are increasingly mediated by AI, with new agents for Sunday Saver and for explaining bills.

For Read and his team, this is partly about managing expectations.

"I think our customers, executives and senior leadership teams are beginning to understand that, in order to adopt and operationalise AI at scale, you need sound data foundations. That means having all your data accessible, available to AI agents at a cadence that's acceptable to the operation, and at an acceptable level of quality."

He continued: "We're well on that journey. We've got a sound data platform, and the vast majority of our business-critical data sits on it. For me, the next step is making sure all of that data is well defined and of sufficient quality, so we can trust and operationalise AI agent experiences at scale."