Comment by Connor Galbraith and Paula Bendiek, Data Scientists at Energy Systems Catapult.

Energy Systems Catapult is part of the Ofgem SIF-funded RetroMeter project and is testing and developing methodologies for measuring Metered Energy Savings (MES) from home retrofits.

What are Metered Energy Savings from retrofit?

Deemed energy savings – for example, a change in the Energy Performance Certificate (EPC) after energy saving interventions such as retrofit – involve estimating how much energy is expected to be saved based on the measures installed, for example, the level of insulation, and how those measures are predicted to perform based on engineering-based calculations and laboratory testing.

On the other hand, MES look at the actual metered energy use, both gas and electricity, after the retrofit, and compared it to how much energy would have been consumed in that home during the post-retrofit period, had there not been a retrofit. This is called the ‘counterfactual’ energy use.

What are the benefits of Metered Energy Savings?

Quantified MES have a range of benefits across multiple stakeholders in the energy system:

• It can contribute as part of an overall retrofit evaluation, by helping to see if a retrofit has achieved what the householder and other stakeholders wanted it to achieve.
• Help facilitate and assure high-quality retrofits by holding each actor in the retrofit supply chain accountable for their work. For example, deemed savings may not capture poor installation, but MES would identify it in the metered data.
• Contribute to learning and research about the real-life performance of retrofits, in terms of what types of retrofit measures work best for different properties and occupancy behaviours.
• Help in the planning of our future energy system by estimating how much energy will likely be required when a large number of households transition to more insulated homes – information useful for both households and the wider energy grid.
• Leverage financing for retrofit, by providing more confidence in the energy savings that underpin returns for private sector investment, and additional certainty of measured outcomes for public sector funders. This enables these funders to pay for the performance and measurable value that they receive from a series of retrofit projects, facilitating further collaboration and allowing new “pay-for-performance” business models to emerge.

What types of households and retrofits could RetroMeter Metered Energy Savings Methodologies be applied to?

The work of Energy Systems Catapult for RetroMeter has primarily been focused on situations where metered gas is used pre-retrofit as the main heating source, and a smart meter has been in place for at least a year prior to the retrofit. This gas data is being used to develop counterfactuals for how much gas the household would have consumed in the post-retrofit period, had the retrofit interventions not taken place.

This counterfactual can be compared to the actual usage of gas post-retrofit.  Alternatively, if the household has switched to an electric heating source such as a heat pump, the counterfactual gas usage can be compared with the actual electric heating consumption post-retrofit, if sub-metered data for the heat pump is available. If we are only interested in the total energy saved due to the heat pump and fabric retrofit, this comparison can be done on a simple energy basis. The process is a little more involved if the energy savings from the retrofit need to be disaggregated from the heat pump, requiring some assumptions about the gas boiler efficiency, heating schedule, and daily Coefficient Of Performance (COP) values.

What are the methodologies being tested under RetroMeter?

Accounting for changes in weather – OpenEEmeter

A simple approach to metered energy savings might be to simply look at a household’s pre-retrofit energy use and compare it to a household’s post-retrofit energy use. However, this does not account for the fact that the winter after the retrofit might be colder or warmer than the winter before it.

OpenEEmeter is an MES methodology developed in the US by Recurve, currently maintained by the Linux Foundation, and with a number of successful international case studies. It accounts for the impact of weather on energy consumption using mean hourly external temperature and metered energy consumption in the pre-retrofit ‘baseline’ period, to fit regression models that also account for seasonal and other calendar effects. The most advanced version of this model does this on a daily basis, generating a counterfactual each day for what the energy use would have been given the weather conditions.

Accounting for society-wide changes to energy use – comparator methodology

Society-wide factors such as energy price changes causing people to cut back on their energy use, or changes to home heating practices during Covid-19 lockdowns for example, can also make simple before-and-after comparisons inaccurate for measuring metered energy savings.

The comparator methodology builds further on OpenEEmeter by comparing the energy use in the ‘candidate’ household post-retrofit, to energy use in the same period for similar households which have not had a retrofit. This can help separate out the energy changes due to retrofit from the energy changes happening in society more broadly. There are different ways of finding similar ‘comparator’ households – matching can be done based on:

• Property archetypes – candidate and comparator households having the same built form, property type, property age, EPC rating, and other qualitative factors;
• Total energy consumption during the baseline period – grouping households into quantiles based on their total annual energy consumption, and matching candidate households with comparators in the same category; or
• Energy consumption profile similarity – comparing the gas meter time series during the baseline period of the candidate household with the profiles of the comparator households directly in the same period.

Accounting for changes in internal temperatures in the home – physics methodology

Households may decide to keep their home at a warmer temperature after the retrofit than before the retrofit, because it cost more to warm their house before the retrofit and costs less after the retrofit. This is called ‘comfort take-back’.

The physics-based methodology uses internal temperature data post-retrofit and accounts for comfort take-back. It estimates the energy households would have consumed in the post-retrofit period to achieve the internal temperatures they had in the post-retrofit period, if they still had their pre-retrofit Heat Transfer Coefficient (HTC).

The HTC is a measure of the rate at which the heat generated in a home is typically lost through leakage. The pre-retrofit HTC is estimated by correlating the pre-retrofit weather with the pre-retrofit gas usage.

The model looks at both gas and electricity usage, as it assumes that a certain proportion of electricity usage generates heat in the home indirectly – including electric cooking and kitchen appliances, electronics, and lights. The model factors in solar aperture, estimated using weather data including the external temperature and the solar irradiance. The model also accounts for baseload gas usage, defined as the gas used for purposes other than space heating such as domestic hot water and cooking. These uses are calculated by looking at gas usage during warm weather in the pre-retrofit period. The model also assumes boiler efficiency is equivalent to the industry average.

How is the Catapult testing these methodologies?

This project has made use of anonymised metered gas data from Hildebrand, a smart meter data provider. The Catapult has used data from 2021-22 as the baseline period to generate and test a counterfactual for a reporting period covering 2022-23. It is assumed that no retrofit was performed in these households, therefore the metered data from the reporting period should be predicted well by the OpenEEmeter counterfactual. The testing work examines how closely the counterfactual data and the reporting period metered data align, providing an indicator of the accuracy of the modelling approach in real-world settings.

The results of this testing so far are evaluated in terms of:

• Bias – whether the reporting period predicted gas consumption is, on average, higher or lower than the metered consumption;
• Accuracy – how much the reporting period predicted and metered gas consumption differ, in either direction. This accuracy can be aggregated at daily, monthly or annual levels. Accuracy is measured using a statistic called the Coefficient of Variation of Root Mean Squared Error (CVRMSE), where a high CVRMSE indicates poor accuracy.

Findings so far – accounting for weather changes only (OpenEEmeter)

Because the reporting period winter (2022/23) was colder than the previous year, the OpenEEmeter model expects a higher average consumption over this period. However, the metered gas consumption over this period was lower than the baseline the year before. This is very likely due to the considerably higher gas prices over the reporting period (2022/23) than the year previous (2021/22).

Without adjusting for the discrepancies in energy prices and the consumer behavioural responses, the OpenEEmeter model would appear to show a substantial bias in its energy savings estimations. In terms of accuracy, OpenEEmeter by itself did not perform very well. The median CVRMSE was approximately 53% when looking at daily predictions, and when aggregating those predictions to an annual level, was still around 19%.

Findings so far – if we account for society-wide changes to energy use patterns (Comparator Methodology)

Findings: Comparison methodology, matching households on archetypes

Adding a comparison methodology on top of OpenEEmeter, with the matching of households being based on archetypes (same built form, property type, property age, EPC rating), does adjust for the discrepancies that would impact the bias of OpenEEmeter when underlying consumption behaviour changes. Surprisingly, it made the accuracy at the daily and monthly levels slightly worse, with a CVRMSE of 36% on a monthly basis, worse than the 34% figure for OpenEEmeter alone, but improved upon the annual performance with a CVRMSE of 18%.

Findings: Comparison methodology, matching households on overall energy consumption level

Matching households based on overall energy consumption quartile leads to a CVRMSE of 15% on an annual basis, a significant improvement over the 19% figure for comparison based on archetypes.

Findings: Comparison methodology, matching households on energy consumption patterns

Matching households based on the similarity of their baseline year gas meter time series both eliminates bias and improves accuracy. Additionally, aggregating the group of candidate households into a portfolio was found to further improve the accuracy by eliminating property-specific noise in their energy consumption.

It brings CRVMSE down to 9.4% for individual homes on an annual basis, and down to 5% on an aggregated group of 25 homes, on an annual basis.

Findings so far – if we account for comfort take-back (Physics Methodology)

There are two aspects to validate and test in the physics methodology: the HTC calculation, and the counterfactual energy use calculation (which uses that HTC as an input). To test the HTC calculation itself, the team used data from 15 SMETER homes, for which ‘co-heating’ HTC values are available and 100 Hildebrand homes. ‘Co-heating’ HTC is calculated by using a home as a heating laboratory for a period whilst the home is unoccupied.

The HTC was generated using the RetroMeter model using smart meter data and external temperature data only, and then compared with the co-heating HTC. The results indicated that the HTC model underpredicts slightly when internal temperature is not used. To mitigate this, we use a residual filter which seeks to eliminate days with low internal temperature (without knowing the internal temperature). The counterfactual energy demand calculation was tested on the SMETER homes.

Using the co-heating HTC to predict the energy use, results in a slight over-prediction of demand. Using the modelled HTC reduced this bias of overestimation but decreases the accuracy (increasing the CVRMSE). The mean monthly CVRMSE of the end-to-end pipeline, which used the modelled HTC as input to the energy demand model, is 36%Both models will be validated on 15 additional SMETER homes when those become available.

Summary

Way forward

During the final months of the project, the Catapult will be investigating how the three methodologies can be combined into a single framework through which MES can be measured, enabling changes in temperature, energy price, and behavioural factors leading to comfort take-back to be evaluated. This will set the stage for further work beyond RetroMeter, where the Catapult hope to turn the methodologies presented here into a fully integrated open-source package for others to use and experiment with, ultimately opening up MES assessment to the broader community for feedback and evaluation.