Quality assurance of administrative data report for electricity and gas, August 2017

1.1 Background

National Accounts and Economic Statistics Directorate (NAES) within Office for National Statistics (ONS) requests data from the Department for Business, Energy and Industrial Strategy (BEIS) on the electricity, gas, steam and air conditioning industry. These data form one source in the calculation of short-term economic output indicators namely gross domestic product (GDPO)) and Index of Production (IoP) for the UK.

This report outlines the process data take from initial collection through to the output of the release. It identifies potential risks in data quality and accuracy as well as details of how those risks are mitigated.

This report forms the latest of a series of quality assurance of administrative data (QAAD) reports produced by NAES to investigate the administrative data sources we use in the production of short-term economic output indicators as set out by the UK Statistics Authority. As such this report specifically focuses on our administrative data use for the electricity and gas industry and does not aim to cover other aspects of Standard Industrial Classification: SIC 2007 division 35, (SIC_35), for example, SIC 35.3. Separate industries where we utilise administrative data will be considered in other QAAD reports in the series.

Further information relating to quality and methodology for the short-term economic output indicators can be found in the UK Index of Production Quality and Methodology Information.

1.2 Standard industrial classification (SIC) overview

The electricity, gas, steam and air conditioning industry covers all activities under UK SIC 2007 division 35. This division includes electricity activities such as the production, distribution and trade of electricity. It also includes the manufacture and distribution of gas and the supply of steam and air conditioning.

Based on the UK Standard Industrial Classification 2007 the industry is classified to three groups:

• 35.1 - Electric power generation, transmission and distribution
• 35.2 - Manufacture of gas; distribution of gaseous fuels through mains
• 35.3 - Steam and air conditioning supply

According to the Inter-Departmental Business Register (IDBR) ¹ there were 4,535 enterprises classified under division 35 in March 2016. This is an increase of approximately 965 enterprises (27.0%) from the previous year (March 2015).

The majority of enterprises within division 35 were allocated to “35.1 – electricity” ; which equates to 4,420 enterprises (97.4% of the total division).

Within division 35, there were 3,665 enterprises with fewer than five employees (80.8% of the division), compared with just 100 enterprises with 20 or more employees (2.2% of the division).

Of the 4,535 enterprises in division 35, the annual turnover of 2,655 (58.5%) enterprises was below £250,000.

For the vast majority of division 35 enterprises, 4,105 (90.5%) had registered turnover below £2 million, with only 75 enterprises in total having registered turnover of over £50 million.

2.1 UK Statistics Authority QAAD Toolkit

The assessment of our administrative data sources has been carried out in accordance with the UK Statistics Authority Quality Assurance of Administrative Data (QAAD) Toolkit.

Each administrative data source investigated has been evaluated according to the toolkit’s risk and profile matrix (Table 1) reflecting the level of risk to data quality concerns and the public interest profile of the statistics.

The toolkit outlines four specific areas for assurance and the rest of this report will focus on these areas in turn. These are:

• operational context and administrative data collection
• communication with data supply partners
• quality assurance principles, standards and checks applied by data suppliers
• producer’s quality assurance investigations and documentation

In the assurance of our data source, we have chosen to give a separate risk and profile matrix score (Table 1) for each of the four areas of assurance. This will allow us to focus our investigatory efforts on areas of particular risk or interest to our users. Scoring for the electricity and gas industry is shown in Table 2.

2.2 Assessment and justification against the QAAD risk and profile matrix

The risk of quality concerns and public interest profile have both been set as “Medium” due to the contribution that electricity and gas statistics feed into Index of Production (10.4%) and gross domestic product (1.5%). As such, a score of A2 (enhanced assurance) is deemed appropriate for this data source.

All scoring was carried out by National Accounts and Economic Statistics Directorate (NAES) based on the level of risk of the data and interest of our users. Results for each area of assurance for electricity and gas are shown in Table 2 .If you feel that this report does not adequately provide this level of assurance or you have any other feedback, please contact stoi.development@ons.gov.uk.

Notes for: Quality assurance of administrative data assessment

1. The Inter-Departmental Business Register (IDBR) is a comprehensive list of UK businesses that is used by government for statistical purposes. It provides the main sampling frame for business surveys carried out by both ONS and other government departments. It is also a main data source for analyses of business activity.

3.1 Operational context and administrative data collection (QAAD matrix score - A2)

This relates to the need for statistical producers to gain an understanding of the environment and processes in which the administrative data are being complied and the factors that might increase the risks to the quality of the administrative data.

3.1.1 Electricity power generation, transmission and distribution (SIC 35.1)

The Department for Business, Energy and Industrial Strategy (BEIS) collect data on fuel use, electricity generation and supply by fuel type, sales of electricity, and imports and exports of electricity on a monthly basis. BEIS can request the data by law in accordance with the Electricity Act of 1989, section 98. They are published monthly, quarterly and annually, the latest publication can be found at Digest of UK Energy Statistics (DUKES): annual tables.The major strength of the electricity data provided by BEIS is its legislated comprehensive coverage, particularly for generation, with data collected accounting for around 99% of sales.

Data are collected via the following methods.

Major Power Producers (MPP) supply data on fuel use, electricity generation and major sales, which are collected by BEIS via a monthly survey and a census of approximately 35 companies surveyed electronically, Submissions are then returned to BEIS via the oil portal, a secure transfer system used instead of emails. There are no built-in data checks on the portal as BEIS conduct their own checks on the data received. BEIS share aggregate data collected with National Accounts and Economic Statistics Directorate (NAES) and other downstream users, however, they do not supply raw data to protect individual companies from potential disclosure. As this is a survey it falls outside of the scope of the assurance of data from administrative sources.

Elexon (an electric utility company) supply monthly import and export data, collected by a metering system to BEIS, via the National Grid. In addition the National Grid also supply data on generation by fuel type, these data cover Great Britain and the National Grid data (it does not included embedded generation ) and are used to forecast current month generation figures. The deadline for BEIS to deliver the month’s figures to NAES for the Index of Production (IoP) is around the 25th day of each month after the reference period, therefore BEIS estimate the remaining days left in the month by up-scaling the data to give a full month’s figure and apply growth factors to forecast upcoming months. BEIS do this by collecting half-hourly data from the National Grid.

The final part of data collected for electricity is Autogeneration ³ fuel use and supply, which is collected on a quarterly basis via an Office for National Statistics (ONS) survey (Electricity Generated Inquiry) and therefore falls outside of the scope of the assurance of data from administrative sources.

BEIS confirmed there have been no changes to data collection, data quality or methodology procedures and the statistics are produced independently from political decision-making and policy-makers as set out under the Code of Practice for Official Statistics. Further, there are no related targets, incentives or pressures either from the public, media or Parliament, which may impact the data.

However BEIS and NAES have noted three potential limitations users should be aware of when using this data.

Firstly, the fuel use generation and major sales data deadline falls after the deadline to submit data to NAES. Practically, this means 65 to 70% of sales data are supplied to NAES and the remaining sales data are then estimated for current and forecast months. The missing sales data is commonly due to large companies who are slow in returning data.

Whilst users should be aware of this limitation in the completeness of the dataset, 65 to 70% of sales data is still a large proportion of the industry to estimate a final figure and can be understood in the context of strong user demand for the most timely production of gross domestic product (GDP(O)) and Index of Production (IoP) and the relative limited weight that this industry feeds into them. The effect of this is mitigated by the submission of detail briefings from BEIS to NAES explaining data trends.

Secondly, tight deadlines that lead to short turnaround times for BEIS to conduct quality assurance of current month data. Timings allow them 2 to 3 days in which they run basic validation checks to minimise the sources of uncertainty, including checking of the data and remedying issues. However, there is the potential that this could still impact on quality that more time would reduce. As described in the previous point, quality must be balanced with timeliness considering the user demand for the most timely production of GDP(O)) and IoP and in this context validation checks currently run are deemed proportionate relative to the weight this industry composes and time available.

Thirdly, the National Grid generation data is used by BEIS as a comparative index, using this dataset BEIS then impute data for missing values, these are usually wind or solar returns, in particular where a company has a large number of sites to report on, which eventually leads to recurrent revisions. Wind and solar are seasonal, wind makes up larger proportion of supply in the winter months and solar a larger proportion of supply in the summer months although solar is still a small proportion of total supply. On average in 2016, wind made up 9.6% of total supply and solar 0.6% of total supply. BEIS recognises this is a growing area of electricity generation and are currently developing better ways to measure this sector.

3.1.2 Manufacture of gas; distribution of gaseous fuels through mains (SIC 35.2)

The Department for Business, Energy and Industrial Strategy (BEIS) collect 100% of the data for the manufacture of gas and includes data from all distribution networks in Great Britain (listed in this section) and companies in Northern Ireland. However, as noted previously, the BEIS deadline for gas and electricity submission for the month falls after the deadline to submit data to NAES. For gas this means 70 to 100% of gas output data are supplied to NAES and any remaining data are then estimated for the current month using National Grid entry point data.

BEIS receive a monthly gas report from each of the four gas distribution networks:

• Northern Gas Networks
• National Grid (Distribution Networks)
• Southern Gas Network (SGN)
• Wales and West Utilities

along with an overall National Grid (National Transmission System) report for gas output into the transmission system from Great Britain and a monthly report from the Isle of Grain Liquefied Natural Gas terminal for gas sent to Southern Gas Network not via National Grid (National Transmission System).

BEIS also receive two monthly returns from two electricity generation companies in Northern Ireland and two quarterly returns from Northern Ireland, this uplifts the gas sent out figure to a UK level rather than Great Britain level.

Data are submitted to BEIS on a consistent basis every month and can be requested by law in accordance with the Statistics of Trade Act of 1947. The deadline for submission is the last day of the month following the reference month.

BEIS publish this data on a monthly basis but for the purpose of the Index of Production BEIS convert these figures into weekly gas sent out figures, using a 4-4-5 calendar approach this converts calendar months into statistical months where they are 4 or 5 weeks in a month.

BEIS confirmed there have been no changes to data collection, data quality or methodology. The statistics are independent from political decision-making and policy-makers. There are no related targets, incentives or pressures either from the public, media or Parliament, which may impact the data.

Strengths

BEIS can request data by law in accordance with Electricity Act of 1989, section 98 and for gas Statistics of Trade Act 1947.

Coverage is 99% of the UK electricity market.

Coverage is 100% of the UK gas market.

Consistent data collection data quality and methodology procedures produced under the Code of Practice for Official Statistics.

The statisticians within BEIS have a wide knowledge and understanding of the energy industry.

Weaknesses

Tight turnaround times to meet GDP(O) and IoP deadlines.

Estimation of some BEIS sales data deadline falls after the deadline to submit data to NAES for GDP(O) and IoP. Practically means 65 to 70% of electricity and 70 to 100% of gas sales data are supplied to NAES, the remaining sales data are then estimated.

Limited measurement (only imputation) of wind and solar electricity generation.

Next steps

NAES to maintain this level of knowledge of this area.

3.2 Communication with data supply partners (QAAD matrix score A2)

This relates to the need to maintain effective relationships with suppliers (through written agreements such as service level agreements or memoranda of understanding), which include change management processes and the consideration of statistical needs when changes are being made to relevant administrative systems.

3.2.1 Electricity power generation, transmission and distribution (SIC 35.1)

The Department for Business, Energy and Industrial Strategy (BEIS) have established and maintained collaborative relationships with their data suppliers. They have written agreements with all suppliers regarding the submission of fuel use and generation data, with the timetable to be followed supplied by BEIS at the start of each year. The agreements includes companies agreeing to provide accurate and up-to-date data under the legal requirements of the Electricity Act 1989, this outlines the obligation of electricity licence holders in providing statistical information to the Secretary of State as required.

BEIS formally meet with the six largest sales companies:

• British Gas
• EDF Energy
• E.ON UK
• Npower
• Scottish Power
• Scottish and Southern Energy Plc (SSE)

every 2 years to discuss methodologies and any issues with their data .They also hold contact details for all companies for more informal contact when they require further explanation of the data provided.

They organised a user engagement event in 2017 to speak to data suppliers and users of their data including how and why they use it.

BEIS publish a methodology paper detailing the data collection and working methodologies, although this has not been updated since 2009 due to no changes to the data collection, data quality or methodology.

National Accounts and Economic Statistics Directorate (NAES) have established collaborative relationships with BEIS and these are maintained through formal meetings once a year. Details of data sharing are also formalised in a service level agreement, however, this is currently out of date since October 2015. This is due mostly to a shortage in resources due to other business priorities and though this is not ideal the potential for risk to data quality is low as data collection and data quality methodology procedures have not changed and communication between NAES and BEIS is strong. NAES are currently working on a new service level agreement with BEIS and should be in place shortly after this QAAD review.

NAES also have other forms of informal contact with BEIS including regular monthly emails, detailed briefings and phone calls to chase up any queries, with BEIS being quick to respond to queries.

3.2.2 Manufacture of gas; distribution of gaseous fuels through mains (SIC 35.2)

The Department for Business, Energy and Industrial Strategy (BEIS) has established and maintained collaborative relationships with National Grid through annual formal meetings and constant informal contact via email, allowing them to query data constantly.

BEIS do not have regular communication with the four distribution companies and the Northern Ireland companies.

BEIS do not have a written agreement with their gas suppliers. However, it has been company practice to provide these returns on a monthly basis to their department and have always delivered in a timely manner. National Grid and the other distribution networks have been providing these data returns for a few decades and based on this BEIS along with the data suppliers do not feel the need for a written agreement, however, the data is collected by law.

BEIS published a methodology note detailing the data collection and working methodologies, although this has not been updated since 2012 due to no changes to the data collection, data quality or methodology.

National Accounts and Economic Statistics Directorate (NAES) meet formally with BEIS once a year and there is a service level agreement in place, however, this is currently out of date since October 2015 for the same reasons as outlined in section 3.2.5 and is being updated. NAES confirmed other forms of contact with BEIS include regular monthly emails, detail briefings and phone calls and BEIS are quick to respond to their queries.

Strengths

BEIS have written agreements with all electricity energy suppliers regarding the submission of fuel use and generation data, with a timetable.

They meet the six largest energy sales companies every 2years to discuss methodologies and any issues with their data.

They have regular contact with National Grid and meet once a year.

BEIS conducted a user engagement event to speak to data suppliers and users of their data including how and why they use it.

BEIS provide monthly detail briefings covering electricity and gas trends.

BEIS published separate methodology papers detailing the data collection and working methodologies for electricity and gas although the papers have not been updated recently.

Weaknesses

Service level agreement between BEIS and NAES is currently out of date.

No communication or relationships established with the four gas distribution companies including Northern Ireland companies.

Next steps

Update of service level agreement with BEIS.

Opportunity for NAES to attend future user engagement events to gain a better understanding the energy industry.

3.3 Quality assurance principles, standards and checks by data supplier (QAAD matrix score A2)

This relates to the validation checks and procedures undertaken by the data supplier, any process of audit of the operational system and any steps taken to determine the accuracy of the administrative data.

3.3.1 Electricity power generation, transmission and distribution (SIC 35.1)

Once electricity data is submitted, the Department for Business, Energy and Industrial Strategy (BEIS) has automated checks to look at efficiencies, load factors and growth factors for sales data, along with automated checks to ensure that companies are returning data for Scotland and Northern Ireland where relevant. They check each fuel type efficiency ranges and look at factors to ensure the data received is comparable, particularly for wind and solar where efficiency is always 100%.

For sales, they look at growth factors and query large changes with the six largest sales companies (see section 3.2.1).

For each individual company they look at load factor for each fuel type they have used (that is, their generation as a proportion of how much electricity they could feasibly generate given their operating capacity).

For efficiencies they look at each company’s volume of fuel inputs against the fuel output in a monthly time series and query any fluctuations with the companies; a drop in efficiency is usually due to a plant outage.

The findings are recorded on an Excel spreadsheet and further checks are applied. These include comparisons with other data sources of electricity sales data, for example, Ofgem and other external sources such as the Cornwall Energy Independent Suppliers report in order to improve the coverage by picking up large and growing suppliers they may have missed. They also look at the growth factors for import and export data and query any potentially incorrect figures with Elexon and the National Grid.

BEIS compare the generation data with the National Grid and with the Bloomberg terminal to monitor UK generation to ensure their data looks comparable.

BEIS has recently improved the quality of data submitted by including guidance notes for the completion of all their returns and they make themselves available to answer queries on the process from companies.

BEIS revisions policy allows companies to update their previous month’s returns throughout the year; these revisions are regularly checked each month by BEIS. Most are automated checks, as there are 84 sites and incorrect data may slip through and can lead to incorrect revisions in the Index of Production, although this is now picked up before publication.

3.3.2 Manufacture of gas; distribution of gaseous fuels through mains (SIC 35.2)

The aggregated figure from UK returns is validated by automated checks against previous data collected; this involves checking month-on-month data through the Petroleum Processing Reporting System (PPRS). There are also validation checks for each gas terminal to ensure all data collected aligns.

They compare the PPRS data with national entry point data on a terminal basis. They compare data submitted by the National Grid on an aggregated basis. The aim is to publish the data for gas output into the transmission system within 0.5% of the gas available from indigenous production and imports collected from PPRS. They explained this is a BEIS policy to ensure that the statistical difference between supply (PPRS and BEIS data) and demand (National Grid) is within 0.5%; they achieve this consistently on a 3-monthly basis.

BEIS will also contact National Grid to confirm any suspicious returns and reasons for the differences.

Strengths

BEIS carries out automated checks for gas and electricity insuring consistency of checks month-on-month. Query any potentially incorrect figures with Elexon and the National Grid.

Compare data recieved against National Grid entry point data and data they receive from PPRS from individual gas companies.

BEIS improve data submitted by including guidance notes for the completion of all their returns and answer queries on the process from companies.

Weaknesses

Changes to methodologies and policies are reported back to NAES once a year.

Next steps

Service level agreement agreeing providing advance warning of any changes in their methods.

NAES to maintain understanding of quality assurance checks carried out by BEIS, offer assistance.

3.4 Producers quality assurance investigations and documentation (QAAD matrix score A2)

This relates to the quality assurance conducted by the statistical producer, including corroboration against other data sources.

3.4.1 Electricity power generation, transmission and distribution (SIC 35.1) and manufacture of gas; distribution of gaseous fuels through mains (SIC 35.2)

National Accounts and Economic Statistics Directorate (NAES) is the producer of energy data that feed into the Index of Production and gross domestic product; to proactively check data before delivery of electricity and gas they use a forecast tool to look at 5-year growth trends giving them an estimate of the expected figures. Using past Department for Business, Energy and Industrial Strategy (BEIS) extracts to predict monthly growths over time, these forecast checks look at:

• minimum growth over the last 5 months
• maximum growth over the last 5 months
• median growth over the last 5 months

Data is queried with BEIS if sufficiently different from the forecast result. Response rates are taken into account when querying the data. NAES also compares the revisions data they receive, checking month-on-month returns using Excel.

NAES has established regular quality assurance sense checks on the electricity and gas data, using Excel spreadsheets. The data source is checked by looking at historical trends and reading briefings supplied by BEIS.

The final output is quality assured by two senior managers within the team. There are clear desk instructions in place and the desk instructions are reviewed and updated on a regular basis. The systems are run each month for final output.

User engagement is continual; the feedback tends to relate to the overall impact of the statistics than to the individual data source used, to date no specific feedback on the use of electricity and gas statistics has been provided.

Strengths

Established quality assurance checks in place.

Desk instructions regularly reviewed.

Forecast tool in place to highlight any unexpected shifts in the data.

Notes for: Electricity power generation, transmission and distribution (SIC 35.1)

1 Embedded generation is the term used to describe the process of generating electricity at a specific location and then connecting that supply into the electricity network.

2 Autogenerators are businesses who generate electricity primarily for their own use, and sell surplus to the Public Distribution System.

In investigating the administrative source for electricity and gas industry, National Accounts and Economic Statistics Directorate (NAES) consider the main strengths of the data for our purposes to be:

• detailed knowledge of the subject area by the Department for Business, Energy and Industrial Strategy (BEIS)
• a dataset that is collected by law
• at least 99% UK coverage
• comprehensive quality assurance checks in place
• data are compared with other data sources

We believe that current limitations of this data source are:

• an out-of-date service level agreement
• a tight turnaround time to meet NAES deadline
• estimation of missing sales data is commonly due to large companies who are slow in returning data by NAES deadline, resulting in revisions
• limited measurement of wind and solar electricity generation

In constantly seeking to improve our data sources, we will be taking the next steps to address these limitations and these will be communicated to users in the future quality assurance of administrative data (QAAD) report updates for this topic.

However, despite these slight limitations, based on how the medium risk of quality concerns and medium contribution that the electricity and gas feed into Index of Production (10.4%) and gross domestic product (1.5%) NAES consider this data source to fulfil the requirement of an A2 assurance rating.