Quality assurance of administrative data (QAAD) report for Manufacture of mineral oil refining

1.1 Background

National Accounts and Economic Statistics (NAES) Group within Office for National Statistics (ONS) collects data from the Department for Business, Energy and Industrial Strategy (BEIS) on activities for the manufacture of coke and refined petroleum products. These data form one source in the calculation of short-term economic output indicators, namely gross domestic product (GDP(O)) and Index of Production (IoP) for the UK.

This report outlines the process data take from initial collection through to the output of the release. It identifies potential risks in data quality and accuracy as well as details of how those risks are mitigated.

This report forms the latest in a series of quality assurance of administrative data (QAAD) reports produced by NAES to investigate the administrative data sources we use in the production of short-term economic output indicators as set out by the UK Statistics Authority. As such, this report specifically focuses on our administrative data use for the mineral oil refining industry (Standard Industrial Classification (SIC) 19.2) and does not cover SIC 19.1 – coke oven production and 19.3 – other treatment of petroleum products as the data collected for these industries has more in common with data collected in the coal industry and so will be included in the review of coal.

Separate industries where we utilise administrative data will be considered in other QAAD reports in the series.

Further information relating to quality and methodology for the short-term economic output indicators can be found in our Gross domestic product, preliminary estimate report and our UK Index of Production QMI.

1.2 Standard Industrial Classification (SIC) overview

The activities of manufacture of coke and refined petroleum products cover all activities under UK Standard Industrial Classification: SIC 2007 division 19.

Based on the UK Standard Industrial Classification: 2007 the industry is classified to three groups:

• 19.1 – Coke oven production
• 19.2 – Mineral oil refining
• 19.3 – Other treatment of petroleum products

According to the Inter-Departmental Business Register (IDBR) there were 110 enterprises classified under division 19.2 in March 2016. This is a decrease of approximately 10 enterprises (negative 8.3%) from the previous year (March 2015).

Within industry 19.2, there were 45 enterprises with fewer than five employees (41% of the division), compared with 35 enterprises with 20 or more employees (32% of the division).

Of the 110 enterprises in division 19.2, the annual turnover of 40 (36%) enterprises was below £250,000. For the majority of enterprises within division 19, there were 60 (55%) enterprises with registered turnover below £1 million, with 40 enterprises in total having registered turnover of over £50 million.

2.1 UK Statistics Authority QAAD toolkit

The assessment of our administrative data sources has been carried out in accordance with the UK Statistics Authority Quality Assurance of Administrative Data Toolkit.

Each administrative data source investigated has been evaluated according to the toolkit’s risk and profile matrix (Table 1) reflecting the level of risk to data quality and the public interest profile of the statistics.

The toolkit outlines four specific areas for assurance and the rest of this report will focus on these areas in turn. These are:

• operational context and administrative data collection

• communication with data supply partners

• quality assurance principles, standards and checks applied by data suppliers

• producer’s quality assurance investigations and documentation

In the assurance of our data source, we have chosen to give a separate risk and profile matrix score (Table 1) for each of the four areas of assurance. This will allow us to focus our investigatory efforts on areas of particular risk or interest to our users (Table 2).

2.2 Assessment and justification against the QAAD risk and profile matrix

The risk of quality concern and public interest profile has been set as “low” due to the small contribution that the mineral oil refining statistics feed into the index of production (0.9%) and gross domestic product (0.1%). As such, a score of A1 (basic assurance) is deemed appropriate for this data source.

All scoring was carried out by National Accounts and Economic Statistics (NAES) based on the level of risk of the data and interest of our users. Results for each area of assurance for mineral oil refining are shown in Table 2 .If you feel that this report does not adequately provide this level of assurance or you have any other feedback, please contact stoi.development@ons.gov.uk with your concerns.

Notes for: Quality Assurance of Administrative Data (QAAD) assessment

1. The Inter-Departmental Business Register (IDBR) is a comprehensive list of UK businesses that is used by government for statistical purposes. It provides the main sampling frame for business surveys carried out by both the ONS and other government departments. It is also a main data source for analyses of business activity

This relates to the need for statistical producers to gain an understanding of the environment and processes in which the administrative data are being complied and the factors that might increase the risks to the quality of the administrative data.

Department for Business, Energy and Industrial Strategy (BEIS) collect refinery data using the Downstream Oil Reporting System (DORS) on production, supply and disposal of petroleum products in the UK. DORS is a census of all refineries reporting refinery output and fuel use in the UK. In addition to data supply to National Accounts and Economic Statistics (NAES), collection of these data are necessary to meet the UK’s international reporting requirements and for national reporting purposes where National Statistics are published by BEIS on refinery production.

The DORS system also assists in setting companies’ compulsory stocking obligations, which are stocks companies are mandated to hold for release in the event of a supply crisis. The collection of these statistics is a legal requirement in the UK under EC Regulation Number 431/2014 and a condition of membership for the International Energy Agency (IEA).

All refiners and all major suppliers of oil to UK consumption (over 50 kilotonnes in a 12-month period) complete a DORS form, which is transmitted to BEIS either via the DORS energy portal (secure online transfer system) or email on the 15th working day of each month. Data collected also include oil stocks, imports and exports, trades with other companies, product transfers and refinery intake. BEIS data is published monthly, quarterly and annually, and is reported to the IEA and Eurostat on a monthly and annual basis.

Data collection is currently voluntary, but can be mandated under legislation (the Statistics of Trade Act, 1947) if necessary. However, oil refiners have a long history of completing DORS forms and submission to BEIS. At point of delivery to NAES, data has 100% coverage of oil refiners, who are established data suppliers and have excellent data management systems in place.

Data is collected on a consistent basis using the same collection form, coverage and guidance each month with data collection, quality and methodology procedures produced under the principles of the Code of Practice for Official Statistics.

The DORS form is laid out as a balance, with input (supply) to one side and outputs (demand) to the other, with a statistical difference showing how well the figures balance.

BEIS have a change-management process in place for data collection processes. Changes to the data collection are discussed with the refineries and explained before they are rolled out. Changes are always supported by explanations, updated guidance and work examples where needed. There have been no recent changes to the methodology used to compile refinery outputs.

Strengths

• Electronic collection system

• Secure portal

• National Statistics status

• 100% coverage at point of delivery to NAES

• Regular updates on the 15th working day of each month Data collection can be mandated under the Statistics of Trade Act, 1947

• Change-management process

Next steps

• Maintain this level of knowledge in this area

3.2 Communication with data supply partners – matrix score A1

This relates to the need to maintain effective relationships with suppliers (through written agreements such as service level agreements or memoranda of understanding ), which include change management processes and the consideration of statistical needs when changes are being made to relevant administrative systems.

Department for Business, Energy and Industrial Strategy (BEIS) has established and maintained collaborative relationships with oil companies through annual formal meetings and regular informal contact via email and telephone, allowing them to freely query the data.

BEIS provide detailed guidance notes for their oil suppliers that cover each specification of fuel types and product flows based on international energy agency reporting guidance. As previously mentioned in section 3.1 they discuss change management processes, supported by explanations and updated guidance and work examples when needed.

Although BEIS do not have any formal written agreements with their oil suppliers, it has been company practice to provide these returns on a monthly basis to the department and is always delivered in a timely manner. Due to this reliability BEIS do not deem it proportional to enter into a formal written agreement. There have been no changes to the methodology recently used to calculate the mineral oil refining input figures.

National Accounts and Economic Statistics (NAES) Group meets formally with BEIS once a year and there is a service level agreement (SLA) in place, however, this expired in October 2015 due to a shortage in resources, which were allocated to other business priorities. Although this is not ideal, the potential for risk to the data is low as data collection and data quality and methodology procedures have not changed.

Despite the out-of-date SLA, NAES continues to communicate with BEIS through regular monthly emails, detailed briefings and phone calls in regards to any queries or questions with the data.

NAES is currently working on a new updated SLA with BEIS and this should be in place shortly.

Strengths

• BEIS has regular contact with oil suppliers

• Guidance notes provided to fill out forms

• Monthly briefings are provided covering oil trends

• Regular contact between BEIS and NAES

Weaknesses

• Service level agreement between BEIS and NAES is currently out-of-date

Next steps

• Update of service level agreement with BEIS

3.3 Quality assurance principles, standards and checks by data supplier – matrix score A1

This relates to the validation checks and procedures undertaken by the data supplier, any process of audit of the operational system and any steps taken to determine the accuracy of the administrative data.

The DORS form produced by the Department for Business, Energy and Industrial Strategy (BEIS) for completion by oil suppliers has been developed to contain a set of underlying locked formulae, which are in place to ensure internal consistency between parts of the form, for example, cross-checking totals and subtotals. The conditional formatting within the form return flags when an invalid value (such as a negative volume) has been entered, identifying potential errors at source to improve accuracy.

Once the oil data have been submitted, BEIS has automated validation checks to ensure oil refineries return sensible data. Validation checks are applied by BEIS to ensure consistency within the data and where there are large differences they follow up by querying with the companies to either investigate or provide a reason for the discrepancy.

Data is then examined by BEIS to identify outlying data, using changes on previous year and three and six rolling monthly periods to spot unusual trends. Their revisions policy allows companies to update their previous month’s data and data checks are run again prior to their publication. They also receive updates from companies throughout the year and publish revisions as their policy allows.

Regular audits are conducted by the statistical team and each oil company is visited at least once every two years to ensure reporting is accurate and all guidelines are followed correctly. The companies themselves are subject to an extensive regulatory regime to ensure financial probity. In addition to these usual corporate requirements for companies operating in the UK, many are also subject to the Sarbanes-Oxley Act (2002); this is an act passed by US Congress in 2002 to protect investors from the possibility of fraudulent accounting activities by corporations.

Recent audits have detected relatively minor discrepancies in reporting, due largely to the complex nature of the business, however, these minor discrepancies in the data have not had a significant impact on revisions.

Strengths

• Conditional formatting and underlying formulae in form improving initial accuracy

• Automated checks by BEIS

• Previous year and rolling monthly period checks by BEIS

• Query any potentially incorrect figures with oil companies

• Regular audits by statistical team

Weaknesses

• Changes to methodologies and policies are reported back to NAES once a year

Next steps

• Service level agreement agreeing providing advance warning of any changes in their methods

3.4 Producers’ quality assurance investigations and documentation – matrix score A1

This relates to the quality assurance conducted by the statistical producer, including corroboration against other data sources.

National Accounts and Economic Statistics (NAES) Group receives the oil refining data in the form of an Excel spreadsheet, which is delivered via email in sufficient time to ensure NAES can investigate, analyse and query the data in time for publication within the preliminary estimate of output measure gross domestic product (GDP(O)).

NAES has validation checks in place using Excel spreadsheets to carry out summary checks and investigate historical trends including monthly revisions. These are carried out on top of the checks already carried out by the Department for Business, Energy and Industrial Strategy (BEIS); NAES also read detailed briefings supplied by BEIS.

Data are then loaded into internal systems, which use one calculation to aggregate the figure for the whole of Standard Industrial Classification: SIC 2007 division 19. In the event of missing data, a statistical method that forecasts a figure based on previous months’ data is forecasted for the team.

The final output is quality assured by two senior managers within the team. There are clear desk instructions in place for a new starter and these are reviewed and updated on a regular basis. The systems are run each month for final output.

User engagement is continual and the feedback tends to relate to the overall impact of the statistics rather than to the individual data source used and to date no specific feedback on the use of mineral oil refining has been provided.

Strengths

• Established quality assurance checks in place

• Data are compared by looking at historical trends

• Desk instructions regularly reviewed

• Quality assurance by two senior managers

In investigating the administrative source for the activities of mineral oil refining, National Accounts and Economic Statistics (NAES) Group consider the main strengths of the data for our purpose to be:

• detailed knowledge of the subject area by the Department for Business, Energy and Industrial Strategy (BEIS)

• 100% coverage of oil refinery data

• data collection can be mandated under the Statistics of Trade Act, 1947

• comprehensive quality assurance checks carried out by BEIS

• collected under the Code of Practice, with the associated release designated as National Statistics

• regular contact, guidance and briefings between oil suppliers, BEIS and NAES

• NAES comparisons of data with those from previous months and full desk instructions

We believe current limitations of this data source are:

• an out of date service level agreement

In constantly seeking to improve our data sources we will be taking next steps to address these limitations and these will be communicated to users in the future quality assurance of administrative data (QAAD) report updates for this topic.

However, despite these slight limitations based on the low risk of quality concerns and small contribution that the mineral oil refining statistics feed into Index of Production (0.9%) and gross domestic product (0.1%) NAES considers this data source to fulfil the requirements of an A1 assurance rating.