Quality assurance of administrative data used in consumer trends statistics: September 2019

Quality assurance of administrative data used in estimates of household final consumption expenditure as published in Consumer trends

This publication documents the quality assurance (QA) assessment of all data sources used to produce estimates of household final consumption expenditure published in Consumer trends.

Household final consumption expenditure (HHFCE) is spending by households on the goods and services they consume. It does not include investments (such as buying a house or valuables). Transactions between households are excluded because there is no overall impact on how much households consume. It is produced on a quarterly basis and contributes over 60% to the expenditure measure of gross domestic product (GDP). Provisional high-level estimates form part of the GDP first release. More detailed figures are published in the Quarterly National Accounts and Consumer trends three months after the end of the latest quarter. The statistics are also supplied to Eurostat, the statistical office of the European Union. HHFCE is a National Statistic.

Estimates of HHFCE are compiled from over 30 data sources. Some are produced internally by the Office for National Statistics (ONS), such as survey data from the Living Costs and Food Survey (LCF), and others are supplied by external bodies from the public and private sectors.

The UK Statistics Authority Code of Practice requires producers of official statistics to review the quality of their statistics. To improve our understanding of the quality of data sources used to estimate HHFCE, we conducted an exercise to review the administrative data used to share with our users.

In summer 2017, we sent detailed questionnaires to all of our data suppliers (35 in total) and the results were assessed using the Quality Assurance of Administrative Data (QAAD) toolkit.

A level of assurance was assigned to each of the suppliers from “basic” to “enhanced” to “comprehensive”. Of the 35 questionnaires dispatched, 17 were assigned as comprehensive, six as enhanced, seven as basic and five as non-responders. The non-responders were contacted at least five times by letter and by telephone.

The overall quality of data sources used to compile HHFCE was assessed to be of low-quality concern and medium public interest which equates to an overall risk profile of A1/A2.

Under the Code of Practice, statistics producers must review the quality of data used to produce official statistics. The UK Statistics Authority encourages the use of the Quality Assurance of Administrative Data (QAAD) toolkit.

More information about the UK Statistics Authority QAAD initiative can be found on the website of its regulatory arm, the Office for Statistics Regulation.

We have used the toolkit to investigate the quality of data used to compile estimates of spending on goods and services by members of UK households. This exercise covers both administrative and other data sources, for example survey sources, in line with the guidelines provided by the UK Statistics Authority. We sent out questionnaires to all the external and internal data suppliers to collect information on their processes of quality assurance.

Use of the Quality Assurance of Administrative Data (QAAD) toolkit

We used the Quality Assurance of Administrative Data (QAAD) toolkit to determine compliance for each of the 35 household final consumption expenditure (HHFCE) data sources. Although the QAAD is designed for administrative data many of the principles contained within it also apply to other data sources, so we used it to assess the quality of data provided by all the HHFCE suppliers.

The QAAD toolkit is built around the Quality Assurance (QA) Matrix which presents the levels of assurance for four areas of practice related to the quality assurance of official statistics and of the administrative data used to produce them. These are:

• operational context and administrative data collection

• communication with data supply partners

• quality assurance principles, standards and checks applied by data suppliers

• producer quality assurance investigations and documentation

Table 1 shows the four possible outcomes from the assessment.

The QAAD toolkit recognises that a proportionate approach is appropriate, whereby some statistics will need greater levels of assurance than others. The levels can be determined by an evaluation of:

• the likelihood of quality concerns arising in the administrative data that may affect the quality of the statistics

• the nature of the public interest served by the statistics

These two levels are used to derive the overall risk of data quality concerns using the Risk Profile Matrix (Table 2).

Use of the toolkit to assess the quality of household final consumption expenditure (HHFCE) data sources

Our approach to assessing the quality of the household final consumption expenditure (HHFCE) data sources involved the following steps:

1. We sent a questionnaire to each of the external and internal Office for National Statistics (ONS) data suppliers. This questionnaire requested information on their data collection activities, data processing activities and how their statistics are used. See Annex A for an example of the questionnaire.

2. We contacted non-responders to the survey by telephone, email or letter. 30 completed questionnaires were returned out of a total of 35 data suppliers.

3. We reviewed each of the responses received from the data suppliers to assess the strength of the quality assurance processes and the data processing operations. We did this by assessing the information they provided against eight categories:

   • processes used to produce the statistics
   • information about data sources
   • revisions policy
   • communication
   • use of the data
   • strengths and weaknesses of the data
   • quality assurance checks
   • quality assurance reviews

4. In our assessment of the 35 data sources, we marked each of their responses to the individual questions with the rating of either “no assurance”, “basic”, “enhanced” or “comprehensive”. We used this information to determine a level of risk of data quality concerns for each data source. Two people assessed each of the 35 data sources independently before reconciling their results.

5. The importance of data quality for different items of expenditure depends on their contribution to overall household spending. We allocated weights to each data source according to the contribution to the overall level of household final consumption expenditure in the UK by resident and non-resident households. We used this as our measure of public interest in the official statistics produced from the data source.

6. We used the level of risk of data quality concerns, and public interest profile allocated to each of the 35 data suppliers to produce a rating of risk of data quality concerns and public interest profile for the overall HHFCE statistical output.

The household final consumption expenditure (HHFCE) statistics are compiled using a total of 35 data sources (Table 3).

Introduction

Each of the completed questionnaires received from the data suppliers was reviewed to ascertain the level of quality assurance applied to the data, and whether the submitted evidence complies with the Code of Practice.

Eleven of the data sources are internal, that is, surveys or outputs which are operated by the Office for National Statistics (ONS). Annex B at the end of this document lists the ONS publications for these data sources. Twenty-four of the data sources are external and these range from private research companies producing consumer product prices to government departments and other public bodies. Five of the 35 data suppliers were non-responders to our survey and these were all external suppliers. The combined weight of these non-responders, in terms of their contribution to the overall level of household final consumption expenditure (HHFCE), is not judged to be significant (5%).

A weight was allocated to each data source according to the contribution to the overall level of HHFCE domestic data (Table 4). There is a high overall weighting for internal sources when compared to external sources.

Overall assessment of data sources

The overall risk of data quality concerns was low, with 23 out of 30 data sources rated as having enhanced or comprehensive assurance levels (Table 5).

The public interest profiles for the data sources based on their weights were medium or low, giving an overall profile of medium for total HHFCE.

This gives an overall risk profile of A1/A2 – statistics of low quality concern and medium public interest.

Analysis of internal sources

Table 6 below shows our assessment of the different criteria of the Quality Assurance Matrix for the internal data suppliers; it also contains our comments on the responses provided for each area of processing.

Analysis of external sources

Table 7 below shows our assessment of the different criteria of the Quality Assurance Matrix for the external data suppliers; it also contains our comments on the responses provided for each area of processing. It should be noted that some of the quality assurance categories are not applicable for some of the data suppliers.

Quality assurance checks carried out by the Office for National Statistics (ONS)

The household final consumption expenditure (HHFCE) branch of the ONS carries out extensive quality assurance checks on the data which it produces. The checks carried out by the team are as follows:

• clear desk instructions are in place for quality assurance processes and an issues log maintained

• data files received from each data source are checked to ensure they appear in line with past deliveries; suppliers are contacted to confirm any anomalies or, if necessary, to request resupply of the data

• revisions to the HHFCE data are checked by analysts; briefings sent by data suppliers are used to validate the revisions

• following the results run consistency checks are carried out to ensure accuracy

• growth rates are reviewed using graphs; estimates are corroborated against other survey data to assess consistency

• internal briefings are produced which record our data during the quarterly balancing process

• the finalised HHFCE data is checked by all team members prior to publication

Adjustments to data as part of supply and use (SU) balancing

After source data has been processed preliminary estimates of HHFCE by product are compared with estimates derived from other sources, including business surveys, as part of supply and use (SU) balancing.

Gross domestic product (GDP) in current prices can be measured using three different approaches: production, income and expenditure. In the UK, we compile and draw on information from all three approaches and balance them together to arrive at a single estimate of GDP which uses all available data.

Following international guidance and best practice, we confront the estimates from the three approaches using the supply and use tables (SUTs) framework. This reconciles, at a detailed level, the supply of goods and services produced domestically and imported with their use – either consumed, invested or exported.

The HHFCE estimates form part of the expenditure measure of GDP and appear in the use table. They are therefore included in the annual balancing process. The balancing process takes place at the 112 products level (CPA 2008). Adjustments to HHFCE data from the SU balancing process are then allocated to the unbalanced HHFCE estimates at the COICOP (Classification of individual consumption by purpose) level using a CPA (Classification of product by activity) to COICOP mapping.

Strengths and weaknesses of estimates of household final consumption expenditure (HHFCE) by individual classification of individual consumption by purpose (COICOP)

Strengths

• The HHFCE estimates are produced according to well-established and well-documented methods and comply with international standards such as the System of National Accounts: SNA 08 and European System of Accounts: ESA 2010.

• The estimates are compiled by experienced analysts using integrated data processing systems and employing extensive data checking techniques.

• The supply and use balancing process reconciles estimates of household spending on specific products with estimates of the value of output produced by businesses. Respondents to household surveys may over or underestimate expenditure on some products; in these cases information from business surveys is used to adjust the estimate of household expenditure.

• Patterns of household expenditure keep changing so it is important to review methods used to estimate HHFCE regularly to ensure all relevant expenditure is captured. We have an ongoing programme to review the methodology used for HHFCE.

Weaknesses

• HHFCE is currently produced using COICOP 1999 – patterns of consumer spending have changed considerably since then. For example, COICOP 1999 treats postal services as communication (COICOP 8) whereas now the expansion in online shopping means most of this is delivery of parcels so it sits better within Transport services (COICOP 7). A new expanded version – COICOP 2018 (2.45MB) – has just been produced and we are considering when to adopt this for HHFCE statistics.

• The current systems used to produce HHFCE are complex. They have recently been improved but it takes a long time to make changes to the methods used to calculate HHFCE. Further investment would enable estimates of HHFCE to reflect changes in consumer spending more quickly.

• We will continue to work closely with both internal and external suppliers to improve the quality of data used to produce household final consumption expenditure (HHFCE). Improved quality documentation and procedures will be requested in future service level agreements and contracts.

• We will ensure data quality is discussed at all regular meetings with suppliers including the importance of holding regular quality assurance reviews.

• We will seek updates about any methodology or policy changes that could affect the data.

• We will seek further clarification about data quality from sources rated as basic or having no assurance.

• When engaging in the future with our external data suppliers, we will highlight the importance of formal revisions policies which again will be fully documented in future service level agreements and contracts.

• The review of our quality assurance of administrative data will now become part of our ongoing quality review procedures.

Example of consumer trends quality assurance questionnaire

The following Office for National Statistics (ONS) releases contain data which is used to produce the consumer trends estimates:

• Family spending in the UK (data source: Living Costs and Food Survey) – average weekly household expenditure on goods and services in the UK, by region, age, income, economic status, socio-economic class and household composition.

• Annual Business Survey – financial information for the UK non-financial business economy; variables covered include turnover, purchases and approximate gross value added.

• Retail Sales Index – a first estimate of retail sales for Great Britain in value and volume terms, seasonally and non-seasonally adjusted.

• UK trade – total value of UK exports and imports of goods and services in current prices, chained volume measures and implied deflators.

• Consumer price inflation, UK – price indices, percentage changes and weights for the different measures of consumer price inflation.

• Overseas travel and tourism (data source: International Passenger Survey) – visits to the UK by overseas residents, visits abroad by UK residents and spending by travellers, using passenger traffic data.

• MQ5 investment by insurance companies, pension funds and trusts (data source: financial corporations: insurance, pensions) – investment choices of financial institutions based on financial transactions, including balance sheet data for short-term assets and liabilities, and income and expenditure data.

• Estimates of the population for the UK, England and Wales, Scotland and Northern Ireland – national and subnational mid-year population estimates for the UK and its constituent countries by administrative area, age and sex.

• Vital statistics in the UK: births, deaths and marriages – annual UK and constituent country figures for births, deaths, marriages, divorces, civil partnerships and civil partnership dissolutions.

• Business investment in the UK (data source: gross fixed capital formation) – estimates of short-term indicators of investment in non-financial assets; business investment and asset and sector breakdowns of total gross fixed capital formation.