Introducing alternative data sources into consumer price statistics: May 2019

Accurate measures of inflation play a vital role in business, government and everyday life. From rail fares to taxes to pensions, financial transactions in every area of our lives are regularly adjusted to reflect the change in prices over time. It is crucial to measure these changes in price as accurately as possible.

The Office for National Statistics (ONS) is currently undergoing transformation across many areas of its statistics, including identifying new data sources and improving its methods.

As part of this transformation, we are updating the way we collect price information to reflect our changing economy and produce more timely and granular inflation statistics for businesses, individuals and government.

We are introducing two data sources to help us transform our collection.

Scanner data

Scanner data are collected by retailers at the point of sale. Scanner data will provide us with significantly more information on the number and type of products sold, allowing us to more accurately reflect changing consumer spending patterns.

The Office for National Statistics’s (ONS’s) ambition is to collect tens of millions of prices from the UK’s leading retailers, which will improve our understanding of how prices are changing in the UK economy. We are engaging directly with several of the UK’s largest retailers to gain access to scanner data on items ranging from groceries to clothing. Importantly, the data will only show what shops have sold – not what individual people have bought.

We are also working with GS1 UK – the provider of industry-standard product identifiers (barcodes) – whose catalogue of product descriptions will help streamline the data collection process.

Web scraped data

Web scraped price data are collected from shops’ websites and can provide a wealth of additional product information about online prices, such as product descriptions, directly from retailers’ websites. For example, as well as obtaining the price of a laptop, the ONS will collect information such as the laptop’s RAM and processor speed, which help us make sure the products are comparable over time.

In 2018, we announced the first phase of external data collection from web scraping with around 750,000 prices being collected every week from 30 different retailers across 80 different types of product, including clothing, electronic items and package holidays. mySupermarket does the web scraping on behalf of the ONS.

Alternative data sources provide many benefits compared with more traditional methods of data collection, including improved product coverage, high frequency of collection, as well as potential cost savings. There is also potential to provide greater regional coverage of prices and expenditure such as, for example, regional inflation measures.

However, a significant period of research and analysis will be undertaken before these new sources can be fully integrated into headline estimates of consumer price indices.

We will continue to use manually-collected data when they cannot be replaced by scanner or web scraped data (for example, from independent shops who do not have a website and where it is not practical to collect scanner data). In addition, we will continue to use so-called “administrative” data sources that are already currently used for some areas of the consumer basket.

Our ambitious plan is to include data from alternative sources in our headline consumer price statistics by January 2023.

Web scraped data

Web scraped data are already received on a monthly basis. We are currently assessing the quality of these data and how they can inform our price indices. We have today published an article with some early, indicative results on how this might look in future.

Scanner data

We are in advanced discussions with a number of large retailers about using their scanner data. We aim to create some experimental indices based on scanner data by the end of this year.