Transformation of consumer price statistics: August 2024

Accurate measures of inflation are, more than ever, playing a vital role in business, government and everyday life. From rail fares to taxes and pensions, financial transactions and many other areas of our lives are regularly adjusted to reflect the change in prices over time. It is crucial to measure these changes in price as accurately as possible and continue to improve our statistics in line with the latest research and in response to market trends.

We are undergoing a programme of transformation across many areas of our consumer price statistics, including identifying new data sources, improving methods, and moving to sustainable statistical systems. This is to improve both the Consumer Prices Index including owner occupiers’ housing costs (CPIH) and the Consumer Prices Index (CPI). This allows us to reflect our changing economy and produce more robust and granular inflation statistics.

For the first time, in March 2023, we published headline consumer prices statistics including alternative data sources, marking a major milestone in this transformation process.

Using rail fares transaction data (included from February 2023) from Rail Delivery Group, we published six new indices. These were based primarily on ticket type, improving the granularity of information we provide, as detailed in our Impact analysis on transformation of UK consumer price statistics article.

In March 2024, we continued to transform our statistics by incorporating second-hand cars data from Auto Trader (included from February 2024) and changes to private rental statistics for Great Britain into our headline consumer price statistics.

In 2025, we plan to expand our improvements to private rental prices to include Northern Ireland. We also plan to introduce grocery scanner data from retailers covering approximately half the groceries market, as well as improve the processing of our traditionally collected price data. We explain what our planned changes for collecting prices will mean for consumer price statistics in our Explaining the transformation of consumer price statistics video.

To assist the transformation of our headline statistics, CPIH and CPI, some changes may be proposed for Retail Prices Index (RPI). Any proposed change to the RPI will go through the usual governance process. For more information, see Appendix 2 of our Consumer Prices Indices Technical Manual, 2019.

Our priority is the quality of consumer price statistics. Our timelines are subject to continued systems development, research and impact analysis, and any decisions will be made through continuous engagement with our stakeholders and users.

Our timelines and plans are detailed in Section 5: Future developments.

As we progress the transformation of consumer price statistics, they will increasingly become a mix of traditionally collected data, existing administrative data, and new data sources. These new data sources are outlined in our Transformation of consumer price statistics: July 2023 article.

Traditionally collected price data and our existing administrative data sources will be used when they cannot be (or have not yet been) replaced by alternative data, such as for small independent shops who do not have a website.

Alternative data sources provide many benefits compared with our current sources, including improved product and geographic coverage, higher frequency of collection and additional information, such as expenditure per product. By adopting multilateral index number methods, we can make better use of these data sources and account for dynamic changes in the market and consumer spending habits. This is outlined in our Introducing multilateral index methods into consumer price statistics methodology.

Many types of data provide more detailed information so that we can produce additional, more granular indices, as we have done with rail fares, second-hand cars and rental price statistics. These additional indices are outlined in our Impact analysis on transformation of UK consumer price statistics article and our Redevelopment of private rental prices statistics, impact analysis article.

Through the richness of information that is increasingly available to us, we are not only improving our headline measures of inflation, but also better informing the narrative around the drivers of inflation.

These new data sources will result in a huge increase in price quotes being processed each month, with a richness and granularity of data that was not previously available. For example, with the introduction of grocery scanner data, we will move from collecting 25,000 price points per month to approximately 300 million per month, which are derived from the sales of over a billion units of products per month. To maximise the benefit from these data, we have made, and will continue to make, significant changes to our methodologies and systems.

For any changes in methodologies, we have undertaken, and will continue to undertake, extensive research and stakeholder engagement. We publish our Research and developments in the transformation of UK consumer price statistics articles series, as well as the papers and minutes of our discussions with the Advisory Panels for Consumer Prices (Stakeholder and Technical) on the UKSA website.

Systems changes have been made to allow us to integrate these large data sources into our consumer price indices. We make use of cloud infrastructure, in line with the Government Digital Service Cloud Strategy, and best practice in line with the Reproducible Analytical Pipelines (RAP) strategy, defined by the UK Government Analysis Function (GAF). We have built our production systems ensuring flexibility so that future categories of data can be incorporated into the production with minimal additional development.

Further details of our systems development (focusing on rail fares) can be found on the United Nations Economic Commission for Europe (UNECE) website, under “Meeting of the Group of Experts on Consumer Price Indices”: Developing reproducible analytical pipelines for the transformation of consumer price statistics: rail fares.

To enable the introduction of grocery scanner data in March 2025, our system to process traditional data is being updated. This will allow indices using new data and traditional data to be integrated at the lowest level to enable full market coverage. These changes also allow us to move away from legacy systems, resulting in more secure and sustainable production processes.

Data from Auto Trader for second-hand cars and from the Rail Delivery Group for rail fares are already processed within this new system before being included in our headline statistics. For grocery scanner data and traditional locally collected data we will soon be entering a parallel run phase. In this phase, indices using these data sources are produced on a monthly basis in our cloud environment in parallel to those indices produced on our legacy systems. This will ensure our new indices and processes are robust and reliable before they are incorporated into our headline inflation statistics.

The changes outlined are part of a continuous programme of improvement for consumer price statistics. Our ambition is to bring in new data sources for further categories and for additional retailers to continue to improve consumer price statistics over the coming years. Our priority is the quality of consumer price statistics, delivered on sustainable statistical systems. Our timelines are subject to continued systems development, research and impact analysis, and any decisions will be made through regular engagement with our stakeholders and users.

Throughout each phase, we will be liaising regularly with our Advisory Panels for Consumer Price Statistics, our users, and the Office for Statistics Regulation. This is to ensure that our plans for consumer price inflation measurement are appropriate for improving the quality of our statistics and meeting our ongoing user requirements.

Grocery scanner data for food and non-alcoholic beverages, and alcohol and tobacco will be introduced in Quarter 1 (Jan to Mar) 2025. Our system to process traditional data will also be updated at this time to enable this introduction. This is because indices using new data and traditional data will need to be integrated at the lowest level to enable full market coverage.

Also, in Quarter 1 2025 we plan to include administrative microdata on private rents for Northern Ireland. Private rental prices' statistics are used to inform the owner occupiers' housing (OOH) costs element of the Consumer Prices Index, including OOH (CPIH); this is the Office for National Statistics’ lead measure of consumer prices inflation. Private rental prices' statistics are also used to inform the "actual rentals for housing" aspect of Consumer Prices Index (CPI) and CPIH. More information on the redevelopment plans for private rental prices statistics can be found in our Private rental prices development plan, UK article.

Our communication plan for 2024 is as follows:

• we are looking for feedback on how the price quote microdata and item indices that we publish alongside the consumer price inflation bulletin are currently used via our consultation webpage; this will inform our future data publication strategy for grocery scanner data and, in particular, whether any new aggregate outputs need to be produced 

• in December 2024, we will publish additional methodological articles regarding grocery scanner data and changes to traditionally collected data processing, as part of our Research and developments in the transformation of UK consumer price statistics articles series

• in December 2024, we will produce an impact analysis release that covers data for groceries, changes to traditionally collected data processing, as well as private rents for Northern Ireland, published following feedback from the Advisory Panels on Consumer Price Statistics

• in Quarter 1 2025, we aim for these new estimates for groceries, changes to traditionally collected data processing, and private rents for Northern Ireland to be incorporated in our Consumer price inflation, UK Statistical bulletins

We continue to gather feedback from users on this approach and our communications from users and ensure we are best meeting user and stakeholder needs.