Impact analysis on transformation of UK consumer price statistics: January 2025

• We plan to introduce improved imputation methods, consumption segments and Northern Ireland private rental price statistics from February 2025 (published in March 2025) as part of our ongoing improvements.

• These methods changes will be introduced into our headline measures: the Consumer Prices Index including owner occupiers' housing costs (CPIH), the Consumer Prices Index (CPI), and the Retail Prices Index (RPI) from February 2025 (published in March 2025), and the Household Costs Indices (HCIs) from February 2025 (published in May 2025); we will not discuss HCI changes in this article.

• The introduction of improved Northern Ireland private rental price statistics has minimal indicative impact across all our consumer price statistics.

• The average indicative change to the CPIH annual rate from the introduction of consumption segments and improved imputation methods between January 2019 and June 2024 was negative 0.17 percentage points; the largest differences were seen during the years affected by the coronavirus (COVID-19) pandemic because of enhanced reliance on imputation procedures.

• We have produced estimates of consumer price statistics including groceries scanner data, however, because of the complexity of data and methods and the importance of these statistics, further quality assurance is required to ensure they are of the highest quality.

• We plan to parallel run groceries scanner data for a year before incorporating them into live production in 2026; these changes will not be discussed here but in a scanner data impact analysis to be released in April 2025, however we have published a selection of consumption segment level indices, showing how changing prices affect change in consumer purchases.

We are currently undertaking an ambitious programme of transformation across our consumer price statistics. This is to safeguard the production and publication of official statistics that serve the public good, as stated in the Code of Practice for Statistics. We explain this further in our Transformation of consumer price statistics: August 2024 article. This includes:

• identifying new data sources

• improving methods

• developing systems

• establishing new processes

We have been delivering these improvements over several years, reflecting the complexity and intricacy of what we are aiming to achieve.

We aim to make improvements annually as part of a continuous improvement cycle. Our first introduction in 2023 delivered the inclusion of new data for rail fares, whereas the second introduction in 2024 delivered private rents, excluding Northern Ireland, and second-hand cars.

For 2025 we plan to introduce:

• using a new sustainable cloud-based system to produce our consumer price statistics

• consumption segments for certain areas of the consumer prices basket

• improvements to our imputation methodology

• improvements to our Northern Ireland private rental price statistics

Our priorities are improving the quality and upholding the integrity of our statistics. While we intend to introduce these changes from February 2025 (published in March 2025), we are also currently completing final quality assurance and testing of our systems and processes.

We are able to produce consumer price inflation including groceries scanner data, however, because of the complexity of data and methods and the importance of these statistics, further quality assurance is required to ensure they are of the highest quality.

We plan to parallel run groceries scanner data for a year before incorporating into live production in February 2026 (published in March 2026). This is because methods and data changes can only be made in February each year at the same time as the basket and weights update. We will discuss these changes in a scanner data impact analysis to be released April 2025. However, to demonstrate our new capability and the value of scanner data, we have published a selection of consumption segment level indices, showing for the first time how changing prices affect change in consumer purchases.

New sustainable system to produce our consumer price statistics

For 2025, we are prioritising moving our consumer price statistics production onto a new sustainable cloud-based system, seeing us move away from legacy systems.

The traditionally collected data are provided by price collectors who visit shops in various locations across the country, or request price quotes from suppliers over the phone. The data will remain an important mode of collection, but the system used to process these quotes is being improved, using more advanced technology. This will allow changes to methodology to be more easily implemented in future. It will enable us to integrate traditional and new data sources simultaneously, providing improvements to processes and quality assurance measures.

Consumption segments

In 2025 we plan to introduce new consumption segments for the following sections of the consumer prices basket of goods and services:

• food and non-alcoholic beverages

• alcoholic beverages and tobacco products

• rail fares

• second-hand cars

Consumption segments are more broadly defined than items, and as a consequence there will be fewer elements in the groceries basket than previously.

"Items" are sample units which we use to estimate inflation rates at higher levels, for example the item "bananas" is a representative sample unit for the higher-level Classification of Individual Consumption According to Purpose (COICOP) subclass "fresh or chilled fruit".

Consumption segments, on the other hand, are intended to reflect all consumer expenditure and, once scanner data are introduced, will have much broader coverage.

We are therefore moving away from having a granular sample of basket items, to a less granular but more comprehensive basket of consumption segments. For example, the consumption segment "berries, fresh" includes items such as strawberries and raspberries collected in-store, but also a wide range of "berries, fresh" from scanner data including strawberries, raspberries, blackberries, blueberries and so on. However, for the majority of the basket where scanner data is not being used, items and consumption segments will remain equivalent and have a one-to-one mapping, for example "men's jeans". This change aligns with international guidance.

For more details on our introduction of consumption segments, please see Section 5: Changes to local collection methods.

Improvements to our imputation methodology

We plan to introduce several improvements to the current imputation methodology. One of the largest changes will be updating the imputation lag, specifically moving from a two-month to a one-month lag for missing non-comparable product data. Another large change will be moving from forward-imputing for seasonal consumption segments each month using the average price movements of "in-season" products (see Section 9.5.1.1 in the Consumer Prices Indices Technical Manual, 2019) to using a one-month imputation lag from aggregates. The new system allows us to tailor the imputation calculation to the inflation measure used for the first time.

Consumer Prices Inflation indices will be used to impute base prices in the Consumer Prices Index including owner occupiers' housing costs (CPIH) and the Consumer Prices Index (CPI) as opposed to the currently used Retail Price Inflation indices. The update in imputation lag will be introduced across all our consumer price statistics, while the improvement of using CPI indices to impute base prices will only be introduced to our CPIH and CPI.

Impact analyses show that the change towards using CPI to impute base prices in CPIH and CPI would slightly reduce these two measures at headline-level. However, this analysis covers a period during the coronavirus (COVID-19) pandemic and war in Ukraine when there was a notable increase in the number of imputed base prices and where inflation rose substantially faster than historical averages. As such, it should not be assumed that this impact would be as large in future periods. Limitations in the previous system used to produce consumer price inflation statistics mean that we have been unable to make these improvements earlier.

For more details on our improved imputation methodology please see Section 5: Changes to local collection methods.

Improvements to our Northern Ireland private rental price statistics

For Northern Ireland private rental price statistics, the same methodology previously implemented for England, Wales, and Scotland in March 2024 will be introduced in March 2025. These data are representative of the full range of local market rents in each Broad Rental Market Area (BRMA) and include information on the:

• property address

• rental price

• property characteristics

For further information on the transformation of private rental statistics, please see our Impact Analysis on Transformation of UK consumer price statistics: private rents and second-hand cars, December 2023 article.

We plan to introduce enhanced methods for our inflation statistics and Northern Ireland private rents transformations simultaneously from 2025. For this article, we have only included indicative impacts from the improvements in our imputation methods and the introduction of consumption segments into our consumer prices hierarchy structure. This is because of the negligible impact the new Northern Ireland private rents transformation has on our indicative consumer price statistic estimates. For more information, please see Section 4: Indicative impact of continued transformation of rents.

All indicative impacts for the introduction of consumption segments and improved imputation methods presented have been produced on the new system. When using this new system, we have replicated the currently published series and defined this as our baseline series. There are four main areas of explainable differences between our published and baseline series.

Coronavirus (COVID-19) pandemic years

During the coronavirus pandemic years of 2020 and 2021, the published indices used several different imputation methods. This is because it was not possible to collect prices in stores because of restrictions in place, which could not be readily replicated in our new system. For information on how imputation differed during the pandemic, please see our Coronavirus and the effects on UK prices article.

Missing information requiring imputation

In our baseline series, where necessary, the new system has imputed in line with the new approach, as outlined in Section 5: Changes to local collection methods. This is dependent on the level at which the information is missing.

For missing base prices, imputation is based on the overall inflation rate of all available products within the same strata.

For missing strata indices (where there are no valid price quotes to contribute to an index), imputation is based on the monthly inflation rate of all available strata within the same consumption segment.

For missing consumption segment indices (where there are no valid strata indices which could occur because of seasonal unavailability or difficulties with the traditional price collection), imputation is based on the monthly growth rate of all available consumption segments within the same Classification of Individual Consumption According to Purpose (COICOP) level 4 category.

Seasonal consumption segments

Seasonal consumption segments (previously known as seasonal items) have marked seasonal purchasing or consumption patterns. In previous methodology, the base prices for seasonal consumption segments not collected in January were calculated each month using the average price movement of the "in-season" products to forward impute (see Section 9.5.1.1 of our Consumer Prices Indices Technical Manual, 2019). In the new system, base prices for "out of season" seasonal consumption segments are calculated using the overall COICOP4 monthly growth rate. For example, if "men's shorts" are reintroduced in April, the new system would use the overall COICOP4 within year index for "garments" for April to impute the base prices for those products. The quotes for "men's shorts" will then start contributing in May.

This change to seasonal consumption segments will be introduced from February 2025 (published in March 2025); however, it could not be isolated in the new system and therefore will be presented as part of the baseline scenario.

Historical adjustments

There are also a small number of isolated historical adjustments that cannot be replicated in the new system without introducing several otherwise unnecessary modifications within the pipelines; we expect the impacts of these to be minimal.

Figures 1 to 3 show differences between the currently published series and our baseline series from the new system for Consumer Prices Index including owner occupiers' housing costs (CPIH), Consumer Prices Index (CPI), and Retail Prices Index (RPI) respectively.

For each of our consumer prices statistics - CPIH, CPI and RPI - we have assessed the impact of introducing improved imputation methods and consumption segments between January 2019 and June 2024, had the changes been introduced from February 2019. These are covered separately throughout this section. For further details on the improved methodology, please see Section 5: Changes to local collection methods.

To measure the impact of introducing improved imputation methods and consumption segments, the following section will solely compare with the baseline series produced by the new system.

Indicative impact on Consumer Prices Index including owner occupiers' housing costs (CPIH)

Figure 4 shows the indicative impact of introducing improved imputation methods and consumption segments on headline CPIH between January 2019 and June 2024.

Table 1 shows the indicative impact of introducing improved imputation methods on CPIH annual and monthly rates of change, per year, rounded to two decimal places. Note that the indicative impacts of introducing consumption segments were minimal and so have not been included in Table 1, but can be seen in Figure 4.

Figure 4 shows that the average indicative impact to the CPIH annual rate of change was negative 0.17 percentage points when introducing all improved methodology. This shows a noticeable impact of introducing improved imputation methods, namely moving to a one-month imputation lag, and using CPI base price imputation.

Out of the two methods, the one causing the divergence is the introduction of CPI base price imputation. In general, CPI has a lower price change than RPI, which has reduced the overall rate of inflation. Further information on this change can be found in Section 5: Changes to local collection methods.

The average annual difference of negative 0.23 percentage points to two decimal places between April 2020 and October 2022 contrasts an average annual difference of negative 0.08 percentage points covering the period from January to March 2020 and post-November 2022. This shows the impact of the coronavirus pandemic, cost of living, and war in Ukraine when there was a notable increase in the number of base prices being imputed and a rise in inflation beyond historical averages. As such, it should not be assumed that the impact would be as large as the January 2019 to June 2024 difference in future periods.

There are three possible scenarios where we will see an impact because of changing to use CPI to impute base prices for CPI and CPIH indices, for products entering the collection without a valid or comparable base price. These include:

• invalid base prices in January, for example, when a product is out of stock, or a mistake was made by the price collector during the January collection

• products located at new shops, for example, when a shop previously used for collection has ceased trading

• non-comparable replacements, selected when a previously collected product can no longer be found, and a comparable replacement is not available

During the coronavirus pandemic and late 2021, we saw a notable increase in invalid base prices in the base month of January (for 2021) as well as new shops and non-comparable replacements, because of the unusual economic nature of the time. These increases help explain why we see larger indicative impacts during these years. For more information on our pandemic collection procedures, see our Coronavirus and the effects on UK prices article.

For more details on our improved imputation methodology, please see Section 5: Changes to local collection methods.

Indicative impact on Consumer Prices Index (CPI)

Figure 5 shows the indicative impact of introducing improved imputation methods and consumption segments on headline CPI between January 2019 and June 2024.

Table 2 shows the indicative impact of introducing improved imputation methods on CPI annual and monthly rates of change, per year, rounded to two decimal places. Note that the indicative impacts of introducing consumption segments were minimal and so have not been included in Table 2, but can be observed in Figure 5.

The average indicative impact to the CPI annual rate of change across the period when introducing all improved methodology was negative 0.21 percentage points, this was mostly influenced by the introduction of improved imputation methods. In general, CPI has a lower price change than RPI, which has reduced the overall rate of inflation. Further information on this change can be found in Section 5: Changes to local collection methods.

The average annual difference of negative 0.28 percentage points to two decimal places between April 2020 and October 2022 contrasts an average annual difference of negative 0.11 percentage points covering the period from January to March 2020 and post-November 2022. As was the case with CPIH, this is because of the unusual circumstances surrounding the economic climate at that time. As such, it should not be assumed that the impact would be as large as the January 2019 to June 2024 difference in future periods

Indicative impact on Retail Prices Index (RPI)

Figure 6 shows the indicative impact of introducing improved imputation methods and consumption segments on headline RPI between January 2019 and June 2024.

Table 3 shows the indicative impact of introducing improved imputation methods on RPI annual and monthly rates of change, per year, rounded to two decimal places. Note that the indicative impacts of introducing consumption segments were minimal and so have not been included in Table 3, but can be seen in Figure 6.

The average indicative impact to the RPI annual rate of change across the period when introducing improved methodology was negative 0.01 percentage points, this was mostly influenced by the introduction of improved imputation methods. The impacts on RPI are smaller than on CPI because of the only change being the introduction of the one-month lag in imputation.

In March 2024, we substantially improved data for private rents by introducing the Price Index of Private Rents (PIPR) for England, Wales, and Scotland. We intend to introduce transformed private rents data for Northern Ireland and to introduce minor data improvements for Great Britain in March 2025. This will have minimal indicative impact on consumer price statistics. See further information about the changes we are introducing to PIPR in our Redevelopment of private rental prices statistics, impact analysis, UK: January 2025 article.

For each of our consumer price statistics - Consumer Prices Index including owner occupiers' housing costs (CPIH), Consumer Prices Index (CPI) and Retail Prices Index (RPI) - we have assessed the impact of transforming private rental prices and imputed rental prices between January 2015 and July 2024, had the changes been introduced from February 2015. These changes are assessed against the current methodology for private rents, had they also been introduced from February 2015.

This transformation has a minimal impact on consumer price statistics; where the indicative impact of transforming private rents data on headline CPIH, CPI and RPI between January 2015 and July 2024, was an average of 0.0 percentage points to the annual rate of change (when rounded to one decimal place).

Alongside this article, we have published an accompanying dataset showing the full impacts of this transformation on headline CPIH, CPI and RPI consumer price statistics.

The PIPR index, and the private rents and imputed private rents contribution from PIPR to CPI and CPIH are based on a geometric mean of price relatives (Jevons), while the contributions to private rents from PIPR to RPI are based on an arithmetic mean of price relatives (Carli).

As part of our transformation, we are changing the data provider for private rents and imputed private rents for Northern Ireland for consumer price statistics, including CPIH, CPI and RPI. This new data, supplied by the Northern Ireland Housing Executive (NIHE), is of higher quality but is supplied at a two-month lag compared with that from the other countries of the UK.

From March 2025, we will impute the Northern Ireland private rents and imputed private rents components by inflating the most recently available data by the latest available two-month growth rate for Northern Ireland. You can find more about our proposed imputation method for Northern Ireland private rents in our Redevelopment of private rental prices statistics, impact analysis, UK: January 2025 article.

Introduction of consumption segments

When publishing our inflation statistics, we first calculate our lowest level "elementary aggregates", and then we combine these elementary aggregates (with weights) into higher level aggregates. The pyramid structure we use to combine aggregates in this way is described as our aggregation hierarchy.

We have previously used the following aggregation hierarchy (see Figure 1 in Section 3.2 of our Consumer Prices Indices Technical Manual, 2019):

• five levels of the Classification of Individual Consumption According to Purpose (COICOP) hierarchy (for Consumer Prices Index (CPI) and Consumer Prices Index including owner occupiers' housing costs (CPIH)), or four levels of our RPI structure (for Retail Prices Index (RPI))

• one item level

• one (optional) lower stratum level where items can be stratified by region, shop-type or a combination of the two where, in the latter case, weights are combined to allow simultaneous stratification

However, from 2025, we plan to introduce the following new aggregation hierarchy:

• five levels of COICOP (for CPI and CPIH), or four levels of our RPI structure (for RPI)

• one consumption segment level for basket areas listed in Section 2: Overview of proposed transformation, all other areas will remain at item-level

• one region level for those categories with a new introduced consumption segment, for all other items this remains an optional level

• one retailer or shop-type level for those categories with a new introduced consumption segment, for all other items this remains an optional level

• one or more extra strata level (optional), which includes disaggregated items for traditional data

The main change is to introduce a new consumption segment level. In certain categories, this is broader than item-level but more granular than COICOP5. Each consumption segment is represented by a near-census of products when using alternative data, or by one or more items when using traditional data. For those categories, we are also reordering some of our levels, moving region and retailer or shop-type stratification above of the disaggregated items. For other areas of the basket, the consumption segment level will be equivalent to the existing item categories and the existing aggregation hierarchy will still be used, hence there will be an optional lower stratum level comprised of either region, or shop-type, or both.

Imputing missing aggregates

The most common index methods we use to calculate our elementary aggregates, whether it is the GEKS-Törnqvist with alternative data sources, or the Jevons, Dutot or Carli with traditional data sources, are described as "matched-model" index methods.

A matched-model index method relies on the ability to match the price of a product in one month, against the price of the same product in another month. For example, if a pack of apples costs £1.00 in January, and £1.20 in February, then we can "match" these prices and observe that these apples increased in price by 20% between these two months.

These methods are discussed in more detail in our New index number methods in consumer price statistics article. The GEKS-Törnqvist in particular is discussed in our Introducing multilateral index methods into consumer price statistics methodology. For more information on the methods named, see Section 3.4 of our Consumer Prices Indices Technical Manual, 2019.

It is common for products to enter or leave the market, and when they do so, we are unable to match these products over time because of the lack of a price in one of the two months. In general, the further apart the two months we are comparing are, the more failed matches that occur. If none of the products we are sampling can form a match, then we risk the inability to calculate the index completely. We have some protections against this; for traditional aggregates we can perform comparable or non-comparable replacements enabling us to sustain our matches. We do this using quality adjustment procedures where appropriate, as detailed in Section 9.3 of our Consumer Prices Indices Technical Manual, 2019. In alternative data, the much-increased sample of products reduces the risk of being unable to perform any matching.

Among the many aggregates we calculate across our inflation measures, there is still a risk that some of these may not be calculable because of failed matches, and we will then need a method to account for this missing index. Economic shocks can particularly result in a substantial drop in "product matching rates", and result in more aggregates being incalculable than is usual. This was seen in the coronavirus (COVID-19) pandemic as entire sectors, such as restaurants and hospitality, shut down. This is discussed further in our Coronavirus and the effects on UK prices article.

Where elementary aggregates are incalculable, we have developed an imputation procedure to ensure that these elementary aggregates can be "filled" to allow further aggregation. We describe the approach as "month-on-month aggregate imputation". We remove the missing elementary aggregates from our aggregation hierarchy, calculate the month-on-month rate of the higher-level aggregate without these missing elementary aggregates, and use this to apply an adjustment to the previous month.

The following formula is used:

Where:

is the index of the elementary aggregate in month t, and

is the index of the (higher-level) consumption segment in month t

For example, if there are 30 elementary aggregates feeding into the apples consumption segment, and two of these were missing, requiring imputation, then we would calculate the month-on-month rate of the apples consumption segment over the remaining 28 aggregates. We would then use this to uprate the previous month index for the two missing apples elementary aggregates according to the formula given. Once all aggregates are imputed, we can aggregate again with every stratum accounted for.

These imputation procedures will be applied to alternative and traditional elementary aggregates alike. Note that the need for imputation is relatively rare, occurring in approximately 11% of cases in our baseline scenario in 2024.

Base price imputation

Imputing on a like-for-like basis

When calculating our elementary aggregate indices within our traditional statistics, our index methods rely on being able to compare the price of products within the current month against base month (January) prices to calculate the price change of a product between those two months. There are a variety of circumstances where a new product may lack a valid base price and hence require imputation procedures, as previously discussed in Section 3: Indicative impact of transformation on annual consumer price inflation rates.

A base price is calculated for the new product by assuming that its price change from the base month up until that month equals the average price change for products within the same stratum. This procedure ensures that bringing in the new product has no effect on the elementary aggregate in the month that it is introduced. See Section 9.3 of our Consumer Prices Indices Technical Manual, 2019 for more information.

The base price imputation process is best shown with an example. Suppose that a new product is observed in May at £13.50, and that the index from January to May for the elementary aggregate is 105. Then we can impute the base price by dividing the current price by the elementary aggregate index:

Historically, we have used the RPI index to impute base prices for both CPI or CPIH and RPI. However, if we used the CPI index in the formula instead, we would reach a different imputed price (unless the two indices are the same). From 2025, we will be imputing on a like-for-like basis: using CPI indices to impute base prices used in the CPI and CPIH, and RPI indices to impute base prices used in the RPI. This will improve the consistency in methods and independence of calculations of the CPI and RPI.

Reducing the lag for non-comparable replacements

As mentioned previously, base price imputation is often used in situations where we lack a base price to compare our current price against. One of these situations occurs with "non-comparable replacements". Sometimes a product leaves the market and, to keep our sample counts sufficiently high, we replace the product. Sometimes we can find a comparable replacement, and we can compare the new product's current month price to the original product's base month price. Otherwise, the replacement is described as "non-comparable" and instead of using the original product's base price, we use an imputed base price. Under current methodology, the imputed base price is calculated with a two-month lag. Under new methodology, it will change to a one-month lag.

To show the change, consider an example where we identify a non-comparable replacement in March. Historically, we would use the April product price and April index to impute the January base price and then introduce the product into our indices from May, comparing the observed May price with the imputed January price. Given we identified the replacement in March but only introduced the product into our indices in May, this left a two-month lag from identification to introduction.

From 2025, we will be reducing this to a one-month lag. If the non-comparable replacement is made in March, we will use the March product price and March index to impute a January base price, then start using the imputed January base price from April.

Seasonal consumption segments

Another situation where base price imputation is required is to process seasonal consumption segments. Note that some products are only collected for the part of the year that people tend to make purchases. For example, "men's shorts" are only collected from April to August, mostly covering the warmer months of the year they are typically purchased in. As is the case with non-comparable replacements, we therefore lack January base prices for the products belonging to the consumption segment and need to make an imputation. However, unlike with non-comparable replacements, every single product within the consumption segment lacks a January base price.

Until now, for seasonal consumption segments unavailable for collection in January, we have been rolling forward the last collected price from the year before to set base prices. In the case of men's shorts, when this item is out of season between September and March, we impute an index by using the movement in indices from the parent aggregate "men's outer clothes". Once the item comes back in season in April, we use the price from the last time the product was in-season to set the base price. If a base price for the matching item is not available however, we impute a base price for May based on April's men's shorts stratum index. For more information, see section 9.5.1.1 of the Consumer Prices Indices Technical Manual, 2019.

From 2025, we will start using the base price imputation strategy as described previously for non-comparable replacements. However, note for seasonal consumption segments, since none of the products have base prices, the consumption segment index cannot be calculated directly and used to set base prices. Instead, we will use aggregate imputation to first impute the consumption segment index, then use this to impute the base prices.

Ahead of entering production with the improved methodology, we are in the process of finalising and testing our systems development to allow sustainable index calculations within a production environment.

Following quality assurance and a year-long parallel run of transformed data and methods for grocery scanner data, we plan to incorporate this into live consumer prices statistics in February 2026 (published in March 2026). An impact analysis on grocery scanner data will be published in April 2025.

You can find more about our plans for the continued development of our rental price statistics in our Redevelopment of private rental prices statistics, impact analysis, UK: January 2025 article.

Since the plans outlined in this article will necessitate changes to our item indices dataset and price quotes microdataset, we recently ran a user feedback survey to understand how they are currently being used. See our response to the user feedback survey.