Comparing different international measures of excess mortality

This report has been produced jointly between Government Office for Science (GOScience) and the Office for National Statistics (ONS).

The Office for National Statistics (ONS) has been producing comparisons of excess mortality throughout the coronavirus (COVID-19) pandemic. As well as the ONS measures, there have been measures produced by the Organization for Economic Cooperation and Development (OECD), the World Health Organisation (WHO), the European Statistical Office (Eurostat) and various data journals.

This article looks at the different measures of excess mortality available, the strengths and limitations of each and how the UK compares across each measure.

Throughout the coronavirus (COVID-19) pandemic, excess deaths have been used to measure the impact of COVID-19 on mortality. The reason that we do not use deaths which mention COVID-19 on the death certificate as a measure, especially when comparing internationally, is to avoid the problem of countries recording COVID-19 deaths in different ways. Excess deaths also consider the indirect impact of the coronavirus pandemic, such as deaths from other causes that might be related to delayed access to healthcare.

Age-standardised mortality rates (ASMRs) show the number of deaths observed per 100,000 people, taking into account differences in population age-structure between places and over time. Taking into account population and age-structure allows for comparisons between countries with different population structures, as the ASMRs are standardised to the European Standard Population 2013.

This section will give a brief overview of the methodology used by the Office for National Statistics (ONS) and some of the other measures that are produced by other organisations.

Comparisons of all-cause mortality between European countries and regions

Since July 2020, the ONS has produced Comparisons of all-cause mortality between European countries and regions. This release uses data published by Eurostat, ONS, Northern Ireland Statistics and Research Agency (NISRA) and National Records Scotland (NRS). There are strict criteria that data must meet to be included, so analysing data from this source provides an opportunity to be as comparable as possible. This means we are reliant on the availability of data submitted to Eurostat by participating countries. More information about the data and methods used in the ONS article can be found in the accompanying methods paper.

Age-standardised mortality rates (ASMRs) are used to account for population size and age-structure over time and geographies, using the European Standard Population 2013. Therefore, the international comparison in these reports is based on European countries.

Building on ASMRs, relative age-standardised mortality rates (rASMRs) are used as a measure of excess mortality. These are expressed as the percentage difference per week in the observed period (2020 to 2022), compared with the average ASMR in that week when looking at the years 2015 to 2019.

Measures of cumulative mortality evaluate the total extent of mortality from one time point to another, relative to the average for an equivalent period in the past. The week ending 3 January 2020 is used as the start point in this article. The relative cumulative age-standardised mortality rates (rcASMRs) show whether age-standardised mortality from week ending 3 January 2020 to week ending 26 August 2022 has been above or below average overall. The rcASMR expresses this difference as a percentage relative to the average for an equivalent period in 2015 to 2019.

The average of 2015 to 2019 is used for each year as it represents a non-coronavirus-pandemic period. A zero value would mean that the rASMR or rcASMR is the same as expected, a negative value means that the rate is below the expected, and a positive value means that the rate is above what is expected, when compared to an average of 2015 to 2019.

Strengths

• This method accounts for both the size and age structure of the population.

• It measures both the indirect and direct impact of the coronavirus.

• It compares with a non-coronavirus-pandemic baseline.

Limitations

• This only includes European countries where weekly deaths and populations by five-year age groups from 2015 onwards are available on Eurostat.

• It does not calculate the number of deaths due to coronavirus (COVID-19) across Europe

• Comparisons between countries should be approached with caution, because countries with smaller populations (and therefore, a smaller baseline) will see greater fluctuation in the rASMRs and rcASMRs.

• Comparisons of weekly changes in rcASMR between earlier weeks in the period and later weeks should be interpretated with caution, because there will be greater fluctuation in rcASMRs when the cumulative period is smaller (and therefore, weekly excess has a larger impact) than later in the time period, when the cumulative value is higher (and therefore any additional excess has a smaller effect).

Proportional all-cause excess-mortality scores (P-Scores)

Proportional all-cause excess mortality scores (P-scores) are a measure of excess mortality. They can also be calculated as a cumulative proportional score for a whole year or for individual weeks. An article was published in June 2020 recommending the use of P-scores in evaluating excess mortality in the COVID-19 pandemic.

As with the other measures used to calculate all cause excess mortality, P-scores are defined as the number of deaths observed, minus the five-year average (2015 to 2019 or alternative five-year average) and then divided by the five-year average (expected mortality).

Strengths

• The measure is a simple calculation for all-cause excess

• It does not require population denominators

Limitations

• Because populations are not used in the calculation of P-scores, the measure does not consider the different sizes and age-structures of the geographies being compared.

Estimating excess mortality due to the COVID-19 pandemic

Estimating excess mortality due to the COVID-19 pandemic: a systematic analysis of COVID-19-related mortality, 2020-21 is an article published in The Lancet. Full details of the methods used and links to the statistical code can be found in the article itself, the following is a summary.

The measures presented are:

• reported COVID-19 deaths

• reported COVID-19 (crude) mortality rate per 100,000

• estimated excess deaths

• estimated excess deaths (crude) mortality rate per 100,000

• ratio between excess mortality rate and reported COVID-19 mortality rate

To create this analysis a database of all-cause mortality for 74 countries and regions, plus 266 sub-national locations, was collected. This data was then adjusted to account for reporting delays, anomalies such as heatwaves, and under-registrations of deaths.

To estimate expected mortality in the absence of COVID-19, six models were created and fit separately by location. Four estimated the trend (when removing seasonality) and provided a predicted level of mortality for 2020 and 2021 based on the trend. The other two models were a Poisson model, and a model that assumed that the mortality in 2020 and 2021 was the same as each week in 2019. This method uses more complex models than some others and this is a very simplified summary; more information on the details of the model can be found in the article itself.

Data prior to 1 March 2019 was used to fit and train the aforementioned models and their performance was evaluated against data from March 2019 to February 2020 to measure their accuracy. The final expected mortality estimate was based on an ensemble of the expected mortality of the six estimates weighted by each model's out of sample predictive validity. Excess mortality was then calculated by subtracting the observed mortality from the modelled expected mortality.

The next part of the analysis estimated the number of deaths due to COVID-19 using covariates that pertained to both the COVID-19 pandemic and background population health-related metrics at the population level before SARS-CoV-2 emerged. Confidence intervals have been provided; information on these intervals can be found in the article.

Strengths

• Comparisons are not limited to European countries with the paper providing estimates for 74 countries.

• The developed models use historical data to identify trends and seasonality when predicting expected values in 2020 and 2021.

• The analysis provides estimates for the number of excess deaths due to the pandemic and number of deaths due to COVID-19.

Limitations

• Changes in population have not been considered when developing the models.

• The excess mortality rates for some countries (for which weekly or monthly mortality measures were unavailable) were calculated with a statistical model; direct measurement from the locations themselves would be more robust.

• The methods used in this analysis are complex

Global excess deaths associated with COVID-19, January 2020 to December 2021

In May 2022, the World Health Organisation (WHO) released analysis based on deaths directly and indirectly associated with the coronavirus pandemic. It looked at excess mortality as expected (2015 to 2019) minus observed, and the number of reported COVID-19 deaths.

For countries with no data, an overdispersed Poisson log-linear regression model was used to predict the number of deaths. More information is available in the WHO Methods for estimating the excess mortality associated with the COVID-19 pandemic publication (PDF, 715 KB).

In summary, excess mortality is calculated in the same way as in the other studies to give the number of observed excess deaths (more than expected) for a given time-period. Populations are provided (where available), however, ASMRs have not been calculated.

Strengths

• The analysis allows for a global number of excess deaths and deaths associated with COVID-19.

Limitations

• Different methods of reporting and recording may exist for deaths associated with COVID-19 between countries.

• The data available hasn't been provided as rates, meaning the differing populations size and age-structures have not been taken into account.

• The analysis used two different methods to calculate excess depending on data availability, making comparisons between all countries difficult.

The Organisation for Economic Co-operation and Development's (OECD) Mortality Statistics

In October 2020, the Organisation for Economic Co-operation and Development (OECD) published OECD Health Working Paper No. 122 Excess mortality: measuring the direct and indirect impact of COVID-19. OECD deaths data and metadata can be found on OECD's website. The working paper acknowledges the difficulties, strengths and weaknesses of estimating all-cause and COVID-19 specific mortality rates in order to make international comparisons.

The paper uses the same method as ONS to calculate the excess mortality measure: number of deaths observed minus number of expected deaths (as an average of 2015 to 2019). The release looks at comparisons between the number of excess deaths reported and the number of deaths associated with COVID-19 reported, noting the limitations of comparing across countries. There is also a section comparing changes in population to level of excess death, however there are no rates.

Strengths

• The analysis allows for a comparison of excess and COVID-19 deaths across several European and non-European countries.

Limitations

• Different methods of reporting and recording deaths related to all causes, including COVID-19, may exist between countries.

• The data available hasn't been provided as rates, meaning the differing populations size and age-structures have not been taken into account.

Summary

Comparisons of all-cause mortality between European countries and regions (ONS)

• Comparisons of all-cause mortality between European countries and regions publication

• This covers the UK, the 27 European Union countries (EU27) and European Free Trade Association (EFTA) countries at country, major city and NUTS3 level

• This measures age-standardised mortality rates, relative age-standardised mortality rates, relative cumulative age-standardised mortality rates

Proportional all-cause excess-mortality scores (P-scores) (ONS)

• Comparisons of all-cause mortality between European countries and regions reference tables

• This covers the UK, EU27 and EFTA countries at country level

• This measures the percentage change in the number of deaths compared to the expected number of deaths (based on the five-year average)

Estimating excess mortality due to the COVID-19 pandemic (Lancet article)

• Estimating excess mortality due to the COVID-19 pandemic: a systematic analysis of COVID-19-related mortality

• This covers a systematic analysis of COVID-19-related mortality, 2020 to 2021, for 74 countries

• This measures crude mortality rate per 100,000

Global excess deaths associated with COVID-19, January 2020 - December 2021 (WHO)

• Global excess deaths associated with COVID-19 (modelled estimates)

• This covers all countries included in the WHO Global Database

• This measures number of deaths

The OECD's Mortality Statistics

• OECD Health Working Paper No. 122 Excess mortality: measuring the direct and indirect impact of COVID-19

• This covers global figures on the number of deaths using OECD data and metadata

• This measures number of deaths

Depending on which measure is chosen, the excess mortality ranking of the countries vary. Table 1 outlines the percentage change or excess mortality, depending on the measure used.

Across the different measures, the majority of countries remained at a similar ranking. The five countries with the lowest cumulative excess mortality up to week 26 2022 in the ONS' comparisons of all-cause mortality between European countries and regions release, were amongst the lowest across all measures available; these were Denmark, Iceland, Luxembourg, Norway, and Sweden.

The countries with the highest excess mortality in the comparison of all-cause mortality release, Bulgaria, Czechia, Poland, Romania, and Slovakia, were also amongst the highest excess mortality across all measures available.

The UK placed around the middle of the rankings, regardless of which excess mortality measure used.

Cyprus and Malta were two of the countries which showed differing results depending on methods. When looking at P-scores at week 52 2021, they were ranked within the highest third of excess mortality, however, at a similar time, all other methods (that consider population) had them within the lowest third.

All excess deaths calculations use the same basic principles, comparing the observed number of deaths with an expected number of deaths. The calculation of expected number of deaths can differ between methodologies with some to choosing the simple calculation of five-year averages (2015 to 2019) and others using more complex methods.

The benefits of some of the modelled approaches is that they allow for comparisons wider than Europe without having to adjust for the different population sizes and age structures. However, by using age-standardised mortality rates (ASMRs), we can compare countries across Europe whilst accounting for differences in age-structure and population sizes, enabling a more robust comparison.

As the coronavirus pandemic developed, it was essential to measure the impact on a wider scale than usual, which is why a number of these methods have been adopted. Different measures will be created based on user needs (for example, the geography range needed). However, from Table 1 we can see that, although the numbers will change based on method, the comparisons of geographies are quite similar.