Small Area Population Estimates: Summary of methodology review and research update

Small Area Population Estimates (SAPE) are of particular importance to central and local government users for planning and monitoring services and for the calculation of a range of rates and indicators.

Understanding the accuracy of the SAPE is important for users, and also for informing plans for the 2021 Census and the further development of methodologies.

In November 2015 ONS published a report on the accuracy of SAPE compared to the 2011 Census. The report found that:

• Estimates for Electoral wards (wards), parliamentary constituencies (PCs) and clinical commissioning groups (CCGs) have an equivalent accuracy (compared with the 2011 Census) to estimates at lower layer Super Output Area (LSOA) and middle layer Super Output Area (MSOA) level.

• Additionally, this analysis found that estimates for PCs, National Parks and CCGs are of similar or better accuracy compared to the 2011 Census than the Local Authority (LA) estimates, while estimates for wards, LSOA, MSOA and Output Areas (OAs) were less accurate than the LAs when compared to the 2011 Census, with the degree of difference increasingly inversely with the average population size of the area.

This review concluded that, for the great majority of SAPE areas, the methodology used produces useable estimations of those geographies. However, at the extremes of the distribution there are a minority of areas where the methodology does not appear to have performed so well.

In parallel to this work a Methodology Division Quality Assurance (MDQA) review of the methodology used to produce the small area population estimates for England and Wales was conducted. This review included the LSOA and MSOA estimates currently designated as National Statistics, the additional geographies (Wards, Parliamentary Constituencies, Clinical Commissioning Groups and National Parks) currently designated as experimental statistics, and the OA data currently published as supporting information.

The LSOA and MSOA estimates were granted National Statistics status in 2011 with the recommendation that they should be reviewed once 2011 Census data were available.

Other geographies were considered as part of the methodological review process required to gain designation as National Statistics.

The MDQA assessment was based on the OA and LSOA level estimates meeting a certain level of quality (a coefficient of variation of 20% or less). In 2011, LSOA, Ward, PC, CCG and National Park estimates all met this standard, while the OA estimates did not. As we move away from a census year, the quality of population estimates drift. The cumulative effect of this was assessed by comparing the rolled-forward estimates against the 2011 Census.

As the SAPE are constrained to the LA estimates, the SAPE accuracy will be affected by the drift that occurs in the accuracy of population estimates at the local authority level over the decade. It would be unrealistic to expect the SAPE accuracy, at lower levels of geography than the LA level to be more accurate than the LA estimates. LSOA estimates for some age and sex groups fell outside the 20% criteria for the coefficient of variation. Patterns of variation and bias were also observed, by age, sex and geography.

The MDQA assessment made the following key recommendations:

• Communicate uncertainty around varying levels of quality

  For different geographies and population groups, the effect of the ratio change methodology is to produce differential patterns of both variance and bias, as a result of differential patterns of coverage in the patient register across sex, age and geography. LSOA and OA-level population estimates are the building blocks by which ONS users build population estimates for bespoke geographies. The level of geography at which these bespoke estimates reach acceptable levels of quality will vary, depending on the population under consideration. This uncertainty is challenging to communicate but should be attempted by providing quality information in the metadata that support these statistics.

• Publish unrounded figures

  Communicating uncertainty around the accuracy of the small area population estimates would be supported by providing unrounded (non-integer) estimates. The current process of rounding in the production of these estimates in itself introduces minor errors. Rounded figures may also imply a higher degree of accuracy in the estimates.

• Research alternative methodology for the treatment of empty cells

  Consider whether error in the OA estimates could be reduced by finding an alternative approach to the treatment of empty cells in the patient register data.

• Research possible methodological improvements

  Consider whether the quality of the small area population estimates would be improved by using a structure preserving estimation methodology (SPREE).

• Clarify the status of the published estimates

  Clarify at which age disaggregation the LSOA estimates are designated National Statistics (and that single year of age data is supporting information). Conduct analysis to assess whether the LSOA and OA-level estimates by five year age group are of suitable quality for National Statistics status. Submit methods used in the Census rebasing for MDQA.

ONS aim to address the recommendations made through further explanation of the uncertainties within the SAPE methodology in the metadata accompanying future statistical bulletins. ONS also aims to conduct research to clarify the status of the published estimates, identify any potential improvements to the methodology such as changes to the treatment of empty cells in the patient register data and the use of additional data sources to supplement the patient register data.

To inform this research ONS welcomes feedback on the quality of the SAPE estimates for different geographies, in particular in relation to the recommendations above. If you would like to comment on the SAPE estimates please email pop.info@ons.gov.uk by 30 June 2016.

Following user feedback a timetable for this research will be considered and defined alongside other research requirements within the Population Statistics Division. We will inform users of progress once a timetable has been determined, and consult users on any recommendations for changes to methodology. If you would like to be kept informed of developments please register to receive our quarterly updates. News on our population statistics can be obtained by subscribing to the quarterly newsletter (email your request to population.statistics@ons.gov.uk) or following the Twitter account @paulvickers_ONS