How we assured the quality of Census 2021 estimates

When planning Census 2021, we recognised that the census results would only be of value when they were fit for purpose and trusted by users as a basis for decisions. High-quality statistics rely on having good-quality processes all the way from designing the questions to preparing the final estimates for publication.

This report summarises the quality assurance we conducted on the census data to check that:

• the data were being processed correctly

• unexpected features of the data were dealt with appropriately

• the final estimates were plausible in the context of the other information available to us

This adds to other published information on the census quality assurance.

Our planning of the census quality assurance (QA) activities started in 2018. We began by outlining what that work needed to cover, and then we started work specifying and acquiring the other data sources we intended to use in our analysis. We used the 2019 Census Rehearsal as a test of our first methods and tools and, in January 2020, we published our planned strategy and approach to census QA.

The published planned approach evolved as a result of users' feedback, developments in availability of comparator data, and the priorities that emerged from analysis of the census data. Major changes to the published planned approach are highlighted in this report.

This strand of our quality assurance looked at how the census processes worked, from respondents completing the questionnaire to how we imputed data for people missed from census responses. It also monitored the census data moving through different stages of processing.

This strand went largely as planned. As in 2011, we ran a Census Quality Survey to assess how accurately individual respondents completed each question, and we will publish the results of this survey. Initial quality assurance for other processes was part of the standard work for the team responsible for that process. Analytical teams conducted further assurance as the data progressed through each stage of processing.

This two-stage approach to the assurance of processes allowed us to be flexible in where we concentrated our efforts. It allowed us to ensure all processes were checked but prioritise areas where processes had not worked as planned or where the data contained unexpected features that required us to make changes to our statistical design.

Some examples of issues identified through this assurance of processes are:

• data capture - the original coding rules for paper questionnaires did not reflect the different ordering of the tick boxes for the national identity question on the Welsh version of the questionnaire; when we identified this issue, we recoded the records affected to correct the data

• addressing - we used an address matching tool to identify the addresses provided as responses to the questions on address one year ago, term-time address and visitor address; we found that when the address provided was in Northern Ireland, the tool sometimes matched this to an address in England and Wales, so we corrected this process to ensure that these addresses were correctly assigned to Northern Ireland

• misclassification - we received a small number of returns where the respondent had completed the wrong type of questionnaire, which meant that some household addresses were being classified as communal establishments; we identified and corrected them using other information to validate the change of address type

This strand of the work looked at whether the statistics based on the census data were plausible in the context of other information available to us.

The national population estimates derived from the census were assessed by a team of demographic analysts. This assessment covered, among - the size and age-structure of the population

• sex ratios at each age

• population distribution among households of different sizes and different types of communal establishment

• evaluation of implied fertility, mortality, and migration since the 2011 Census

Population estimates for each of the 331 local authorities in England and Wales went through a comprehensive set of standard checks. We also conducted further investigation of any evidence provided by a local authority as part of our Local Authority Insight initiative described in the following section. A "case study" using a fictional area, illustrating the checks we conducted and how the evidence from different sources was evaluated, is provided in our Quality assuring the local authority census population estimates, England and Wales methodology article.

In addition to our assurance of the population estimates, we also conducted detailed assurance of the census results for each topic included on the census questionnaire. This analysis included:

• assessments of response-rate profiles

• comparisons with available data sources on each topic

• comparisons with 2011 subnational patterns

• consideration of any potential issues raised by users or arising elsewhere in the quality assurance process

We will be publishing more information on the findings of our quality assurance work for each topic alongside the results for that topic.

A major addition to our published planned approach was the extended involvement of local authorities in supporting our quality assurance of the estimates for their area. Our original plans involved local authorities in three ways.

Firstly, as part of our initial published strategy and approach for quality assuring the census results, we invited all users, including local authorities, to help shape the way we conducted and reported our quality assurance work.

Secondly, as part of our partnership work with local authorities, we invited them to tell us about checks they thought we should conduct on the census data, data sources we could use, and particular aspects of their areas' populations we should know about when assuring the results.

Thirdly, we set up a Quality Assurance Working Group with representatives of a range of local authorities to consider the detail of our approach and whether it met local authorities' needs and expectations.

In 2021, we decided to extend that involvement by giving all local authorities the chance to see indicative census estimates before the data processing was complete. This meant that they could tell us about any aspect of those estimates where other evidence suggested we needed to look at that aspect again within our quality assurance process.

We developed this plan in consultation with the Office for Statistics Regulation (OSR) to make sure that providing indicative estimates for quality assurance purposes in this way was consistent with the Code of Practice for Statistics.

The final plan allowed us to offer local authorities, county councils and combined authorities the opportunity for controlled early access to indicative rounded census estimates.

Representatives of the organisations who accepted this offer were asked to identify inconsistencies between the shared rounded census estimates and any locally held sources or intelligence. This feedback was used in validation processes, alongside the investigations already underway.

The rounded census figures that we shared included:

• population by single year of age and sex at local authority level

• population total at Lower layer Super Output Area (LSOA) level

• full-time students at local authority level

• short-term residents at local authority level

• population by selected country of birth at local authority level

• occupied households at LSOA level

• usual residents by broad type of communal establishment at local authority level

In total, 255 organisations accepted the invitation, and we received feedback from 174 (including feedback stating no inconsistencies were being raised).

We identified some themes in the feedback we received. These included:

• a concern that the "usually resident" population at Census Day was different from what the population would have been without the impact of the coronavirus (COVID-19) pandemic on internal and international migration

• lower population estimates than comparator sources in LSOAs with concentrations of students

• differences between the census estimates of the number of households and counts of council tax records

Identifying these themes helped us develop standard approaches to dealing with some comments. More generally, the feedback received from local authorities was another source of evidence feeding into our quality assurance, and we used quality panels to check that we had completed the appropriate investigations of what we had been told.

The need to publish results as early as possible, and the constraints on discussing statistics before they are published, meant we could not discuss the feedback provided by local authorities with those authorities before the release of the first estimates from Census 2021. After that, we met with several local authorities to explain how we had investigated the issues they had raised and how we had quality assured the results for their area.

The work described in the previous sections generated a huge amount of evidence on the quality of the census data. However, this was only of value when we could assess it and act on any issues we identified. Our approach to this was consistent with that described in our published planned strategy and approach.

The first part of this aspect of the work was the daily Data Quality Management Forum (DQMF) meetings. These brought together representatives of the teams processing or assuring the data, allowing a very quick identification of issues together with an initial proposal of how they should be addressed.

Issues and proposed actions from the DQMF were considered at operational management meetings to endorse the proposed approach or agree an alternative, and to make sure that the necessary actions were integrated into the overall plan. Where necessary, the issue was discussed in more detail at a Statistical Contingencies Escalation Forum.

We used quality assurance (QA) panels, consisting of two or more Office for National Statistics (ONS) experts on population statistics who were independent of the QA team, to evaluate the evidence we collated on population estimates (nationally and for each local authority) and the topic results. We ran more than 100 panels looking at population estimates, making sure that each area received individual consideration both in the initial investigation and in the evaluation of that investigation. The vast majority of panels resulted in the estimates presented being endorsed as of good quality and suitable for publication. Where a panel did not endorse the estimates, these were escalated for further discussion.

Where the conclusion of the panel's work was that there was a quality issue that needed to be addressed, there were a number of possible actions. These included:

• direct editing of records where a process had not worked as desired

• adjustment of the statistical models through which estimates are produced

• the addition of new methods

• the development of quality notes to accompany statistics alongside their release

More information on adjustments resulting from this work is provided in Maximising the quality of Census 2021 population estimates.