Methods for assessing the benefits of Census 2021, England and Wales

• The Office for National Statistics (ONS) will be assessing the benefits of Census 2021 for users from central and local government, the private sector, and the voluntary and community sector.

• We previously carried out an exercise to calculate the forecast benefits of Census 2021.

• We intend to use the same methods to assess the realised benefits wherever possible, so that the results can be compared.

• We will engage with stakeholders who contributed to developing the forecast benefits, as well as a widened stakeholder base, including the voluntary and community sector.

• We will use focus groups, meetings and surveys to ask stakeholders whether the uses and assumptions made in the forecast exercise remain relevant, and to identify any new uses.

• We will share our initial estimates with stakeholders and overarching bodies to ensure they are reasonable and representative.

We have outlined our intended approach in our Assessing the benefits of Census 2021, England and Wales: 2021 article. To develop our approach, we reviewed our 2011 Census benefits evaluation report, and previous work to produce a forecast of the benefits of Census 2021 data. We looked at similar benefit assessment exercises, including Statistics New Zealand's Valuing the census 2011 paper, Statistics New Zealand's Value of the New Zealand census: August 2021 paper and the Australian Bureau of Statistics' Value of the Census 2019 report.

Our previous publication summarised the methods used to develop forecasts for the expected benefits of Census 2021. We intend to replicate the methods used to create the forecast, so we can validate the forecast benefits and understand if they have been realised. By using the same methods to assess the benefit, we will help to ensure that the results can be compared with the forecast benefits. Our approach also takes into consideration the available resource to complete the data collection and assessment.

The method described has been subject to an internal and external review process. Following feedback, we have added additional stakeholders that may improve our understanding of the value of Census 2021 data for academic users. We are also exploring how we can add to and validate the information we will collect from private sector representatives, allowing us to understand whether the data can be scaled up for their sector.

We have added a review of business-critical models used in central government to help us to better map the uses of Census 2021 data. We have also included explicit questions about what would be done in the absence of census data, which will help us to build a counterfactual to better understand how benefits arise.

We identified a list of census users who we will approach to obtain evidence to inform our assessment of the benefits of Census 2021 data: our contributor stakeholders. This includes users who engaged with previous benefits assessments, users who already engage with or are known to the Office for National Statistics (ONS) and additional users identified from desk research. We have prioritised stakeholders into three groups, according to their previous contribution to developing the forecast, and the degree of quantifiable evidence we expect them to provide. The three groups are:

• benefits priority one, which includes previous engagement with benefits quantification; we must engage with these stakeholders so we can validate whether the forecast benefits have been realised for them

• benefits priority two, not part of previous benefits quantification but likely to contribute quantifiable or very valuable qualitative evidence of benefits

• benefits priority three, may contribute some evidence or may be captured as part of other groups

Our final list of users also considers our capacity to engage, while maintaining good coverage of census users across all topics. The full list of stakeholders is available on request.

We have prioritised quantifiable benefits, which will be useful in calculating the benefits. We also recognise the value of benefits that cannot be directly quantified and intend to identify and describe these in our final report. We will also explore whether this "long tail" can still be accounted for in our benefit calculations.

Purposely selecting stakeholders who are, or have been, engaged with the ONS may mean that they are more likely to be receptive to taking part in this exercise. They may also be experienced and knowledgeable about how census data can be used. It may, however, also mean that they are more likely to use ONS data than less-engaged organisations, and so may be less representative of the sectors that they represent. Our use of surveys with our expanded stakeholder base may help to mitigate against this. We are continuing to explore how we can address this further when we seek to scale up our initial estimates to wider sectors.

We intend to engage with stakeholders through focus groups, meetings, online surveys and correspondence. We have identified our expected first and second approaches, by stakeholder group and priority.

For all priority one stakeholders, we will check that the data are being used as anticipated, and validate the assumptions and values used in the forecasts. We will also check whether there are any new uses or benefits of Census 2021 data that were not included in the benefit forecast.

For all other stakeholders, we will validate the uses of census data we are aware of and identify any new uses. We will use the information collected to develop assumptions to inform our calculations.

Where use of Census 2021 data are identified, we will also explore what stakeholders would do in the absence of Census 2021 data. This will enable us to establish a counterfactual to understand how any benefits from census data are generated.

Local government

We expect that the types of benefit generated by census data will be common across local authorities (LAs) (or, at least, LAs of the same tier and rurality). We have a good understanding of the uses of census data in local government, based on the forecast benefits and ongoing ONS engagement with users. It is unlikely that any evidence shared will be business sensitive. We therefore intend to hold a focus group. This will include priority one LAs, and priority two LAs, chosen to provide good geographical coverage and wider representation across different LA tiers (such as unitary authorities and district councils).

Participants will be sent a pre-group information pack at least two weeks before the meeting, outlining the assumptions and values used in the forecast benefits. This will allow time to consider the information and collate any relevant evidence.

Any clarification or follow up required will be conducted either by correspondence or a meeting. A note of the meeting will be shared with participants to confirm our understanding of what has been discussed. The data collected will be inputted into the benefits model and an initial estimate of the benefits associated with each use will be calculated. This will be shared with the focus group participants for review.

Once initial estimates have been reviewed by the contributors and any feedback incorporated, we will share findings with all remaining LAs and the Local Government Association and Welsh Local Government Association. We intend to use a survey to confirm that calculated benefits are reasonable and representative, and to collect information about uses that have been missed.

Central government

We have a good understanding of the uses of census data in the priority one central government departments. Although departments may have similar uses for census data, to fully understand the detailed benefits we intend to meet representatives from each department separately.

Participants will be given an information pack at least two weeks before the meeting, outlining the uses, assumptions and values used for the forecast benefits. Participants will be asked to consider whether the activity is still reliant on census data. They will be asked to review and verify each assumption and to consider if there are any additional uses that have not been captured.

Any clarification or follow up required will be conducted either by correspondence or a meeting. A note of the meeting will be shared with participants to confirm our understanding of what has been discussed. The data collected will be input into the benefits model and an initial estimate of the benefits associated with each use will be calculated. This will be shared with the relevant department for review.

Our understanding of uses of census data by priority two and three central government stakeholders is more limited, so our initial approach will be a survey to gain more information about their uses. This survey will establish uses of census data and information that will inform assumptions to be used in benefit calculations.

We will follow up by meeting and correspondence if quantifiable benefits are identified and further information is required. We will then input information into the benefits model and share initial estimates of benefits with the department for review, as we have described.

Private sector

The approach taken when producing the benefits forecast was to engage with one representative from industries understood to use census data. Representatives were drawn from a membership organisation of commercial users of government demographic datasets, including:

• retail grocery

• leisure

• real estate

• market research (including Big Data analytics and geodemographic resellers)

• utilities

• banking

• insurance

• management consultancy

• advertising

• direct marketing

We have a relatively good understanding of the uses of census data in the priority one private sector stakeholder group. This is based on the information in the forecast calculations, coupled with our ongoing engagement with users. However, we anticipate that some sectors will have experienced considerable change since the forecasts were made. We therefore consider that we need to have detailed conversations with priority one stakeholders to explore the benefits. Because of the potentially business-sensitive nature of these conversations, they will be conducted as separate meetings.

Participants will be provided with a pre-group information pack at least two weeks before the meeting, outlining the uses, assumptions and values used to create the forecast benefits. This will allow them time to consider the information and collect any relevant evidence.

For each sector, we will seek to understand whether the business decisions are still reliant upon census data, and if the assumptions applied are still reasonable. We will also ask about any new uses for census data.

Where quantifiable benefits are identified and further information is required, we will follow up with a meeting and/or correspondence.

For priority two and three organisations, where we have less prior knowledge of their uses of census data, our initial approach will be a survey to collect initial information.

We will follow up by meeting or correspondence with any priority two or three organisation that provides evidence of a quantifiable benefit unless sufficient information is included in the survey response.

We will input information into the benefits model and share initial estimates of benefits with relevant stakeholders for review.

Once initial estimates have been reviewed by the contributors and any feedback has been incorporated, we will scale up the initial estimates to reflect the sector. We will share the initial estimates and scaled-up estimates for the private sector with the membership organisation of commercial users of government demographic datasets for review.

Voluntary and community sector

Benefits to the voluntary and community sector, and academia, were not directly assessed in the benefits forecast. We intend to follow the approach described for priority two and three private sector organisations.

Where quantifiable benefits are identified and further information is required to develop assumptions, we will follow up by meeting and/or correspondence. We will then input information into the benefits model and share initial estimates of benefits with the contributor.

Additional sources of data

The benefits forecasts made use of published data, including budgets and external reports, to inform the assumptions made. Where available, the most recent version of this information will be collected. Where a more recent version of a report is not available, we will seek to find an equivalent source.

Some government decisions are highly model-led. Each government department maintains a list of business critical models and is required to document the inputs to these models. In addition to our direct engagement with government departments, we will review published inputs to business critical models to identify the main domains based upon census outputs.

Local government – informing spending decisions

To understand the value of data to inform local authority spending decisions, an assumption was made in the Centre for Economics and Business Research's (CEBR) The Value of Big Data and the Internet of Things to the UK Economy report (PDF, 5,007 KB). It was assumed that local government accounts for 50% of this value. This was then adjusted for England and Wales and calculated as a percentage of the total capital and revenue spend on relevant categories, by local authorities (LAs) in England and Wales, per annum.

To inform the forecast benefits, five LAs in England and Wales were asked to rank relevant areas of capital and revenue investment according to how reliant they were on the census data. This determined how much of the value of data could be attributed to census.

We will confirm with priority one and two LAs that the importance of census for each area of spend remains appropriate. We will update calculations with the most recent published capital and revenue investment figures. We will seek to update the value of Big Data overall by locating a more recent source. If this is not possible, we will increase the original value in line with inflation.

When contributors have reviewed our initial benefit estimates, we will invite all remaining LAs to review the assessment to confirm whether it represents the benefits they have realised. This will be done through an electronic survey.

Central government – funding allocation

Previously, government departments used their modelling to assess the sub-optimal allocation that would result from using the next best available data (rather than census data) to inform funding allocations. We used this to calculate net welfare loss, for example what is the welfare loss of a pound being spent where it is not needed, compared with where it is needed. Based on this, it has been assumed that the net welfare gain, when census estimate data are used to inform funding allocations, is 0.015% of the total allocated. For the net welfare gain when census attribute data are used to inform funding allocations, it is assumed that this is 0.053% of the total allocated.

We will engage with stakeholders to verify that the funding is still in place and relies on census data. We will seek to understand whether there have been any changes in the funding formulas and whether the assumptions are still reasonable. If this is not possible, we may need to rely on expert opinion to estimate changes to the difference in allocation since the previous exercise. This estimate would be based on what is known about Census 2021 data, compared with the next best data source.

Central government – policy research 

The granular-level data provided by the census can improve the productivity of the policy-making process by:

• reducing time spent by analysts

• reducing costs in collecting data and acquiring commercial data

• reducing risk 

In the forecast benefits, the commissioned cost of research based on census or census-derived data was used as a proxy for the value of the research, in the absence of other information. If, without census data, the department would need to undertake their own research or rebuild their models, the benefit was assumed to be the cost of doing so.

We expect to use the same methodology to assess the realised benefits. Information that helps us build a counterfactual may help us to better understand and calculate the benefit rather than relying on the cost of research as a proxy.

Central government – evidence-based investments 

We assessed the expected project delays, such as major transport schemes, and estimated the associated costs of not having census data available.

We expect to use the same methodology to assess the realised benefits.

Private sector

Some benefits to the private sector can be attributed directly to census data. In other cases, census data is an intermediate input to geo-demographic resellers. We assumed in our benefit forecast, based on information collected from the sector, that 20% of any benefit generated from using commercial geo-demographic software can be attributed to census data. 

To inform our benefit forecast, we engaged with one representative from each industry, drawn from a membership organisation of commercial users of government demographic datasets. The value of a business decision was estimated from evidence from this engagement and/or published evidence, such as Gross Value Added (GVA), which is the value generated by any unit engaged in the production of goods and services) for a sector. We asked stakeholders to estimate the proportion of the value of the decision that could be attributed to data. They then estimated how much of this was direct census data, and how much was census-based geo-demographic software.

We expect to replicate the same method. We will, however, carefully review assumptions and values with stakeholders and adapt methods to reflect changed circumstances.

Voluntary sector

Benefits to the voluntary and community sector, and academia, were not directly assessed in the forecast exercise so there is no existing methodology to replicate. We do, however, anticipate that most quantifiable benefits are likely to fall into the primary categories of benefit identified in the forecast exercise, such as:

• policy research

• informing business or spending decisions

• evidence-based investment or funding allocation

Our approach to calculating the benefits would therefore draw from the existing methods for other sectors. We will engage with stakeholders to develop and verify any assumptions.

The most recent stakeholder engagement exercise to develop the forecast benefits was in 2016 and 2017. Some of the assumptions were based on information collected in 2013. Many factors may have affected the uses and benefits of census data since these forecasts were developed. These include the coronavirus (COVID-19) pandemic, Brexit and the growing green economy. In our engagement, we will establish whether any of the existing benefits in the forecast have changed or no longer apply.

We will also seek to identify uses of Census 2021 data emerging since, or not captured in the benefits forecast. These may be from stakeholders who engaged with the forecast exercise that have begun to use census data in additional ways. They may also come from our expanded stakeholder base.

In addition to our Census 2021 Microdata samples and aggregate data, Census 2021 data will be available to users in new ways to previous censuses. The Integrated Data Service (IDS) will accelerate the availability of underlying data from Census 2021, in more accessible and interoperable formats, so that statistical outcomes can better inform policymakers. These data will be made available with more detail, and many years earlier, than for previous censuses, which will significantly increase their use and the benefits generated.

As with our approach to the voluntary sector, we anticipate that most new benefits will fall within the categories already identified. We will adopt a similar approach for comparable benefits.

Some international benefits assessments have included an estimate of the monetary value of the long tail of smaller benefits of census not directly quantified. We will consider this, based on the findings of our stakeholder engagement.

We assumed in the benefits forecast that the value of census data reduces at a rate of 5% per annum on a straight-line basis. We expect to apply this assumption in a similar way. We will review this if evidence collected from users suggests that the expected data quality has changed over time.

We will assess the degree of optimism bias for each benefit and adjust our findings accordingly. We will also adjust for sensitivity and discounting.