1. Introduction
This article describes how we assessed the Census 2021 questionnaires to understand the potential impact of the coronavirus (COVID-19) pandemic on how respondents answer questions. These impacts may be because of changes in respondents' circumstances, or in their understanding of the question. It also outlines how we developed and tested additional guidance to support respondents in answering Census 2021 questions in line with data users' needs for information. Lastly it provides the wording of the additional question guidance added to the electronic questionnaires themselves. This is separate to the guidance added to the Census website help pages.
Additional information on the overall design of the Census 2021 questionnaire can be found in the question and questionnaire development overview for Census 2021. This article also provides links to detailed question development reports.
The questions and response options for Census 2021 have been finalised through the census secondary legislation: the Census (England and Wales) Order 2020, and Census Regulations for England and for Wales and cannot be changed. In addition, the paper forms and accompanying letters and information leaflets have been printed.
This work therefore focused on identifying potential issues and methods of mitigating these via census products still under development. The main method discussed here is via additional guidance on the online questionnaire information pages, instructions and guidance accordions. However, other products such as the Census website help pages and some initial contact materials have also been updated with similar messages.
Timeline
In spring 2020, we carried out an internal review of census questions to consider the impact the coronavirus (COVID-19) pandemic could have on each question. This took into account findings of ongoing user experience (UX) testing on respondent interaction with the questionnaire (2020:2). This review was shared with main stakeholders and census advisory groups for comment.
Following this, in summer 2020, the Office for National Statistics carried out:
UX testing using the online questionnaires (2020:3)
cognitive interviewing using the paper questionnaires (2020:4)
This work considered the impact of the coronavirus pandemic on respondent response to census questions. The UX testing was used to iteratively test coronavirus-related question guidance displayed with the question. The cognitive testing was used to test more detailed guidance to be provided on the online help pages of the census website.
A short description of the different research methods and sampling techniques is given in the question and questionnaire development overview for Census 2021. Details of all our research and testing can be found in the summary of testing for Census 2021.
The coronavirus-related guidance for Census 2021 is now finalised. Ongoing work around this topic will focus on understanding and meeting user requirements for outputs.
References to tests take the form (Year: Test number). "Year" refers to the calendar year the test was undertaken in and the test number is the position of the test within the year considering all testing that took place in that year. A full description of each of these items can be found in summary of testing for Census 2021.
Back to table of contents2. Research that led to the updated online question guidance
Review of census questions
For the majority of questions, for example those on demographic and sociocultural topics, we identified little or no potential for the coronavirus (COVID-19) pandemic to impact on response.
The topics where the coronavirus pandemic could potentially affect response, either through a change in circumstances, or because of a change in interpretation and understanding, were:
who usually lives here
visitors
second addresses
internal migration
health
unpaid care
economic activity
occupation
industry
hours worked
travel to work
These assessments were reviewed by expert users and advisory groups, for example, Methodological Assurance Review Panel, Census Advisory Group, and Welsh Census Advisory Group.
In many cases the potential for impact was related to language use within the questions, words such as "usually" and "mainly", which had previously been well understood, require clarification in a period of change.
Following this review, the testing focused on the topics in this list. Purposive sampling was used to obtain respondents from impacted groups. However, all questions were included in the tests, so that unforeseen impacts could also be discovered.
Stakeholder engagement
We engaged with expert users and advisory groups regarding their data needs, so we could decide what guidance respondents should be given where change in response was caused by reduced clarity of question meaning because of the coronavirus (COVID-19) pandemic. The options were, broadly:
to look backwards, to before the impact of the coronavirus pandemic
to respond as of the current circumstances
to look forward, to what is expected after the coronavirus pandemic
In general, data users wished to continue to collect data as of Census day, as a snapshot in time. However, data on travel to work are used in long-term transport and infrastructure planning, so data as at Census day would not meet their needs.
In this case, ideally data from after the coronavirus pandemic is over and transport use has settled would be needed. This is not possible to collect through Census 2021 as respondents do not know what policies their employers will have in the future.
We have therefore initiated work to investigate how to meet this data need in other ways. If you are a user of travel to work data, and wish to be consulted on your data needs, please contact Census Customer Services.
We therefore agreed consistent messaging that the census provides a snapshot in time and respondents are asked to answer about their current circumstances across all questions.
In a limited number of scenarios a valid answer based on current circumstances is not possible. For example, in questions on hours worked, and travel to work where the respondent is currently away from work on furlough. In these cases, stakeholder engagement confirmed that they should answer based on their situation immediately before the current period away from work. Guidance has been added to the census website help pages.
Online questionnaire testing
User experience (UX) testing on the impacts of the coronavirus (COVID-19) pandemic was conducted across multiple iterations, with 6 to 10 interviews in each iteration (2020:3). The aims of the UX testing were:
to consider the impact of the coronavirus pandemic on respondent understanding and response to online census questions
to design and test coronavirus-related question guidance displayed with the question, to consider how respondents can be steered to respond to the questions to best meet the user need
The information gathered on user need from topic experts was used to inform the design of the guidance tested. Testing focused on the questions identified, set out previously, as being potentially impacted by the coronavirus pandemic.
This testing included respondents whose usual work has been impacted by the coronavirus pandemic, and respondents who usually live at more than one address. The sample also included respondents from:
households where people have moved in or out of the household because of the coronavirus pandemic
households with people who are in full-time education, including students who usually live away from their family home
The aim of the guidance was generally to reinforce the message that Census 2021 is a snapshot in time, therefore the respondent should answer based on the situation as of 21 March 2021. To do this:
two additional interstitial pages were added, one at the start of the form, and one at the start of each usual residents section
additional information was added to the existing interstitial page at the start of the main job section
additional instructions were added to questions where wording was causing ambiguity
In each case the overall message was "Circumstances may have changed during the coronavirus pandemic. Answer all questions based on the situation as it is now".
A small number of questions also had specific guidance added. Specifics of the recommended updated guidance in each case is detailed in the section Updated online question guidance for Census 2021.
The UX testing found that:
adding guidance related to the coronavirus pandemic was successful in helping steer respondents to answer about their current circumstances when answering the questions
most participants noticed the additional guidance on the interstitial page added to the start of the census, but fewer respondents noticed the guidance on the interstitial page at the start of the individual section
using bold font helps to draw the respondent's attention to the guidance
the word "usually", if included in the question, can sometimes override additional guidance to answer about the situation "now", dependent on the extent the respondent considers their situation now to have become their "usual" situation
guidance related to particular groups, such as those on furlough, is best placed after the question rather than in the applicable response option
adding detailed guidance, for those in particular circumstances, increased burden across all respondents and therefore should be placed in the census website help pages rather than on the electronic questionnaire itself
Overall, the additional guidance developed aided the majority of respondents in understanding the questions as intended and therefore reduced the potential for the coronavirus pandemic to have a negative impact on respondent burden and the quality of data collected.
Where the impacts are because of a change in circumstances, rather than respondent understanding, engagement with main data users suggests that the data will still meet their data needs, with the exception of data on travel to work where alternative methods of meeting the data need are being investigated.
Paper questionnaire testing
The paper questionnaires and associated initial contact materials have been finalised and no changes can be made. However, mitigation can be made by adding additional coronavirus-related guidance to online help, which is provided for paper and online questionnaires. The telephone contact centre and other field staff will also refer to online help to answer queries regarding the questions.
Testing was carried out on the paper questionnaire via a series of cognitive interviews (2020:4). The aims of this testing were:
to consider the impact of the coronavirus (COVID-19) pandemic on respondent understanding of, and response to, the census questions on paper
to consider what online help guidance may be useful
This testing included respondents whose usual work has been impacted by the coronavirus pandemic, and respondents who usually live at more than one address, as in the UX testing. Also included in the sample were respondents who have moved in or out of the household because of the coronavirus pandemic; and respondents who are in full-time education.
The paper questionnaire testing found that:
some participants would not have accessed guidance because they were confident they had answered correctly, even when their answer was inaccurate
comprehension issues occurred for some questions when circumstances had changed because of the coronavirus pandemic
the "employment activity last week" question caused particular difficulties; where participants read the guidance it was generally well received
the word "usually", if included in the question, meant that the guidance to answer about "now" was found to be unhelpful in some instances because it does not match the question; this was especially true for the questions on hours worked and travel to work
Overall, the testing showed the importance of maximising online response, as those completing on paper cannot be supported in answering in line with the data need as effectively as those completing online. It is especially important to encourage the working population to complete online as these are the questions most impacted by the coronavirus pandemic.
To provide further support to those completing on paper, further mitigating actions are being considered including training of census engagement and field staff, and messaging within the census campaign materials.
Back to table of contents3. Updated online question guidance for Census 2021
Updates to the design of the Census 2021 question guidance for the electronic questionnaire were informed by the research and testing detailed in this article.
The following section details the changes to the online form we have made as a result of this work. Full detail of each question is provided in the question development articles.
Interstitial pages
A new interstitial page has been added to the start of the form, as shown in Figure 1.
Figure 1: New introductory online interstitial page
Source: Office for National Statistics – Census 2021
Download this image Figure 1: New introductory online interstitial page
.png (61.7 kB)
The interstitial page at the start of each individual's section is identical, except the page title is shortened to "Coronavirus (COVID-19)" and the name of the person the section refers to is inserted into the first sentence to read "Circumstances for [name] may have changed during the coronavirus pandemic."
A similar line has been added to the existing interstitial page defining what is a main job in the occupation and industry section of the form, seen in Figure 2.
Figure 2: Updated online main job interstitial page
Source: Office for National Statistics – Census 2021
Download this image Figure 2: Updated online main job interstitial page
.png (39.1 kB)
General messaging applied to specific questions
The general message to answer based on the situation on Census day is reinforced on specific questions where testing showed this was necessary.
In the question "One year ago, what was your usual address?" the general message caused increased burden to respondents as the start of the England and Wales lockdowns was very close to one year prior to Census day.
Therefore, because of the minimal expected impact on the results, the guidance was modified slightly to reference the situation immediately before circumstances changed. This reads "If the coronavirus pandemic affected your usual address one year ago, select where you were living before your circumstances changed."
In the labour market section the instruction "If the coronavirus pandemic affected your [subject of question], select the answer that best describes your current circumstances" has been added to the last four questions:
In your main job, how many hours a week do you usually work?
How do you usually travel to work?
Where do you mainly work?
Do you mainly work in the UK?
Question-specific guidance
Testing found that specific guidance was needed for large groups of people on a small number of questions. In these cases additional guidance was added to the electronic questionnaire.
When considering their activity last week, those off work on furlough, in quarantine and self-isolating were unsure how to answer the employment activity question, and many chose "None of these apply". This answer directs them to the questions for those not currently in employment, rather than for those in employment. This would impact on outputs in that we want them to answer the questions about their current employment.
An instruction was added directing them in how to answer; "If you have a job but have been off work on furlough, in quarantine, or self-isolating, select 'Temporarily away from work ill, on holiday or temporarily laid off'". This is shown in Figure 3 and was read and understood by relevant participants.
Figure 3: Updated online question on employment activity last week
Source: Office for National Statistics – Census 2021
Download this image Figure 3: Updated online question on employment activity last week
.png (157.0 kB)
As shown in Figure 4, in the question "Have you ever done any paid work?" the instruction "Furlough is considered paid work" was added to ensure that those who had been on furlough at the start of the coronavirus pandemic, and subsequently lost their jobs, did not get classified as long-term unemployed.
Figure 4: Updated online question on ever worked
Source: Office for National Statistics – Census 2021
Download this image Figure 4: Updated online question on ever worked
.png (88.2 kB)
Finally, on the supervisory status question seen in Figure 5, the instruction "This could be remotely or in person" was added to clarify that supervision did not require physical proximity because some respondents were unsure if remote supervision was included.
Figure 5: Updated online question on supervisory status
Source: Office for National Statistics – Census 2021
Download this image Figure 5: Updated online question on supervisory status
.png (34.4 kB)
Questions where no changes were recommended
The testing largely confirmed the potential impacts identified in the question review and stakeholder engagement processes. No additional large impacts were identified.
However, some areas where a potentially large impact had been forecast had a lower impact than expected. In these cases no additional guidance was added to the electronic questionnaire.
For example, although testing found that comprehension of the "Who usually lives here" questions was impacted by the coronavirus (COVID-19) pandemic, the existing guidance and instructions on the electronic questionnaire remains applicable and useful in the current situation. For example, we guide people to record themselves at their "permanent or family home" and for those with more than one address to count themselves where they "generally spend most of their time".
Similarly, respondents considered how to answer the health and care questions. However, the additional guidance on interstitial pages and other questions was sufficient to guide them to answer based on the situation as it is now.
Because of all these questions already containing substantial instructions and guidance, and the additional guidance added on other pages already helping the respondent, it was agreed not to add additional guidance to the electronic questionnaire on any of these pages.
On the term-time address question, the impact of the winter 2021 lockdown makes it hard for students to define their term-time address. As this is a complex definition, needed by a large proportion of students, we are looking to address this need for information through other products as discussed in the Online help and additional messaging subsection.
Acceptability of the visitor questions was also lowered, as some respondents thought they may aim to check on compliance with coronavirus pandemic restrictions. These questions already had guidance accordions explaining the purpose of these questions; "This is to ensure that everyone is counted in the census. Add any visitors, even if they have been included at a census questionnaire at another address." Therefore no change was made to the online questionnaire.
Online help and additional messaging
Where there was insufficient need to add to the electronic questionnaire guidance, detail has been added to online help. For example, for the question "Are you a schoolchild or student in full-time education?" the online help includes the following section:
"Changed circumstances due to the coronavirus (COVID-19) pandemic
If you're home schooling, or studying from home because your place of study has been affected by the coronavirus pandemic, answer based on the hours you were formally studying before your circumstances changed."
We have also added the additional electronic questionnaire guidance discussed in the General messaging applied to specific questions and Question specific guidance subsections to the online help pages for paper and online completion, so that it is available to paper respondents and our engagement and field staff.
Two key areas identified as needing additional guidance in online help are second address, and students and term-time addresses. Further detail of the definitions for these areas is provided in the Methodology Assurance Review Panel paper Methodology for accurately enumerating students in Census 2021 which will be published in spring 2021. This information is summarised below.
Second address (with particular reference to students)
Additional messaging has been included in online help to guide respondents to the definition of a second address as one where they would normally stay for more than 30 days a year, even if they have not stayed there this year because of the coronavirus pandemic. This includes students’ term-time address (see Students and term-time address subsection).
An address that a respondent stayed at this year for more than 30 days only because of the coronavirus pandemic is not defined as a second address for the purposes of this question.
Students and term-time address
A term-time address is an address that the student intended to stay at regularly during term-time in this academic year, even if because of the coronavirus pandemic they did not stay there as much as planned so long as they have the right to return on or after census day (even if only for one night), for example, as part of a rental contract. Students not intending to return to their term-time address for a period of six months after census day do not have to complete a census form for that address although we will request that they do so.
As in previous censuses, wherever the student is staying on census day itself, students should:
complete the census in full at their term-time address if they have one (ordering a new access code on the census website if necessary)
complete a small number of questions at their home address providing basic demographic information and details of any second or term-time addresses
International students will be enumerated at their term-time address if they are:
currently in the UK and intend to stay for three months or more
not currently in the UK but attended in person during the Autumn Term 2020 and/or Winter Term 2021, and intend to return this academic year, or before 21 March 2022
Communication of messaging
We will work with National Union for Students and Universities UK to ensure these messages are widely distributed. We had always planned a student-specific campaign, and this will include this guidance. For students, the main method of information dissemination will be via the Census campaign and the educational institutions, for example, pre-written email communications that the universities will be asked to cascade on our behalf. These are being cognitively tested (2021:1) to ensure the messages are clearly understood.
Welsh language guidance
The new question guidance on the online questionnaire will be translated by our contracted specialist Welsh language translation service provider and quality assured by the Welsh Language Census Question Assurance Group.
This group includes Welsh language and policy experts from the Welsh Language Commissioner's office and the Welsh Government and was convened to give advice on the accuracy, clarity and acceptability of the language as well as other policy issues pertaining to the Welsh language and bilingual design.
Back to table of contents4. Next steps
Alternative data sources for travel to work statistics
We are taking forward work to transform the population and social statistics system. This work, and the 2021 Census, are important bridges to the future in laying the foundation for a fully transformed statistical system based firstly on administrative data held by the public sector, supplemented with surveys.
The impact of the coronavirus (COVID-19) pandemic raises the importance of this work in relation to travel to work statistics, and related products such as origin-destination tables, and the workplace zones geography.
We have therefore initiated work to investigate how to meet this data need in other ways. If you are a user of travel to work data, and wish to be consulted on your data needs please contact Census Customer Services.
Question evaluation
The question designs for Census 2021 are based on extensive research and assessment using evaluation criteria that were set out in the publication The 2021 Census – Assessment of initial user requirements on content for England and Wales: Response to consultation (PDF, 796KB).
All questions were evaluated for the effect inclusion on the census would have on data quality, public acceptability, respondent burden, financial concerns, and questionnaire mode. A description of this evaluation tool is provided in the question and questionnaire development overview for Census 2021.
Although we have not repeated the evaluation following the updates made as a consequence of the coronavirus (COVID-19) pandemic we will continue to take the effects of the pandemic into account following Census 2021. For example, when conducting the Census Coverage Survey, the Census Quality Survey and developing the metadata accompanying census outputs.
Question harmonisation
The approach of guiding respondents to answer about their situation as of now is harmonised with the Northern Ireland 2021 Census approach.
Scotland's Census will take place in 2022, so no updates have been made to their question guidance.
We will continue to work closely with colleagues in Northern Ireland and Scotland to maximise harmonisation of outputs across the UK.
Back to table of contents