Volunteering topic research for Census 2021

In December 2018, the government presented to Parliament a White Paper Help Shape our Future: The 2021 Census of Population and Housing in England and Wales. This outlined the Office for National Statistics’s (ONS’s) recommendation to not collect information on volunteering in Census 2021. There were two reasons for this: first, our testing had shown that respondents found the proposed question difficult, which raised issues of data quality, and secondly, it suggested that more than one question would have to be asked to meet the need, presenting unacceptable burden.

The request to include the topic of volunteering in Census 2021 was identified in the June 2015 consultation on census topics for 2021. Other topics requested included information on the armed forces community, gender identity, health conditions, place of birth, reason for migration, sexual orientation and income. The differing needs were balanced against increasing the burden on the public from having a longer questionnaire. A stated aim of the consultation was not to increase the burden from the previous census.

Taking this into account, as well as the difficulties in designing a question suitable for the census, we concluded that for the topic of volunteering, the user need would be best met by collecting data on this topic through other surveys.

This report aims to explain the findings of research and testing that led to the final recommendation not to include a question on the topic of volunteering for Census 2021 in England and Wales.

Additional information on the overall design of the Census 2021 questionnaire can be found in the Question and questionnaire development overview for Census 2021. Details of all our research and testing can be found in the Summary of testing for Census 2021.

Work to understand the user need for information on the topic of volunteering is presented in Section 4: Research to define the user need. We discuss the evidence base for the recommendations made in the White Paper in Section 5: Research that led to the 2018 White Paper recommendations.

In June 2015, the Office for National Statistics (ONS) held a formal, 12-week consultation process asking census users for their views on the topics that were required in the questionnaire in England and Wales. The aim of the consultation was to promote discussion and encourage the development of strong cases for topics to be included in Census 2021.

In May 2016, the ONS published its response to the 2021 Census topic consultation. This set out our updated view on the topics to be included in Census 2021, including:

• a summary of proposals for new topics
• next steps to understand the user need and the type of questions needed
• a plan to develop and test questions to determine whether it is possible to meet the user need for the topic of volunteering

Our report, 2021 Census – Assessment of initial user requirements on content for England and Wales: Response to consultation (PDF, 796KB), showed that the user need for volunteering was above the threshold to be considered for further research with the potential to be included in Census 2021. However, it was the lowest user need of other new topics being considered for inclusion. New topics included gender identity, sexual orientation and armed forces. Initially, the user need for the armed forces topic was lower than that for volunteering; however, the research concluded that the administrative data available did not provide full coverage of those who previously served in the armed forces. This resulted in an increased user need for the armed forces topic.

In our response to the topic consultation, we committed to explore how to meet the user need for data on the topic of volunteering.

To do this, we first looked to understand the user need and type of questions needed. We carried out stakeholder engagement with users of volunteering information and found the main areas of interest were the time spent volunteering, to allow a measure of the contribution to the economy, and how often a person volunteers, to measure participation by volunteers.

We tested questions on both measurements, details of which are presented in Section 5: Research that led to the 2018 White Paper recommendation.

A full list of the tests used in the development of the topic of volunteering is provided in Annex 1. Further details are provided in the Summary of testing for Census 2021.

The tests utilised a range of qualitative and quantitative research methods. A short description of the different research methods and sampling techniques is given in the Question and questionnaire development overview for Census 2021. Testing included people from a wide range of backgrounds, such as employed, unemployed, student and retired participants. It also included participants who regularly or irregularly volunteered or had no volunteering background.

References to tests are provided in the form (Year: Test number). Year refers to the calendar year the test was undertaken in and the test number is the position of the test within the year considering all testing that took place in that year. For example, the fifth test conducted in 2017 would be referenced as (2017:5).

Different questions were tested to assess whether the ONS could design a question that, if included in Census 2021, would provide sufficient quality data to meet the user need. Our testing included two rounds of cognitive testing in 2016 (2016:5) and 2017 (2017:4) using an electronic questionnaire. The volunteering question was presented alongside employment questions and a question on unpaid care activities. This approach was used to see if participants were able to distinguish between time spent in unpaid care and volunteering activities.

We included a question on volunteering in the 2017 Test (2017:7) and in the follow-up Census Test Evaluation Survey (CTES). Details of the findings are presented in Section 5: Research that led to the 2018 White Paper recommendation. In December 2017, we published a census topic research update, which included the recommendation not to include a question on volunteering in Census 2021.

In December 2018, the government presented to Parliament a White Paper Help Shape our Future: The 2021 Census of Population and Housing in England and Wales. The White Paper reaffirmed our recommendation not to include a question on volunteering in Census 2021. Instead, the government should continue to gather the data needed to estimate the contribution of volunteering to the economy through household surveys such as the Community Life Survey (CLS).

During the process of making recommendations for Census 2021, we have evaluated questions for their potential impact on data quality, public acceptability, respondent burden, financial concerns and questionnaire mode. We present details of this evaluation for the volunteering question in Annex 2.

Following the 2021 Census topic consultation (PDF, 796KB), users informed us that they wanted information on volunteering to: calculate the contribution of volunteering to the economy; measure volunteer engagement; and improve volunteer-based services. Users were interested in time spent volunteering and frequency of volunteering.

While most users were more interested in the frequency of volunteering than in hours spent volunteering, we tested questions aligning with both of these needs. Data users also made us aware of some of the alternative sources available, for example, the Community Life Survey (CLS).

After the topic consultation, we engaged with several charities and central government departments to get a better understanding of users’ needs. Contact was made with stakeholders who had responded to the Census 2021 topic consultation suggesting collecting data on the topic of volunteering.

We used the National Council for Voluntary Organisations (NCVO) definition, which defines volunteering as follows:

“We define volunteering as any activity that involves spending time, unpaid, doing something that aims to benefit the environment or someone (individuals or groups) other than, or in addition to, close relatives. Central to this definition is the fact that volunteering must be a choice freely made by each individual. This can include formal activity undertaken through public, private and voluntary organisations as well as informal community participation and social action. Everyone has the right to volunteer and volunteering can have significant benefits for individuals.”

There are two types of volunteering – formal and informal. Formal volunteering includes organising events, fundraising or participating in a sponsored event. Informal volunteering includes getting shopping for a neighbour or looking after a friend’s children. The user need was for information on formal volunteering, so in defining the appropriate question wording, we chose to use the terms “groups, clubs or organisations” to differentiate between the two.

Our definition of volunteering did not include people working in the sharing economy. The sharing economy refers to the sharing of goods and services between private individuals, typically through the internet.

Developing a question on time spent volunteering

A question on volunteering had not been asked in previous censuses, and there is currently no Government Statistical Service (GSS) harmonised principle for collecting data on the topic of volunteering.

Therefore, we based the development of the question on an analysis of the topic consultation responses, which showed that users were more interested in the frequency of volunteering than in hours spent volunteering. We developed the first question in line with the questions on volunteering in the Community Life Survey (CLS) 2015-16.

The question tested was:

In the last 12 months, did you volunteer for any official groups, clubs or organisations?

Do not count any paid activities, or any activities you are required to do, for example volunteering required to obtain a qualification or community service.

Choose one of the following answers

[ ] No
[ ] Yes, less than 2 hours a week
[ ] Yes, more than 2 hours per week

The response options were developed to fit in with a former government target of individuals volunteering for 100 hours per year – approximately two hours per week of volunteering per person.

We used instructions to help participants identify what volunteering activities to include or exclude and used “volunteering required to obtain a qualification or community service” as examples of activities to exclude.

These instructions were based on the CLS instructions, which were given to participants immediately prior to being presented with the CLS question and read:

“I'd like you to think about any groups, clubs or organisations that you've been involved with during the last 12 months. That's anything you've taken part in, supported, or that you've helped in any way, either on your own or with others. Please exclude giving money and anything that was a requirement of your job.”

Research findings

This question was cognitively tested in 2016 (2016:5), from which we drew the following conclusions.

Question stem

The question was misunderstood by most participants, in particular the word “official”. There was a lack of consensus among participants as to which groups and activities to include as official or exclude as not official.

There was confusion of which activities to include in this question and what to include as unpaid care. This led to inaccurate answers to the unpaid care question.

The instructions that supported the question were too long and the phrases “activities you are required to do”, “volunteering to obtain a qualification” and “community service” all caused confusion. The terms “clubs, groups or organisations” were confusing for some participants from organisations involved in volunteering in the sharing economy.

Some participants did not include existing volunteering roles because they thought the question related to starting a new volunteering role within the last 12 months.

Participants were able to recall their volunteering over a 12-month period if they volunteered regularly. However, participants who volunteered irregularly found it difficult to remember occasions when they volunteered and how long they volunteered for.

Question guidance

The guidance was considered too long and some participants chose the quickest and easiest option over considering the accuracy of their answer.

The phrase “activities you’re required to do” was understood by some participants as actions carried out as part of a volunteering role, because they were required to do them by the organisation they volunteered for.

The term “volunteering to obtain a qualification” caused confusion to some participants. They were unsure whether it meant volunteering to obtain a qualification or volunteering to enhance their CV.

The phrase “community service” led participants to think of a volunteer as someone helping or giving back to the community, instead of someone who had to carry out work in the community as punishment ordered by a court.

Response options

Some participants thought these options were inappropriate because less than two hours was an insignificant amount of time and more than two hours did not capture the full extent of the volunteering they did.

Participants who volunteered regularly had no problem picking an option. However, participants who volunteered on an ad hoc basis or had the usual routine of their volunteering disrupted (for example, because of ill health) had difficulty recalling how long they had volunteered for over the last 12 months.

Recommendation following testing of a question on time spent volunteering

We recommended making the following changes to the question based on the findings from the cognitive test (2016:5):

• change the question to “Thinking of the last 12 months, have you taken part in any volunteering for any clubs, groups or organisations?”
• remove the word “official” from the question stem
• develop guidance defining volunteering and telling participants what to include or exclude, and attach a link to the guidance to the question
• shorten and reword the “do not count” instruction sentence
• remove instructions regarding answer options

Developing a question on frequency of volunteering

The difficulties encountered following the testing of the question on time spent volunteering led to us having further engagement with users of volunteering data to discuss a redesign of the question. We discovered that the contribution of volunteering to the economy can be estimated through household surveys and was not required at a small area level.

As a result, we recommended changing the question wording to collect data on frequency of volunteering instead of time spent volunteering. We designed a revised question to meet that user need, taking into consideration findings from the initial testing. This revised question was tested in a cognitive test (2017:4), the 2017 Test (2017:7) and the follow-up Census Test Evaluation Survey (CTES).

The question tested was:

Thinking of the last 12 months, have you taken part in any volunteering for any clubs, groups or organisations?

Exclude any court ordered activities

[ ] No
[ ] Yes, at least once a week
[ ] Yes, less than once a week but at least once a month
[ ] Yes, less often

Research findings – cognitive test

Findings from the cognitive test (2017:4) indicated that this question design was well understood by participants. However, the findings also indicated that difficulties arose for participants when they attempted to select a response option.

Question stem

The question was easily understood and perceived as a typical survey question. Participants were able to describe what the question was asking for, and those with a regular and consistent volunteering schedule had no difficulty in recalling how often they volunteered over the reference period.

However, as with the previous version of the question tested, participants who volunteered irregularly expressed experiencing difficulties answering this accurately.

Question guidance

The instruction was not noticed by some participants. Some participants noticed the instruction but dismissed it, while others found it helpful.

Overall, when read, the revised instructions were understood as intended.

Response options

Participants with regular volunteering patterns were able to select an answer option with ease, while those with irregular volunteering patterns, such as volunteering on a monthly basis, had difficulties in selecting an answer option.

Some participants also found the fourth option, “Yes, less often”, to be too vague.

Research findings – 2017 Test

We included the revised question on volunteering in the 2017 Census Test (2017:7) with the findings reported in the 2021 Census topic research: December 2017 update. The percentage of people not answering this volunteering question was low at 3.4%, which suggested that people were happy to answer this question.

Furthermore, the drop-off rate of 1.3% from the 2017 Test suggests that the volunteering question did not stand out compared to other questions. The drop-off rate is where a respondent completing the questionnaire online exits the questionnaire in that session.

Research findings – Census Test Evaluation Survey

The Census Test Evaluation Survey (CTES) was carried out following the 2017 Test and included questions that had been included in the 2017 Census Test such as the volunteering question.

The CTES helped us gauge people’s understanding, ability, willingness and attitudes towards completing the Census Test online. It also gave a better understanding of the quality of answers to specific questions in the 2017 Test, by comparing answers between the two. Unlike the 2017 test, which was conducted online, the CTES was an interviewer-led survey.

The agreement rate between responses from the 2017 Test responses and CTES for the volunteering question was 78% (reported in the 2021 Census topic research: December 2017 update). This is the percentage of respondents choosing the same response option in the two surveys. This was a lower agreement rate than for other topics, such as sexual orientation, and below the acceptance threshold that had been set at 80% in advance of the CTES.

During the CTES, respondents were asked the following three questions:

• “Which of the following activities did you consider when you said you volunteered for any groups, clubs and organisations?”
• “Which of the following did you take part in volunteering for?”
• “Which of these activities, if any, did you consider when you said you looked after family members, friends, neighbours or others?”

The list of options that a respondent could choose from is provided in Annex 3.

We found that respondents understood what was meant by a volunteering activity and what was meant by an unpaid care activity. There were no cases where an activity that we considered as volunteering was included in the unpaid care responses, nor were there cases where unpaid care was included as volunteering. However, participants also provided detailed and varied responses to the “other specify” option, which suggested that they were unable to fit their volunteering or caring activity under the broader categories in the list of response options provided.

Although the 2017 Test data were not nationally representative, we compared the estimates from the test with those from the CLS (2017 to 2018). The two surveys show a very different distribution of respondent’s volunteering patterns. The 2017 Test estimated that 3 in 10 adults did some volunteering in the past 12 months, compared with over half in the CLS. This difference raised further concerns about the accuracy and quality of the 2017 Test volunteering question.

Welsh language question development

In 2017, an external agency with Welsh-speaking researchers was commissioned to undertake focus groups (2017:17) and a series of cognitive interviews (2017:18). The volunteering question design tested was the design used in the 2017 Census Test. The instruction changed to “Do not include” to be consistent with the provision of unpaid care question design at that time.

All participants were happy with the translation of the volunteering question and its associated response options. One participant suggested simplifying the instruction to “peidiwch â chynnwys unrhyw beth sydd dan orchymyn llys” (do not include anything under a court order):

English

Thinking of the last 12 months, have you taken part in any volunteering for any clubs, groups or organisations?

Do not include any court ordered activities

[ ] No
[ ] Yes, at least once a week
[ ] Yes, less than once a week but at least once a month
[ ] Yes, less often

Welsh

Yn ystod y 12 mis diwethaf, ydych chi wedi gwirfoddoli i unrhyw grŵp, clwb neu sefydliadau?

Peidiwch â chynnwys unrhyw beth sydd dan orchymyn llys

[ ] Nac ydw
[ ] Ydw, o leiaf unwaith yr wythnos
[ ] Ydw, llai nag unwaith yr wythnos ond o leiaf unwaith y mis
[ ] Ydw, yn llai aml

A couple of participants used the online guidance to help them determine whether to include work they had undertaken in the community and considered the guidance helpful.

The translation provided is recommended for future use of this question design.

Results of the Census Test Evaluation Survey (CTES) suggested challenges to collect information of sufficient quality to meet the user need. Alongside this, our understanding of users’ needs for other topics has grown. To manage respondent burden and meet space constraints on the paper questionnaire, we recommended not collecting information on volunteering in Census 2021.

As in previous censuses, there will be separate censuses in Scotland and Northern Ireland. The questions for England and Wales have been developed through close collaboration with the National Records of Scotland (NRS) and Northern Ireland Statistics and Research Agency (NISRA), which are responsible for conducting the censuses in Scotland and Northern Ireland, respectively.

Scotland and Northern Ireland are also not planning to collect a question on the topic of volunteering in Census 2021 but are proposing to collect information on volunteering through alternative data sources, such as social surveys.

References to tests take the form (Year: Test number). “Year” refers to the calendar year the test was undertaken in and the test number is the position of the test within the year considering all testing that took place in that year. For example, the fifth test conducted in 2017 would be referenced as (2017:5).

A full description of each of these items can be found in Summary of testing for Census 2021.

Evaluation of the volunteering question

Potential for impact of this question on public acceptability and financial concerns were assessed as low in the May 2016 evaluation (PDF, 796KB).

Since completion of the research and development phase, we have repeated the evaluation on our recommended questions against the same criteria. This is based on further testing but does not include the 2019 Rehearsal data. The evaluation of topics recommended to be asked in Census 2021 included the 2019 Rehearsal data; therefore, the evaluation criteria for volunteering are not fully comparable to other questions that are recommended to be included in Census 2021.

Table 3 summarises the assessment scores for the topics of volunteering from this updated evaluation.

The potential for impact on financial concerns, public acceptability and questionnaire mode have been assessed as “Low”.

Potential for impact on data quality: “High”

Participants varied in what they considered to be the last 12 months; some were counting back from when they were asked the question, while others considered a calendar year from January. In addition, participants who volunteered irregularly found it difficult to remember occasions when they volunteered and how long they volunteered for. Participants found that it was difficult or simply not possible to answer this question by proxy.

Potential for impact on respondent burden: “Medium”

This would have been a new, additional question for Census 2021. It was a long question with over 20 words in both the question stem and the response options. In addition, some people with irregular volunteering patterns found it hard to decide which response option to choose, which is confirmed by the agreement rate between responses from the 2017 Test responses and Census Test Evaluation Survey (CTES) of 78%. The agreement rate of 78% was lower than for other topics and below the threshold of the acceptance criteria for further research that had been set at 80% in advance of the CTES.

Question 1

Thinking of the last 12 months, have you taken part in any volunteering for any clubs, groups or organisations?

• No
• Yes, at least once a week
• Yes, less than once a week but at least once a month
• Yes, less often

Question 2

Which of the following activities did you consider when you said you volunteered for any groups, clubs and organisations?

• General administration and organizing social media
• Coaching or supporting sports activities
• Activities funded by your employer
• Activities that led to a qualification
• Activities abroad
• Fundraising
• Leading and organizing activities and events
• Visiting, befriending or mentoring people
• Providing transport/driving
• Representing and campaigning
• Managing money for a person
• Helping a person to wash, get dressed or eat
• Taking care of a child
• Specify other

Question 3

Which of the following did you take part in volunteering for?

• Sports club
• Religious group
• A family member, neighbour or friend
• A company you have worked for
• Children/Youth/Older people’s group, clubs or organisation
• Political group
• Health, Disability or Social welfare group
• Environment/Animal organisation
• Citizen group
• Neighbourhood or community group
• Specify other