Shielding Behavioural Survey (SBS) QMI

This Quality and Methodology Information report contains information on the quality characteristics of the data (including the European Statistical System’s five dimensions of quality) as well as the methods used to create it.

The information in this report will help you to:

• understand the strengths and limitations of the data

• learn about the existing uses and users of the data

• understand the methods used to create the data

• help you to decide suitable uses for the data

• reduce the risk of misusing data

• The Shielding Behavioural Survey (SBS) is a representative survey of the approximately 2.2 million clinically extremely vulnerable (CEV) people in England who were advised to shield during the coronavirus (COVID-19) pandemic; the survey includes information on their behaviours and well-being since receiving shielding guidance.

• The SBS was compiled rapidly in response to policy questions on whether the population who had been advised to shield were following the guidance issued.

• It was produced, run and analysed in a collaboration between the Department for Health and Social Care, the Department for Work and Pensions, the Government Digital Service and the Office for National Statistics (ONS).

• The survey sample was selected using implicit stratification from a list of those identified as clinically extremely vulnerable (CEV) and contacted by telephone (from the National Shielding Helpline).

• The achieved sample size for the SBS has been around 4,000 individuals since the second wave of the survey; the response rate by Wave 6 of the survey was approximately 60%.

• The turnaround time for the SBS is around four weeks from finalisation of the questions to delivery of data and reports to customers, including frequency tables.

• All reasonable attempts have been made to ensure that the data are as accurate as possible; however, there are two potential sources of error that may affect the reliability of estimates and for which no adequate adjustments can be made, known as sampling and non-sampling errors.

Overview

This report relates to the Shielding Behavioural Survey (SBS), which collects information on the clinically extremely vulnerable (CEV) population, in England, and their behaviours and well-being under the shielding guidance.

Uses and users

The results of the SBS are used by the Department of Health and Social Care (DHSC), Department for Work and Pensions (DWP), Cabinet Office (CO), Ministry of Housing, Communities and Local Government (MHCLG), and other government departments to obtain a fast turnaround of shielding behavioural and well-being data every two weeks.

The data provide a greater understanding of the extent to which CEV individuals have followed shielding guidance and their behaviour, health and well-being, and support needs since being advised to shield.

Strengths and limitations

The main strengths of the SBS include:

• the survey is a unique source of data for many uses

• it has a quick turnaround from finalising the questions to delivery of outputs which allows for timely communication of data and statistics

• it can respond quickly to changing user needs, as the questions included in the survey are reviewed for each wave

• consistency in several core questions allows data to be combined across waves to increase sample size

• robust methods are adopted for the survey's sampling and weighting strategies to limit the impact of bias

• quality assurance procedures are undertaken throughout the analysis stages to minimise the risk of error

The main limitations of the SBS include:

• the survey asks whether the respondent received a letter or text message advising them to shield; if the respondent did not receive a letter or text message, they are asked the same questions as those who did, despite not knowing they were advised to shield

• in the case of minors or those who are unable to answer the survey themselves, other people can answer on their behalf; this is mainly a limitation for questions relating to well-being and mental health

• the sampling approach was not fully stratified because of operational limitations

• call centres normally operated on weekdays between 9am and 6pm and repeated attempts to contact potential respondents were only made during the second wave of the survey, which may impact response rates and non-sampling error

• when considering subgroups of the surveyed population, the achieved sample size is relatively small which can lead to poor quality estimates for people with specific characteristics in a single wave

• changes made over time to both the questionnaire design and to the shielding guidance limit the range of time series analyses and combined analyses that can be conducted across waves

• measures of uncertainty have not been provided for all estimates to allow timely publication of outcomes

Changes to the shielding guidance

Changes to government guidance for shielding were made between, and during, data collection periods (waves) of the SBS. New guidance for CEV individuals advised to shield was released on 1 June 2020, during the data collection for Wave 3 (28 May to 2 June 2020), and on 6 July 2020 (between Waves 5 and 6). Further changes will apply from 1 August 2020, after the completion of Wave 6 (9 to 16 July 2020).

Changes to the SBS questionnaire

Questions have been improved or removed based on testing and analysis, and to ensure they align with government updates in guidance for people who have been identified as CEV.

Relevance

(The degree to which the survey meets users' needs.)

The Shielding Behavioural Survey (SBS) collects data on topics that were not sufficiently covered elsewhere, from a subgroup of the population not specifically covered elsewhere. As the survey is fortnightly and multipurpose, collecting information on topics for customers that commission the questionnaire modules, data are always relevant for the users.

The SBS is used for:

• providing quick answers to questions of immediate policy interest

• measuring how well the shielding guidance is being followed

Topics that have been commissioned by customers on the survey include questions relating to state benefits and tax credits.

Accuracy and reliability

(The degree of closeness between an estimate and the true value.)

Multiple quality assurance methods ensure that the SBS outputs are as accurate as possible. These methods are applied during the development of the questionnaire, when drawing the sample and after data collection. A series of checks are used to identify invalid responses and inconsistencies in the data, including comparisons of the data between waves. However, there are two potential sources of error that may affect the reliability of estimates for which no adequate adjustments can be made, known as sampling and non-sampling errors.

Sampling error

As the SBS is a representative sample survey, it provides estimates of population characteristics rather than exact measures. In principle, many random samples could be drawn and each would give different results, because each sample would be made up of different people who would give different answers to the questions asked. The spread of these results is the sampling variability. Sampling variability is dependent on several factors, including:

• the size of the sample

• the effects of the sampling method

• the effects of weighting

Confidence intervals are used to present the sampling variability, where it is expected that in 95% of all possible survey samples the range will contain the true value that would be obtained by surveying the whole population.

More information on how we measure and communicate uncertainty for our surveys is available.

Non-sampling error

The main sources of non-sampling error are:

• frame under-coverage

• non-response

• response errors (such as misleading questions or interviewer bias)

• errors when imputing or processing data.

To minimise the effects of non-sampling errors, the questionnaire is carefully designed and tested, interviewers are trained, and extensive quality control procedures are used throughout. Weighting is also used to compensate for non-response and frame under-coverage.

Coherence and comparability

(Coherence is the degree to which data that are derived from different sources or methods, but refer to the same topic, are similar. Comparability is the degree to which data can be compared over time and domain, for example, geographic level.)

Although many of the variables are comparable between waves of the SBS, some of the questions included have changed over time, as have the categories of responses for particular variables.

There is limited comparable data from administrative sources or major surveys for the topics covered by the SBS. The Opinions and Lifestyle Survey (OPN) has collected information on reasons for leaving the home during the coronavirus (COVID-19) pandemic and life satisfaction among individuals (aged 16 years or over) in Great Britain. Comparison of results from the SBS and the OPN have been made as part of the validation process.

Accessibility and clarity

(Accessibility is the ease with which users can access the data, also reflecting the format in which the data are available and the availability of supporting information. Clarity refers to the quality and sufficiency of the release details, illustrations and accompanying advice.)

Our recommended format for accessible content is a combination of HTML web pages for narrative, charts and graphs, with data being provided in usable formats such as CSV and Excel. Our website also offers users the option to download the narrative in PDF format. In some instances, other software may be used or may be available on request. Available formats for content published on our website but not produced by us, or referenced on our website but stored elsewhere, may vary. For further information please refer to the contact details at the beginning of this report.

Timeliness and punctuality

(Timeliness refers to the lapse of time between data collection and data delivery. Punctuality refers to the gap between planned and actual data delivery dates.)

The SBS collects data every two weeks. Initially, the telephone data collection period lasted five days, however, this has increased to two weeks since 9 June 2020. After the data collection period, the data are cleaned, weighted, analysed and tabulated by the research team. Data and reports are sent to customers on an agreed date, approximately two weeks after the data collection period.

For more details on related releases our release calendar provides advance notice of release dates.

Concepts and definitions

(Concepts and definitions describe the legislation governing the output, and a description of the classifications used in the output.)

Proxy response

The SBS collects proxy responses in cases where the clinically extremely vulnerable (CEV) individual is under the age of 16 years, or unable to respond to the survey themselves.

Why you can trust our data

We are the UK's largest independent producer of statistics and its National Statistics Institute. The Data Policies and Information Charter details how data are collected, secured and used in the publication of statistics. We treat the data that we hold with respect, keeping it secure and confidential, and we use statistical methods that are professional, ethical and transparent. More information about our data policies is available.

The SBS is carefully designed and tested, and extensive quality control procedures are used throughout.

The data are collected using telephone interviewing through the National Shielding Helpline. To date, data have been collected over the following time periods:

• Wave 1: between 28 April and 2 May 2020

• Wave 2: between 14 May and 19 May 2020

• Wave 3: between 28 May and 3 June 2020

• Wave 4: between 9 June and 18 June 2020

• Wave 5: between 24 June and 30 June 2020

• Wave 6: between 9 July and 16 July 2020

Sampling frame

The sample for the Shielding Behavioural Survey (SBS) is drawn from the shielded patient list (SPL), which consists of around 2.2 million patients at high risk of developing complications from coronavirus (COVID-19) infection. Patients are identified by hospitals and general practices or from national administrative datasets using a clinical algorithm. The algorithm looks to identify:

• solid organ transplant recipients

• people with specific cancers

• people with severe respiratory conditions including all cystic fibrosis, severe asthma and severe chronic obstructive pulmonary (COPD)

• people with rare diseases and inborn errors of metabolism that significantly increase the risk of infections (such as severe combined immunodeficiency (SCID), homozygous sickle cell)

• people on immunosuppression therapies sufficient to significantly increase risk of infection

• women who are pregnant who have significant heart disease, congenital or acquired

People with specific cancers includes:

• people with cancer who are undergoing active chemotherapy

• people with lung cancer who are undergoing radical radiotherapy

• people with cancers of the blood or bone marrow such as leukemia, lymphoma or myeloma who are at any stage of treatment

• people having immunotherapy or other continuing antibody treatments for cancer

• people having other targeted cancer treatments which can affect the immune system, such as protein kinase inhibitors or PARP inhibitors

• people who have had bone marrow or stem cell transplants in the last 6 months, or who are still taking immunosuppression drugs

The SPL covers all patients that interact with NHS services in England, which includes some patients that live in other countries outside of England. The sampling frame used for the SBS covers patients that live in England only.

The SPL is managed, published and disseminated by NHS Digital. The SPL is updated daily with hospital data, address changes and where patients have passed away and is published weekly.

Sampling

Each wave, between 10,000 and 16,000 individuals have been selected to take part in the SBS. A sample was drawn from the SPL, across three explicit strata:

• individuals registered on the government website, or by telephone, who indicated that they require support

• individuals registered on the government website, or by telephone, who indicated that they do not require support

• individuals who have not registered

Within each stratum, the list was ordered by geographical region, sex and age. A systematic sample was then drawn from a random starting place to give an implicitly stratified sample.

Based on management information from the first wave of the survey and the assumption that 20% to 30% of the sample would lead to a response (accounting for non-contacts and refusal to participate), the following sampling strategy was implemented:

• registered and receiving support: 1,800 individuals selected each wave

• registered and not receiving support: 4,200 individuals selected each wave

• not registered: 8,000 individuals selected in Wave 2 and 10,000 individuals selected in Waves 3 to 6

People without known telephone numbers and those who had been selected to take part in previous waves of the SBS were excluded from selection. Repeated attempts to contact those selected were only made in the second wave of the survey. For operational reasons and speed of data collection, the number of individuals selected from the SPL was increased in subsequent waves to achieve the same sample size without the need for repeated calls. Proxy interviews were permitted in cases where the clinically extremely vulnerable (CEV) individual is under the age of 16 years or unable to respond to the survey themselves.

Although initial response rates were higher, the average response rate across all three strata was approximately 60% by Wave 6 of the survey. The response rate was higher among those registered and receiving support (around 70%) and lower among those not registered (around 50%). The achieved sample size was around 2,700 respondents for Wave 1 and between 4,000 to 4,300 for each subsequent wave.

The sample estimates were weighted to provide estimates for the entire population of CEV individuals in England (approximately 2.2 million). The estimates were weighted by taking the following into account:

• whether the respondent was registered and receiving support, registered and not receiving support or not registered

• gender

• age group: aged under 20 years, 20 to 29 years, 30 to 39 years, 40 to 49 years, 50 to 59 years, 60 to 69 years, 70 to 74 years, and aged 75 years and over

How we process the data

The response data are validated, variables are derived, and weights are applied. As the SBS collects information on a sample of the population, the data are weighted to enable us to make inferences from this sample to the entire CEV population.

Incomplete responses are excluded from all analyses. If respondents chose not to report their age or gender, these variables were imputed to allow weights to be applied. Replacement values were randomly drawn from the distribution of demographics from the underlying CEV population in England. Quality assurance methods are applied throughout the process, with validation checks conducted for derived variables and aggregated data and to identify inconsistencies in data between waves.

Statistical disclosure control methodology is applied to all outputs produced from the SBS. This ensures that information attributable to an individual is not identifiable in any published outputs. The Code of Practice for Statistics and specifically the pillar on trustworthiness, sets out principles for how we protect data from being disclosed.

All SBS outputs that are based on a count of fewer than 10 individuals are suppressed from publication to avoid disclosure. Outputs that are based on a count of fewer than 30 individuals are suppressed on quality grounds, but for completeness they may be included in charts with quality concerns highlighted.