Crime Survey for England and Wales data quality review: June 2024

• The quality of Crime Survey for England and Wales (CSEW) data in year ending (YE) March 2023 was broadly similar to CSEW data for YE March 2020, when they were accredited official statistics.

• Statistical tests showed the decreased sample size in YE March 2023 had a minimal impact on precision and uncertainty of CSEW headline crime estimates; the YE March 2023 survey was able to detect a 4.9% statistically significant change in CSEW headline crime at the 95% confidence level compared with 4% for YE March 2020.

• The CSEW response rate fell from 64% for YE March 2020 to 42% in 2022 to 2023 but has now stabilised; this decrease has not affected the representativeness of the survey and previous research provides evidence that lower response rates have only a minor impact on estimates.

• The main sample characteristics in CSEW 2022 to 2023 were more closely aligned to Census 2021 population figures than CSEW 2019 to 2020 and further post-survey adjustments allowed the survey to be representative of the household population of England and Wales.

The Crime Survey for England and Wales (CSEW) is a face-to-face victimisation survey, which asks people resident in private households in England and Wales about their experiences and perceptions of crime. The core sample is designed to be representative of the population of households in England and Wales and people living in those households. Findings are used by policymakers to evaluate measures in place to reduce crime and assess the performance of policing and criminal justice organisations.

The CSEW was first conducted in 1982 and the core methodology of a face-to-face survey has remained consistent over time. Before the coronavirus (COVID-19) pandemic, CSEW response rates were relatively consistent year to year and the target sample sizes were always achieved. In the 2019 to 2020 fieldwork period, both the total number of interviews achieved and the final response rate were below target as a result of fieldwork being halted in mid-March because of the pandemic.

During the pandemic, a revised approach was launched – the telephone-operated Crime Survey for England and Wales (TCSEW) – while face-to-face interviewing was suspended. The TCSEW results were based on a follow-up sample of telephone interviews from respondents who had taken part in the CSEW in the last two years. TCSEW data collection took place between 20 May 2020 and 31 March 2022.

Although the TCSEW was set up as a temporary measure to overcome pandemic restrictions, it did demonstrate the feasibility of a telephone follow-up approach. As the main measures of crime captured in both the CSEW and TCSEW were found to be broadly comparable when certain adjustments were applied, the CSEW was changed into a longitudinal survey when face-to-face fieldwork resumed in October 2021. While the approach of the survey at Wave 1 remains largely unchanged compared with the long-term CSEW methodology, Wave 2 and subsequent interviews are conducted via telephone with respondents who had taken part in a face-to-face interview 12 months before (Wave 1).

For more information on the CSEW, see our User guide to crime statistics for England and Wales.

This report refers only to the quality characteristics of Wave 1 data, as statistical outputs are currently based on Wave 1 data only. We are currently investigating the comparability of Wave 1 and 2 data to assess how these data can be used together in future statistical outputs.

Aims of the review

Accredited official statistics designation for CSEW statistics were suspended in 2022 because of the quality concerns for these estimates as a result of the reduced data collection period (6 instead of 12 months) for year ending (YE) March 2022 and the potential impacts of lower response rates and sample size on the quality of the estimates for YE March 2023.

Accredited official statistics (formerly called National Statistics and still referred to as National Statistics in the Statistics and Registration Service Act 2007) are official statistics that have been independently reviewed by the Office for Statistics Regulation (OSR) and confirmed to comply with the standards of trustworthiness, quality and value in the Code of Practice for Statistics (PDF, 577KB). Therefore, reinstatement of the accredited official statistics badge is important to demonstrate to users that the concerns we had about the quality of the CSEW have been fully investigated and resolved.

CSEW has now achieved a consistent 12-months data collection and there is evidence that the lower response rate has limited impact on quality of estimates, which were the two main concerns resulting in the temporary suspension of the badging.

To demonstrate CSEW quality compliance to the quality requirements of the Code of Practice for Statistics, we concentrated on exploring the impact on quality for the following three areas:

• sample size, which affects the accuracy, reliability and generalisability of the survey results; a larger sample size generally leads to more accurate results that better represent the target population

• response rate, as a higher response rate increases the likelihood the survey sample accurately represents the target population; similar to the sample size, a higher response rate generally increases reliability and validity of the data collected

• representativeness of the sample, as a representative sample ensures that survey results accurately reflect the characteristics, experiences, opinions and behaviours of the target population, leading to more accurate and credible results and reduced bias

To assess the quality characteristics of the Crime Survey for England and Wales (CSEW) data for the year ending (YE) March 2023, we are comparing sample size, response rates and sample composition with CSEW data for YE March 2020, when these statistics last had accredited official statistics status.

Sample size

In general terms, the larger the sample, the less uncertainty there is around the estimates. The 2022 to 2023 CSEW sample was designed to yield interviews with a nationally representative sample of 34,000 households in England and Wales. This is similar to the number of interviews achieved in the 2019 to 2020 CSEW (33,734). However, severe disruption to face-to-face fieldwork during the coronavirus (COVID-19) pandemic resulted in a sharp decrease in the survey response rates and inability to meet the required interview targets for 2022 to 2023. In October 2022, a boost sample was issued to achieve an additional 5,850 interviews. Following further challenges with fieldwork for the boost sample, 31,183 interviews in total were achieved for the 2022 to 2023 CSEW. We expect a similar sample size for 2023 to 2024.

Table 1 presents CSEW headline crime estimates for YE March 2020 and YE March 2023 alongside three measures for uncertainty related to sample size:

• 95% confidence intervals

• standard errors

• coefficients of variation

These were also calculated for selected crime types (Appendix Table 1). CSEW headline crime captures a range of personal and household victim-based crime in the interviewer-led parts of the survey, including theft, robbery, criminal damage, fraud, computer misuse and violence with or without injury.

As CSEW is based on a complex sample design, the standard errors presented are complex standard errors that take the sample design into account. Standard errors for incidence of total theft and CSEW headline crime (that include both individual and household offence incidents) were calculated by averaging standard errors of corresponding personal and household offences. Coefficients of variation (relative standard errors) were then calculated by dividing the standard error of an estimate by the estimate itself.

To calculate confidence intervals around an estimate, we used the standard error for that estimate. The estimate and its 95% confidence interval are presented as the estimate plus or minus the margin of error. The lower and upper 95% confidence limits are given by the sample estimate plus or minus 1.96 standard errors. The margin of error is calculated as: margin of error equals 1.96 multiplied by standard error (see our Uncertainty and how we measure it methodology).

The measures showed that there was only a small increase in uncertainty in CSEW headline crime estimates as the sample size decreased. The coefficient of variation makes it easier to understand whether a standard error is large compared with the estimate itself and can be used to compare the relative precision across surveys and outcomes. For example, the standard error itself was slightly lower for the YE March 2023 CSEW than the YE March 2020 CSEW because the estimated number of incidents was considerably lower in YE March 2023.

However, after accounting for the size of the estimate, the coefficient of variation was slightly higher for YE March 2023 (2.2%) compared with YE March 2020 (1.9%). Despite this small decrease in precision, the YE March 2023 survey was still able to detect a 4.9% statistically significant change in CSEW headline crime at the 95% confidence level. The detectable percentage change was calculated by setting a z-score of 1.96 (the threshold for significance at the 95% confidence level) and keeping the sample size the same for each survey year. This suggests a high level of precision in both samples despite the challenges with the sample size of the YE March 2023 survey.

Appendix Table 1 shows that the coefficients of variation across all selected crime types were below 10% in the YE March 2020 and the YE March 2023 CSEW, except for robbery and theft from the person. The coefficient of variation for robbery and theft from the person increased from 15.1% and 8.1% in YE March 2020 to 18.0% and 11.7% in YE March 2023, respectively. These decreases in precision were most likely because of a fall in the numbers of respondents reporting these crimes, which has led to more volatility in the estimates because of low volumes. Although there is no set threshold for acceptability when using these coefficients, values are still below commonly-used thresholds of acceptability for household surveys at national statistical bodies, such as the Office for National Statistics (ONS) (20%, according to our Reliability report and Statistics Canada (33.3%, according to their Quality level guidelines).

Response rate

In the 10 years prior to the coronavirus (COVID-19) pandemic, the response rate for the CSEW fluctuated between 70% to 76% (Figure 1). However, the response rate for 2019 to 2020 decreased to 64% as a result of fieldwork being halted in mid-March because of the pandemic. Similar to other social surveys, there have been further falls in the response rate following the pandemic because of ongoing challenges facing survey research. In 2022 to 2023, our response rate decreased to 42%. We have seen improvements in the response rate for the 2023 to 2024 survey year and, although fieldwork is not yet complete, based on performance so far, we estimate that the response rate for 2023 to 2024 will be approximately 45%.

Similar trends have been observed across other household social surveys. For example, our Labour Force Survey's pre-pandemic total response rate for Great Britain excluding imputed cases at Wave 1 was 54.6% in the January to March 2019 period but dropped to 32.7% in the January to March 2024 period. Meanwhile, the Scottish Crime and Justice Survey (PDF, 940KB) saw a decrease in response rate from 63.4% in 2019 to 2020 to 47.3% in 2021 to 2022.

Response rates for household surveys have seen a general decline over time because of increases in refusals and non-contacts, as noted in Survey Nonresponse Trends and Fieldwork Effort in the 21st Century: Results of an International Study across Countries and Surveys. Survey research is now facing further challenges because of further drops in response rates following the pandemic. Our article, Impact of COVID-19 on ONS social survey data collection, suggests that changes in people's behaviours in relation to working patterns, communication preferences and their willingness to invite interviewers inside their homes are related to increases in non-contacts and refusals.

We acknowledge that the response rate for the CSEW is now substantially lower than the pre-pandemic average and we are aiming to improve the response rate over time. However, given the challenges now faced by all social surveys, we think a return to pre-pandemic levels of response rate is unlikely.

Lower response rates can be problematic if this affects the representativeness of the sample and the quality of the survey estimates. However, evidence suggests that lower response rates are not necessarily linked to non-response bias and data quality. There is direct evidence that this is also the case for data from the CSEW and the Scottish Crime and Justice Survey, with estimates based on final achieved samples being very close to estimates after only one or two call attempts and before any post-stratification weighting has been applied. For further information on the effects of lower response rates see the following articles: Rethinking Response Rates: New Evidence of Little Relationship Between Survey Response Rates and Nonresponse Bias, Fieldwork Effort, Response Rate, and the Distribution of Survey Outcomes: A Multilevel Meta-analysis, Nonresponse Rates and Nonresponse Bias in Household Surveys.

Sturgis and colleagues' meta-analysis analysed the distribution of face-to-face household survey outcomes in relation to repeated interviewer calls across household surveys, including the CSEW. They found that most outcome variables across these surveys closely resembled the final sample distribution after one or two call attempts before post-stratification weighting. On average, the expected difference from the final sample proportion decreases from 1.6% after one call to 0.7% after three calls and to 0.4% after five calls.

These findings challenge the assumption of a strong link between response rate and non-response bias and suggest that, as response rates decrease, more focus should be given to ensuring the sample is representative of the population it covers and that there is better use of post-survey adjustments to repair imbalances after data collection.

Representativeness of sample

To assess the representativeness of samples, we compared the sample characteristics for CSEW 2019 to 2020 and CSEW 2022 to 2023 with Census 2021 population figures for England and Wales. Census 2021 provides the most accurate figures of the people and households in England and Wales and the best time point for fairly comparing the representativeness of both CSEW 2019 to 2020 and CSEW 2022 to 2023. This is because fieldwork targets a representative sample at the time the survey takes place, so it was more accurate to compare CSEW 2019 to 2020 with Census 2021 rather than the 2011 Census. Addresses are sampled from the Postcode Address File (PAF) across 42 police force areas and are filtered to exclude obvious non-residential addresses. They are not sampled based on census characteristics. Appendix Table 5 presents the survey profile with the appropriate design weights applied (either household or individual weight) but without the application of the calibration weighting (see our CSEW technical report volume one.

The sample characteristics in both CSEW 2019 to 2020 and CSEW 2022 to 2023 are similar to the census population figures, with the CSEW 2022 to 2023 sample characteristics generally being closer to these figures than the CSEW 2019 to 2020 (see Appendix Table 5).

Similar to CSEW 2019 to 2020, the CSEW 2022 to 2023 sample was over-represented by older age groups (those aged 55 to 64, 65 to 74, and 75 to 84 years) and those in higher managerial, administrative and professional occupations. The sample was under-represented by younger age groups (those aged 16 to 19, 20 to 24, and 25 to 34 years), those in routine or manual occupations and those who had never worked or were long-term unemployed. However, CSEW 2022 to 2023 was more similar to Census 2021 than CSEW 2019 to 2020 across these categories. For example, men aged 20 to 24 years were under-represented in CSEW 2019 to 2020 by 3.7 percentage points but only 2.6 percentage points in CSEW 2022 to 2023 compared with Census 2021 figures.

At the household level, both CSEW samples were also over-represented by homeowners, those living in whole houses or bungalows and one or two person households while being under-represented by those who were private renting, living in flats or apartments and three or more person households. However, CSEW 2022 to 2023 was more similar to Census 2021 than CSEW 2019 to 2020 across these categories with the exception of household size. For example, private renters were under-represented in CSEW 2019 to 2020 by 2.4 percentage points but only 1.8 percentage points in CSEW 2022 to 2023 compared with Census 2021 figures.

Calibration weights are designed for the CSEW to adjust for known differences in response rates between different age and sex sub-groups and households with different age and gender composition. The calibration weights adjusted the CSEW 2022 to 2023 sample to be closer to Census 2021 figures for age, tenure, and household size. For example, while those aged 65 to 74 years were over-represented in CSEW 2022 to 2023 by 2.7 percentage points compared with Census 2021 figures, calibration weights reduced this to 0.1 percentage points (see Appendix Table 6). These measures further ensure that the CSEW sample is representative of the household population of England and Wales.

Given the impact of the sample size reduction on precision was minimal and the sample characteristics are more aligned to Census 2021, we are confident that the quality and accuracy of the 2022 to 2023 survey data is broadly similar to CSEW 2019 to 2020, when these statistics last had accredited official statistics status. We are also confident that they comply with the standards of trustworthiness, quality and value in the Code of Practice for Statistics (PDF, 577KB).

We will continue to ensure the quality of the Crime Survey for England and Wales (CSEW) by maintaining our sample size, making incremental improvements to increase response rates, and continuing to monitor and make post-survey adjustments to ensure our sample is representative of the household population of England and Wales. We will also ensure that we update relevant documentation on CSEW quality and methodology and keep stakeholders up to date with any changes.

While this report refers to the quality characteristics of Wave 1 data, we are currently investigating the quality of Wave 2 data and the comparability of Wave 1 and 2 data to assess how these data can be used together in future statistical outputs. We will make further information on the progress of this work and our future plans for seeking accreditation available at the earliest opportunity.