Comparability between the Telephone-operated Crime Survey for England and Wales and the face-to-face Crime Survey for England and Wales

The Telephone-operated Crime Survey for England and Wales (TCSEW) became operational on the 20 May 2020. It was a replacement for the face-to-face Crime Survey for England and Wales (CSEW) which was suspended on the 17 March 2020 because of the coronavirus (COVID-19) pandemic. It was set up with the intention of measuring the level of crime during the pandemic. As the pandemic has continued throughout the 2020/21 survey year, questions have been raised as to whether the year ending March 2021 TCSEW is comparable with estimates produced in earlier years by the face-to-face CSEW.

This report explores those factors which may have a bearing on the comparability of estimates between the TCSEW and the former CSEW. These include survey design, sample design, questionnaire changes and modal changes.

It concludes that the main measures of crime (numbers of incidents and prevalence rates in the previous 12 months) are broadly comparable provided the estimates are on the following basis:

• the population of study is restricted to these aged 18 years and over
• overlapping data periods must not be used for the main estimates of crime
• the threat/harassment screener question must be removed from both the current and comparator years for the main estimates of crime

Separate consideration should be given to the comparability of other data gathered from survey questions on the TCSEW; modal differences should be considered particularly where socially desirable responses may affect the estimates

Background to the TCSEW

Following the suspension of the CSEW in March 2020 the Office for National Statistics in conjunction with the Crime Survey contractor KANTAR embarked on a programme of work to investigate alternative approaches to operating the Crime Survey during the coronavirus pandemic. A range of options were considered. Many decisions made at that time assumed that any alternative to the CSEW survey operation would be relatively short lived. With the face-to-face survey still in suspension, the telephone survey has remained in operation throughout the 2020/21 survey year, raising the possibility of making comparisons with data from earlier survey years.

The sample design for the Telephone-operated Crime Survey for England and Wales (TCSEW) was decided on in light of what was practically feasible, as well as what was theoretically possible. Consideration was given to the purpose of the survey, accuracy, and the time taken to implement. This last factor was important as the face-to-face survey had already been suspended, with no consideration to any alternative approach to measuring crime.

Although the initial concern for the sample design was in relation to the achievable sample size, other factors needed to be considered including the potential for any additional bias from a new sample. Whilst consideration was given to a range of strategies, including the use of an online portal to collect telephone numbers, and tele-matching, the eventual design was to make use of respondents to the Crime Survey for England and Wales (CSEW) who had already agreed to take part in further research studies.

In summary, the new sample was formed from all former CSEW adult respondents who had agreed to be re-contacted and supplied a syntactically valid phone number over the previous two years. As this was estimated to be around 40,000 cases, we knew that this would not provide a large enough sample for the TCSEW for the period (May 2020 to January 2021). To rectify this, the TCSEW sample incorporated a panel design (repeat interviewing) similar to other surveys such as the Labour Force Survey (LFS). This enabled the sample to be conserved for a much longer period. In fact, we achieved 36,801 interviews overall, and the survey sample expanded to cover the survey year from May 2020 to March 2021.

TCSEW estimates for the year ending March 2021 are therefore based on 36,801 telephone interviews with 17,167 people aged 18 years and over. Respondents have been interviewed at three monthly intervals to a maximum of three interviews (each interview called a wave). The Wave 1 response rate was 49.7%, at Wave 2 it was 79.4%, and at Wave 3 it was 81.1%.

TCSEW estimates are based on interviews rather than respondents (as in the CSEW) and this distinction is reflected in the associated published tables. As a result, standard errors on the TCSEW are higher than those on the CSEW since they are based on re-interviewing the same person rather than interviewing the person once. In effect, the TCSEW sample design has a stronger cluster effect (and therefore larger standard error) once the original sample strata and primary sampling units are considered.

Drawing a sample based on previous respondents means the sample is not unique and is in fact a subset of the households/individuals selected in the original sample. The CSEW under normal circumstances is based on a unique sample of individuals which changes from year to year. This means one survey year on the CSEW can be compared with the next. This is not possible where the samples are not unique since the time periods over which incidents are captured by any two corresponding survey years overlap.

Figure 1: The reference periods for Crime Survey for England and Wales (CSEW) year ending March 2020 and the Telephone-operated Crime Survey for England and Wales (TCSEW) interviews conducted from May 2020 to March 2021

Source: Office for National Statistics

Download this image Figure 1: The reference periods for Crime Survey for England and Wales (CSEW) year ending March 2020 and the Telephone-operated Crime Survey for England and Wales (TCSEW) interviews conducted from May 2020 to March 2021

.png (92.0 kB) .xlsx (21.7 kB)

The two diagrams above represent the reference periods over which crimes are collected by the TCSEW and the year ending March 2020 CSEW. A person being interviewed is asked about any incident they experienced in the 12 months prior to interview. This means that over two consecutive survey years the reference period in which incidents are collected are not unique.

This is not an issue with the CSEW where the respondents are different in different years. However, as the TCSEW is made up of respondents interviewed over the previous two years of the CSEW, the survey potentially recaptures the same incidents recorded by the same individual. Consequently, the TCSEW can only be compared with CSEW estimates where the time period over which incidents are collected are completely separate. Therefore, the most recent year a comparison can be made with is the CSEW year ending March 2019.

The TCSEW also differs from the normal CSEW sample as respondents have all aged since they were sampled. Since the TCSEW sample was derived from the previous two years of the CSEW (which interviews all adults aged 16 years and over) there were few people aged under 18 years in the TCSEW sample. As a result, only those aged 18 years and over were selected for interview in the TCSEW. Therefore, to be able to compare the TCSEW and CSEW, analysis of the CSEW needs to be restricted to those aged 18 years and over.

To better understand the impact of moving from face-to-face to telephone survey operation it would have been optimal to run both surveys in parallel, to account for the impact of seasonality and societal change. This type of experimental design was not possible as the Crime Survey for England and Wales (CSEW) was suspended without notice.

An alternative approach to assessing comparability is to compare the sample composition of the two surveys, with the aim of understanding what bias there may be between one survey and the other. The CSEW and the Telephone-operated Crime Survey for England and Wales (TCSEW) both use complex weighting mechanisms designed to adjust for a range of factors including the probability of selection and non-response. Making use of the fact that the TCSEW was sampled from the CSEW means much of the non-response bias between the face-to-face survey and the telephone survey can be adjusted using an additional weighting mechanism. This improves the reliability of the TCSEW estimates and its comparability with the CSEW.

The analysis conducted therefore compares weighted and unweighted distributions of a range of demographic and socio-economic characteristics between the surveys. The unweighted distributions give some idea of the differences between the two samples whilst the weighted distributions indicate the difference between the two estimates once the data have been adjusted for. Variables used in the analysis were chosen on the assumption that they would not have changed (or changed little) between the time of the interview on the CSEW and that of the TCSEW. The distributions were produced with appropriate robust standard errors and confidence intervals. As almost all of respondents answer questions on their characteristics, the unweighted bases for these variables are large and estimates calculated are precise. Therefore, even only slight differences can be statistically significant.

In addition to the weighting mechanisms already mentioned, the CSEW and TCSEW both use calibration weighting to adjust for differential response rates between regions and age by sex sub-groups. Therefore, the weighted distributions of these variables will be comparable as they employ the same methodology. Analysis of these variables focused on the unweighted distributions to understand differential non-response to the TCSEW compared with the CSEW.

Data are compared for the CSEW for the year ending March 2019 and the TCSEW for data collected in Waves 1,2 and 3 between May 2020 and March 2021. Multiple years of CSEW data are also analysed to examine the normal variation expected between sample years as well as population changes measured by the survey.

Results

Analysis showed the sample composition of the CSEW and the TCSEW is similar. As a result, the bias is unlikely to impact on the comparability of crime estimates. This is because:

• the majority of personal and household characteristics showed no meaningful difference between the weighted distributions of the CSEW and TCSEW
• while there were a greater proportion of older respondents to the TCSEW survey, this difference has been adjusted for in the weighting
• although there were differences observed between the two samples for marital status, disability and tenure, these differences are small or likely reflect genuine changes in society and therefore are unlikely to have a meaningful impact on crime estimates.

Age, sex and region

The CSEW and TCSEW use age, sex and region population totals in the calibration weighting to adjust the sample to known distributions. Despite this correction it remains important to understand the differences in the unweighted distributions between the two sources to highlight the differences in the coverage of the surveys.

The TCSEW showed a greater proportion of older respondents and a smaller proportion of younger respondents compared with the CSEW (see Figure 2). For the CSEW year ending March 2019, 37.4% of respondents were aged 60 years and over. In comparison this age group accounted for a much larger proportion of the TCSEW sample, 50.8%. Similarly, 12.1% of respondents to the CSEW were aged 18 to 29 years compared with only 6.3% in the TCSEW sample.

There was little variation between the two surveys in the unweighted distributions for sex and region.

Individual socio-demographic characteristics

Data collected for the two surveys showed that there was no meaningful difference in the sample composition for the majority of personal characteristics investigated. There was no statistically significant variation between the weighted distributions for ethnicity and religion. This indicates that there is little bias in the TCSEW for these variables that might have an effect on estimates of incidence and victimisation (see Table 1).

For marital status, the only category that showed a significant difference between the two samples was adults that are “separated”. For the CSEW year ending March 2019, an estimated 1.9% (confidence interval: 1.7% to 2.0%) of the population were separated. This is lower in the TCSEW year ending March 2021, which is 1.5% (confidence interval: 1.4% to 1.8%). As this category makes up only a small proportion of the overall distribution it is unlikely to have any meaningful impact on estimates of crime.

Significant differences were evident between the two samples for disability. The CSEW year ending March 2019 dataset estimated that 17.1% (confidence interval: 16.6% to 17.6%) of the population were disabled. This is 2.1 percentage points lower than the proportion estimated by the TCSEW, 19.2% (confidence interval: 18.5% to 20.0%). The definition of disability used is consistent with the Equality Act 2010: an individual is classified as disabled if they have a physical or mental impairment that has a substantial and long-term negative effect on their ability to do normal day-to-day activities, including mental health conditions. Evidence has shown that self-reported mental health and wellbeing has worsened during the coronavirus (COVID-19) pandemic, as a result the greater proportion of disabled adults in the TCSEW sample probably reflects a genuine change rather than sample bias (COVID-19: mental health and wellbeing surveillance report).

Household characteristics

The weighted distributions of household characteristics were similar and showed little variation in the sample composition of the CSEW and the TCSEW. There was no difference identified for area type (urban/rural) and the structure of the household (see Table 2).

Small significant differences between the CSEW and the TCSEW appeared in the sample composition for tenure. A greater proportion of adults reported living in an owner-occupied property in the TCSEW, 65.5% (confidence interval 64.5% to 66.5%). This was 2.6 percentage points higher than the CSEW year ending March 2019, 62.9% (confidence interval: 62.2% to 63.5%).

This was matched by a decrease in the proportion of private renters in the TCSEW, 18.1% (confidence interval: 17.2% to 19.0%) compared with 20.2% (confidence interval: 19.7% to 20.8%) in the CSEW year ending March 2019. The proportion of social renters remained similar between the two samples and the overall distributions remained comparable (Figure 3).

Research has shown that people living in private rented accommodation have a higher risk of victimisation than those in owner-occupied properties. For the year ending March 2020, 22% of those living in private rented accommodation experienced any CSEW crime (including fraud and computer misuse) in the last 12 months, compared with 18.4% of people living in owner-occupied properties (Annual Trend and Demographic Tables year ending March 2020). On balance, the differences in tenure composition observed between the TCSEW and CSEW are small, as are the differences in victimisation by tenure measured by the CSEW for the year ending March 2020. Whilst differences are statistically significant the likely impact on estimates is believed to be negligible.

Whilst differences exist between the samples of the CSEW and TCSEW they appear to have been largely accounted for in the weight systems applied to the respective surveys. This includes the additional weights applied to the TCSEW to compensate for the additional non-response bias between the two survey samples.

As part of establishing the Telephone-operated Crime Survey for England and Wales (TCSEW), it was necessary to consider the survey instrument and its content. Three important factors influenced this part of the redesign work:

• the need to redesign the survey instrument quickly and without piloting
• the length of telephone survey interviews compared with face-to-face
• ethical considerations

The starting point for the TCSEW survey instrument was always going to be the existing survey instrument translated for telephone survey operation. To have redesigned the survey instrument from scratch would have taken time and resources that were not available. This was reinforced by the inability to run any form of pilot.

The overall length of the survey needed to be shortened for telephone operation from an average of around 50 minutes down to 25 minutes. As a result, it was felt that the survey should only carry questions required to provide key estimates of crime (victimisation and prevalence rates of crimes recorded by the survey during the coronavirus (COVID-19) pandemic and the previous 12 months), or those questions which would provide essential information during the pandemic.

Some questions considered sufficiently important to measure during the pandemic were omitted from the final questionnaire following ethical consideration. As a result, estimates are not available in relation to sexual assault, partner abuse or abuse during childhood. This includes the preferred measures of domestic abuse and domestic violence.

Primarily, the Crime Survey for England and Wales (CSEW) provides estimates of the levels of household and personal crimes experienced by respondents. There are two stages to the questionnaire for measuring experiences of victimisation. First, respondents are asked a series of screener questions on the main part of the questionnaire to assess if they have been a victim of crime. Screener questions do not ask respondents if they have been a victim of explicit crime types, but ask about different experiences, such as whether the respondent has had anything stolen in the last 12 months. This design ensures that all incidents of crime within the scope of the CSEW, including relatively minor ones, are included. Following the screener questions, those who have been victimised are asked detailed questions about exactly what happened in a series of victim modules.

Towards the end of the screener section respondents have traditionally been asked “And apart from anything you have already mentioned, in that time, has anyone threatened you in any way that actually frightened you?”. This screener question had been included in the survey since its inception although threats in themselves are not considered within the scope of CSEW crime.

At the onset of the pandemic, there was speculation as to whether the level of harassment within society would increase. As a result, it was decided to change the threat question to read “And apart from anything you have already mentioned, in that time, has anyone threatened, harassed or intimidated you in a way that was intended to cause you alarm or distress?”. This relatively minor change was made with the intention of measuring levels of harassment during the pandemic, something which the original question would not do. Being situated at the end of the screener set of questions and not in scope of CSEW crime, it was considered a relatively safe change, having little or no consequence on the main estimates of crime, or comparability between the CSEW and the TCSEW. With no option to pilot, the new question went live in May 2020 with the TCSEW.

The consequences of changing the wording of the TCSEW threats screener question had a far greater effect than was envisaged. It increased the number of offences that were captured across a range of offence types - in particular, estimates of violence without injury. The introduction of the new screener question can clearly be seen to have affected the number of offences. However, it is not possible to distinguish whether this is due to the change in wording or to a genuine change.

To explore the issue further, we removed all offences resulting from the original screener question (in the comparator year) and those from the new screener question in the TCSEW. The consequence being that both the comparator year (year ending March 2019) and the TCSEW survey year ending March 2021 would be more comparable although they would underestimate the true level of crime. Appendix Table A3 therefore presents percentage changes between year ending March 2021 TCSEW estimates and year ending March 2019 CSEW estimates with the use of comparable data based on the following:

• people aged 18 years and over
• non-overlapping data periods; year ending March 2021 compared with year ending March 2019
• the removal of the threat/harassment questions from both the current and comparator years

The potential difference between responses to survey questions asked via different mediums has long been debated. Interviews that involve social interaction with another person, (such as face-to-face and telephone surveys) are believed to be influenced by respondents taking social norms into account when responding. For example, giving more socially desirable responses, or avoiding answering questions of a sensitive nature.

Although the type of interaction between face-to-face and telephone survey operation is different, it is increasingly common research practice to now use mixed mode designs (for example conducting face-to-face interviews at baseline, with postal or telephone interviews at follow-up). It is believed the differences in responses are marginal. The main victimisation survey in the USA, The American National Crime Victimization Survey (NCVS) (PDF, 4.76MB), found no significant mode effects for estimates of number of victims and prevalence rates when comparing their face-to-face and telephone survey interviews. Whilst this finding offers some reassurance that the impact of these mode changes on prevalence estimates is minimal, the researchers were unable to evaluate their impact on estimates of number of incidents or incidence rates because of the relatively small number of reported incidents in the survey.