Crime Survey for England and Wales transformation research on the ethics of online data collection relating to sensitive topics: October 2020

• In early 2020, 25 in-depth interviews were conducted with "victim-survivors" of domestic abuse, sexual victimisation, stalking and abuse in childhood (representing potential Crime Survey for England and Wales (CSEW) respondents) and with representatives of support agencies, charities and advocacy groups working with victim-survivors (consulted for their expertise and broad perspective on the topic).

• The interviews explored the ethics of moving the CSEW data collection online, including assessing the risks to the physical, emotional and psychological well-being of respondents if asked questions on these topics.

• A focus group was held with CSEW field interviewers about their experiences of administering questions on these topics in the traditional face-to-face survey (which includes respondent self-completion for many of the questions of interest), for context.

• A mixed picture emerged of both positive and negative impacts on respondent well-being and data quality: some victim-survivors would respond to the best of their ability, appreciating the importance of this data being collected and consequently helping others like them, while others would be unable to respond online or would find it difficult to answer questions because of impacts on their emotional, psychological or physical well-being.

• While many victim-survivors will be able to weigh up risks and benefits of responding, and survey design features could mitigate some of the risks to respondents and impacts on data, we cannot estimate the probability or extent to which such risks and impacts might occur in the population of interest and must consider the possibility of serious harm being caused to an online respondent should a perpetrator of one of the crimes of interest become aware of their participation.

• A research programme is being planned to look at the future of CSEW, including the post-coronavirus (COVID-19) context and mixed-mode data collection, taking into consideration the findings from this research.

The research was part of the Office for National Statistics' (ONS') Census and Data Collection Transformation Programme (CDCTP) to improve and modernise data collection, processing and outputs. The findings were to inform decisions by the ONS Centre for Crime and Justice and Social Statistics Transformation as to whether it is ethical to ask the questions of interest in an online-first, mixed-mode¹ Crime Survey for England and Wales (CSEW) and if and how to proceed with work on transforming the survey. The research questions were:

• what ethical issues exist in relation to asking the most sensitive questions - the topic-specific modules on experience of domestic abuse, sexual victimisation, stalking (all since age 16) and abuse in childhood (before age 16), and the questions in the screener and victimisation module about experience of crimes in the last 12 months that encompass such offences - in online, self-complete mode?

• are the ethical issues common to all these crimes or do they differ between them?

• are the ethical issues similar for individuals who experience repeat and/or multiple types of victimisation as for those who experience a single incident, or do they differ?

• how does the ethical dynamic change with the mode change from the traditional design, of face-to-face interviewer administration including computer-assisted self-interviewing (CASI) victimisation modules, to an online-first, mixed-mode design?²

• how might the issues impact on response and data quality?

• what can be done to mitigate?

Notes for: Overview of the CSEW transformation research project

1. Government social surveys that move online will need to offer other modes of collection to non-responders to maximise response rates and reduce coverage, selection and non-response biases.

2. The research was conducted prior to the CSEW adoption of telephone interviewing, necessitated by the coronavirus (COVID-19) pandemic. All references and comparisons to the "traditional" survey mean the pre-COVID-19 situation. See Section 17: Future developments for more information.

Participant reactions to an online CSEW

A mixed picture emerged from the findings, with there being potentially both positive and negative impacts of conducting the Crime Survey for England and Wales (CSEW) online in a mixed-mode design on respondent well-being and data quality. An important theme was that it was seen to be vitally important to collect the information but also to protect respondents: there are some difficulties in achieving both together and uncertainties about the optimal balance between them.

We identified positive findings and benefits. Some victim-survivors would respond online, appreciating the importance and value of reporting their experiences. They would do their best to respond fully and accurately, despite any difficult emotional reactions to the questions that might be experienced. There is potential for higher response to the survey if victim-survivors can complete safely, in private and in their own time, and for enhanced data quality because they feel more anonymous and less likely to provide socially desirable answers than in face-to-face interviews, even when using computer-assisted self-interviewing (CASI).

We also identified negative findings and risks. Some victim-survivors would not feel able to respond online. We identified concerns about respondents being upset or triggered into trauma states by the question content. Further concerns arose around perpetrators, in particular of domestic abuse, preventing victim-survivors from being able to respond and, if they did respond, about consequences of their participation on their well-being, including personal safety, if the perpetrator was aware of it. These concerns might result in item and unit non-response or inaccurate responses.

The topics that caused the most adverse emotional or psychological reactions were sexual victimisation and abuse in childhood, while the topic that raised most concern about ability to take part in privacy and safety was domestic abuse. It should be noted that most victim-survivor participants had experienced more than one of the four types of offence explored.  

Furthermore, the findings were not consistent across all the sample; there were variations in opinions and potential reactions and responding behaviour between participants, which makes drawing firm conclusions and making design decisions harder.

Comparison between traditional and transformed designs

We have identified some potential impacts on data quality, both positive and negative, but cannot say to what extent they would be borne out in actual survey data, nor how they would compare with the traditional CSEW. Some of our findings are specific to online or mixed-mode, but others could equally, or similarly, apply to face-to-face interviewing and CASI.

It is difficult to reconcile these findings into an overall conclusion: online would seem to be a good option for some victim-survivors but less so for others. The relative sizes of the two groups cannot be determined from this qualitative research. Online participation would be more difficult for those who are still experiencing or have recently experienced the offences of interest, in terms of both their mental readiness to take part and cope with the question content as well as their ability to take part safely without risk of abuse as a consequence. It would be easier for those who do not live with a perpetrator of the offences, who experienced them some time ago, or who have received support or dealt to at least some degree with their experiences. However, even some of the latter group might react adversely.

It is difficult to compare between the potential design of an online or mixed-mode CSEW and the traditional survey because there are things we may not fully understand about the latter. We do not know very much about issues around emotional or psychological well-being or physical safety because interviewers might not pick up on them all and respondents do not inform them or the survey organisation. This research identifies issues that potentially apply to the traditional survey too. However, it is important to know that in the history of the survey, there have been no reports from police, other services or individuals of it being implicated in any abuse of a respondent.

Data from the traditional CSEW can be analysed in terms of some aspects of data quality - such as overall response rates and item non-response - but it is more difficult to assess how representative it is of victim-survivors: how many of them respond at all and to what degree interviewer involvement impacts on this. If they do respond, we cannot know to what extent satisficing, social desirability and other measurement effects are in evidence that might be related to collection mode.

These knowledge gaps make it even harder to determine whether online or mixed-mode collection would be "better" or "worse" than the traditional design with respect to respondent well-being, representation or measurement quality.

Discussion of benefits, risks and mitigation

Decisions as to whether and how to continue with online transformation of the CSEW should account for both the risks to the well-being of respondents and the impact on data quality (including unit and item non-response and various forms of measurement error).

Considering the ethical treatment of survey respondents is of primary importance, but the latter cannot be ignored and does pertain to ethics in the sense of the benefits of optimal quality data to victim-survivors and society as a whole.

One of the important findings was the view that, provided the survey does as much as it can to safeguard respondents' well-being, victim-survivors have the capacity to weigh up risks and benefits and should take responsibility to decide whether to respond online.

Giving sampled households enough information to be able to make their own informed decisions and have control over whether and how to respond is central. That includes information in the postal invitation to take part, which in traditional random-probability government surveys is addressed to a household not an individual, so people understand the general nature of the survey but not at a level of detail that either puts victim-survivors off immediately or raises issues within a household around safe and private participation.

More detailed information and warnings can be provided within the questionnaire itself, at relevant points, along with other mitigations such as grounding techniques, support service information, and provisions to maintain privacy and safety. Whether these are an adequate substitute for the interviewer's role, or perhaps even more effective, is unknown. It is possible that such mitigation might prompt increased break-off or item non-response, but importantly it gives control to the respondent.

The issues that might put off some victim-survivors from responding or that could result in harm can be mitigated to a degree, but in likelihood there will always be some who cannot or will not take part. Some of those who would not respond online might be persuaded by an interviewer following-up non-response or take up the offer of interview as an alternative, while some might not want or be able to take part under any circumstances.

Crucially, if a victim-survivor does respond online, there is a possibility that, despite all efforts to mitigate, a perpetrator may become aware of their participation and react in a way that puts the respondent's physical or psychological safety at risk. We cannot say what the likelihood of this eventuality is, only that it is a potential outcome and one that could have very serious consequences for an individual.

Conclusion

On the basis of this research, there are no clear arguments either for or against proceeding with further work on transforming the CSEW for online-first, mixed-mode design. Some questions were not fully answered by the research or arise from it, as described earlier.

Probably the most significant of these questions is the potential risk to a respondent's physical safety, should a perpetrator react negatively to their participation. The decision regarding proceeding with transformation may come down to whether the Office for National Statistics (ONS) accepts the risk, however small or large, of a respondent being physically harmed. Fear of physical assault was expressed among participants as a reason for likely non-response. By logical extension, an unwitting victim-survivor who responds could be subject to an extreme reaction by a perpetrator, that is, be seriously assaulted or even killed, although this eventuality was not spoken of by any participant.

It is worth noting that the traditional CSEW is a general population, random-probability survey. It is not possible to invite named individuals to participate; invitation letters must be addressed to "the resident(s)" at an address and a procedure followed to select one household member (aged 16 years or over) to respond, at random. It includes people for whom the topics on which this research focused may not be as salient as they are to victim-survivors. It covers many additional topics. The survey design needs to be appropriate to as many people as possible. Such features perhaps limit options for addressing and mitigating findings that would be appropriate to a survey specifically about these topics or where the study population was victim-survivors.

Potential further research

As part of the deliberations of how to proceed with CSEW transformation, it would be valuable to undertake further qualitative research, such as cognitive testing and user research around questions and respondent materials as well as quantitative testing.¹

Experiments with online design variants could help us understand some of the potential issues raised and the effectiveness of mitigations. It would also be useful to conduct quantitative comparisons with the traditional survey design (and with potential changes that could be made to it along similar lines to online design, informed by this research), such as to assess response take-up, prevalence of the offences of interest (overall and by different subgroups), item non-response and so on. Any further work should bear in mind the limitations and caveats to this research mentioned in Section 16: Data sources and quality. However, the risk of physical harm befalling a respondent, discussed earlier, would apply to quantitative testing.

More detailed description and discussion of the findings follows in the next sections.

Notes for: Conducting the CSEW online in a mixed-mode design: summary and discussion of findings

1. This leaves aside aspects of survey transformation other than the ethics of asking about these topics to victim-survivors. For more information on our deliberations of how to proceed with CSEW transformation, see Section 17: Future developments.

The research identified potential emotional and psychological effects of the questions of interest on online survey respondents, including:

• reliving experiences of abuse, sexual victimisation and stalking and their aftermath

• triggering of post-traumatic stress disorder (PTSD) about past experiences

• prompting self-realisation for the first time that experiences constituted abuse or assault - including experiences that are current, recent or from many years ago

• evoking shame, guilt or anger; such feelings may be mixed and erratic

There is potential for some respondents to have serious reactions, particularly for those who have not yet dealt with experiences or received support. Some respondents may self-harm, feel suicidal, or abuse alcohol or other substances. Impacts could be experienced while answering the survey and post-survey (for example, losing sleep).

As a consequence, there is potential for undercounting of offences and details of experiences to be not fully accurate because of:

• unit non-response (or partial response): survey break-off as a respondent becomes aware of content and experiences negative feelings

• item non-response: skipping of questions or topics

• measurement error: inaccurate or incomplete responses, including through satisficing and social desirability because of recall difficulty; difficulty in facing up to question content and reliving experiences; lack of awareness, understanding or acceptance of the nature of experiences; and denial of, excusing and rationalising perpetrator behaviour

• selection bias (victim-survivors would be under-represented in the survey)

• differential biases: a lower likelihood of responding or giving accurate answers if a respondent is currently experiencing or recently experienced the crimes or is still living with a perpetrator (although they also can be affected for many years afterwards)

Comparison of online or mixed-mode with traditional design

Questionnaire content was shown to respondents in the context of an online survey, but distress and trauma were generally discussed in reference to the content and wording of questions, so this issue is probably not unique to the online mode.

In an online setting, victim-survivors have personal autonomy and responsibility to decide whether to respond to the survey invitation and to individual questions, provided reasonable efforts to mitigate impacts are put in place by the survey.

Online respondents have greater control than in face-to-face or computer-assisted self-interviewing (CASI), having the flexibility to respond at a time of their choosing, to take as long as they want, in separate sessions if desired, to complete it where they want, and to prepare mentally (once informed of the content).¹

Online, there is potential for expression of distress to be less inhibited than in the presence of an interviewer. The focus group with interviewers suggested they do not have detailed knowledge about respondent reactions. They sometimes are aware of an emotional response to questions, and incidents of respondents not wishing to answer and of satisficing have occurred, but it is not clear to what extent and exactly which questions cause this.

Online, there is a lack of human interaction, removing the potential for noticing when a respondent is in distress, responding to it and providing immediate help. Some participants felt the benefits of online mode outweighed this (even compared with CASI): feeling more anonymous, less sense of social desirability or obligation to please an interviewer, and less likely to feel shame or embarrassment.²

Interviewers can provide some well-being support when respondent distress is apparent (observed or if help requested). This is to a limited degree, such as listening, empathising, offering a break, providing support service information and following standard survey safeguarding protocol, as they are not formally trained in relation to the specific topics. Rather, there is reliance on their personal qualities and experience.

Some burden is also felt by interviewers, reported in the focus group, about what respondents tell them and their concerns when they leave some respondents.

Online, break-off might be easier because of less obligation felt to please the researchers or awkwardness in stopping than when an interviewer is present. No reports were made in the interviewer focus group of known respondent break-off resulting from content; it is possible that item non-response or satisficing happens instead.

Interviewers might reduce some of the various impacts on data quality: they can repeat explanations and assurances and respond to respondent queries more flexibly.

The online mode might sometimes obtain more accurate responses to certain questions in the screener and victim module than face-to-face mode, as indicated in the interviewer focus group.

Mitigation and design considerations for online or mixed-mode

Such reactions and resultant behaviours are probably unavoidable entirely but can be mitigated to some extent in online or mixed-mode, following the ideas in this subsection. The traditional survey already provides much of the mitigation proposed but not all. It could be reviewed in the light of these findings and amended as appropriate.

Content and purpose

Provide information about the content and purpose of modules and individual questions, within the questionnaire.

While over-reliance on information separate from the survey questions is generally to be avoided (since respondents are unlikely to read it, particularly if required to click buttons or links), in this context such information might be important in reassuring, helping or motivating some respondents to respond to the best of their ability.

However, a list of topics or detail of questions should not be provided in the invitation letter or other advance material as that could be off-putting to victim-survivors (see also Section 8: Ability to respond online: issues relating to control regarding not provoking a perpetrator).

Therefore, some information about content should be provided early in the questionnaire to allow victim-survivors to make informed decisions about whether to respond and how to manage the completion process.

Trigger warnings for questions

Provide trigger warnings that some content could be upsetting or (re)traumatising at the points where the questions are asked, including the screener questions, victim module and each of the topic-specific modules.

Acknowledge the potential difficulty but also the benefits.

Provide enough information, throughout, to enable victim-survivors to give informed consent and have control.

Advise respondents to respond only if they are strong enough mentally, to pick a good time and environment, and to respond in private.

Skipping of questions and modules

Provide clear explanations that questions can be skipped (the "don't wish to answer" option) or answered with "don't know/can't remember".

Also, allow skipping of entire modules. This will mean respondents who are distressed, or consider they might be triggered, do not need to see the content of every question. Such options give respondents a sense of control, potentially reducing dropout.

The traditional survey does not allow skipping of modules, just of the entire CASI section or of individual questions: if skipping of modules were allowed, online item non-response might increase.

Preview upcoming questions

Provide flexibility to preview upcoming questions, that is, to page forward and back through the questionnaire without being encumbered by, for example, error messages and presentation of previously hidden "don't know" and other non-response options.

This design might increase item non-response or response errors.

Save and return later function

Provide a "save and return later" function to enable respondents to take a break or quickly escape from distressing questions yet be able to resume without losing answers so far.

This is a standard online survey feature but, in this context, could result in increased break-off, for example, if a respondent intended to return but did not for some reason.

Assurances of confidentiality

Provide assurances of confidentiality and anonymity, including an explanation of safeguarding.

Support and help

Provide information about support services. This could include:

• services at national and regional level

• directories of local services

• those that provide support all day every day

• general services and those for specific groups - such as men, women, lesbian, gay, bisexual and transgendered (LGBT) people, "honour-based" abuse and specific to certain offences (for example, domestic abuse, rape and other sexual assault, stalking, and survivors of abuse in childhood)

Provide information about grounding techniques and tools to mitigate distress or trauma, for example, links to exercises and videos. This could also be provided within the traditional face-to-face with CASI survey.

Placement of information

Consider the best placement of such information. Options include in introductions to the survey and of each topic module, links on each page, at the end of the survey, and on the survey website for access once a questionnaire has been submitted. These are not mutually exclusive options, but consider whether to provide all the information at each of these places or make specific to the point in the questionnaire and for each topic.

Further consideration and potential research

Should work on Crime Survey for England and Wales (CSEW) transformation with an online element be conducted, this article provides some detail of suggestions for design and mitigation of issues. However, not all design requirements have been fully thought through, and the design will require careful consideration, consultation and research.

Subject to timing, resources and feasibility, further qualitative research would be valuable to inform and test proposed designs, using in-depth, cognitive testing and user research techniques. This could include strategies and materials for respondent communication, questionnaire content and the mitigations built into the design.

It would also be informative to conduct quantitative testing of different design options, for example, split-sample experiments to measure their impacts on overall response take-up, item non-response, module skips, survey break-off and other aspects of data quality, and to compare data with the traditional CSEW design.

Consideration could be given to including qualitative questions within a quantitative test, such as to ask for respondents' views on their experiences, response behaviours (including non-response) and aspects of the design.

It would be helpful to increase understanding of the traditional face-to-face with CASI survey such as by conducting more research with interviewers (such as focus groups or a survey), informed by this research.

The method of allowing respondents to skip individual questions and entire modules is a general design consideration for any online questionnaire, but the sensitive modules on CSEW might need to differ from any general design approach. The question is how best to balance between making the ability to skip obvious (so respondents who are likely to be affected by content are not forced to answer, or even read, questions, to reduce the likelihood of distress or trauma) and making it too easy to skip for all respondents, including non-victim-survivors (so inviting satisficing and increased item non-response). For example, the commonly-used "hidden" method, where options such as "don't know" and "don't wish to answer" are presented only when a respondent presses "next" without answering a question, might be less appropriate as it forces the respondent to see the question again or possibly invites an inaccurate response initially.

This should be considered in conjunction with general information and messages to explain, reassure and motivate respondents at the start of the survey and/or each sensitive module.

Notes for: Questionnaire content: emotional and psychological impacts

1. This is aside from considerations about control over survey response and the survey environment – see Sections 8 to 10.

2. The possibility of this being true was also raised by the interviewer focus group, but with little discussion.

There were participants who saw benefits to responding to an online Crime Survey for England and Wales (CSEW). The benefits included feeling personal validation, appreciation of the opportunity to share their experiences and to gain deeper understanding of the self, being prompted to report experiences and seek help, and feeling they can help raise awareness of issues in society, inform government policy and thus improve lives. For them, such benefits outweighed any emotional discomfort felt, of the kind reported in earlier sections.

Such respondents would continue with the survey (assuming they responded to it initially, without detailed knowledge of its content).

They would be likely to provide answers to the best of their ability, subject to difficulties over content and with recall of details or repression of memory - some answers would be missing, estimates or otherwise inaccurate.

Comparison of online or mixed-mode with traditional design

These reactions are likely to be similar in either mode.

Mitigation and design considerations for online or mixed-mode

Mitigation and design considerations for online or mixed-mode is less of an issue, but some of the positive views depend on mitigations, as discussed earlier, being in place.

Further consideration and potential research

As covered in Section 4: Questionnaire content: emotional and psychological impacts.

The general approach to the Crime Survey for England and Wales (CSEW) question design - the content of the questions, the types of question (mostly closed-ended, factual or behavioural) and the amount of detail - was acceptable, on the whole, to participants. However, some questions were described as, for example, "victim blaming", some questions were considered too graphic, and concerns were expressed over confidentiality.

These reactions have potential to cause break-off and incomplete or inaccurate answers if respondents feel upset, judged, offended or have confidentiality concerns.

Comparison of online or mixed-mode with traditional design

It is likely that this is not unique to online as it appeared to be the content of the questions that triggered the reaction. Further research would be needed to ascertain this, as these findings were in the context of asking about the survey in an online mode, but some interviewer comments indicated similar reactions.

Interviewers also had some comments about question comprehension, such as needing to help respondents understand differences between similar questions.

Online, respondents may find it easier to skip such questions or to break-off when no interviewer is present to deter such behaviour.

Online, definitions and explanations can be provided that might help understanding of question wording and purpose, though respondents may not refer to them.

Interviewers can help respondents understand questions and respond to their comments (though there was little evidence from the focus group to indicate what happens in this regard).

Mitigation and design considerations for online or mixed-mode

Changes to questioning can be considered, informed by comments from victim-survivors and agency participants. However, the interviews were not a cognitive test of proposed online questions. Details of data user requirements, the intentions of questions, and general review or redesign of questions for a transformed survey would need to be considered, and suggestions or proposed changes would benefit from question testing.

Further consideration and potential research

Some of the feedback on wording and suggested additions to questioning are potentially useful regardless of mode.

Despite objections, there are perhaps no better ways to ask some of the questions to obtain the required information; therefore, the surrounding explanations of meaning and purpose are important.

The CSEW is currently undergoing a full review of the module of questions on domestic abuse and related information (independent of the wider programme of transformation). This review will take into account the findings in this article and the particular requirements of online or mixed-mode surveys.

There were some views that the survey content could be considered, cumulatively, to be too intense or burdensome by some respondents, especially for those who have experienced multiple crime types and/or are repeat victims.¹

Respondents might find it hard to make time, particularly if currently experiencing the offences of interest. Victim-survivors might want to respond on the go, on mobile phones.

Some thought the order of topic modules and questions within them were suitable. However, we identified potential order effects related to asking about each topic in its entirety (what has been experienced and detailed follow-up questions together) and the topics in turn, rather than splitting between what has been experienced, across all topics, then all the detailed questions, across all topics.

These issues could lead to satisficing, response errors and break-off.

Comparison of online or mixed-mode with traditional design

Length of questionnaires is a general issue for social surveys. The optimum length for the online mode would need to be considered. The full questionnaire content of the Crime Survey for England and Wales (CSEW), as it is, is likely to be unsuitable for online completion, particularly on mobile devices or smaller screens (for example, some questions are long and would require much vertical scrolling).

Survey length and the burden imposed may also be an issue for the traditional survey.

Mitigation and design considerations for online or mixed-mode

The following design considerations may help to mitigate the potential issues related to survey length and burden:

• Design the questionnaire in accordance with Office for National Statistics (ONS) transformation principles, such as respondent-centred design principles and optimisation for mobile devices (PDF, 1.88MB).

• Provide a "save and return later" function to allow people to take the survey in separate sessions.

• If all content is retained, the survey should probably keep the traditional order of the modules as respondents considered it appropriate; however, there are also potential order effects: if questions on one topic cause a reaction, potentially a respondent might not respond to later topics, so consider the potential for asking the questions about what has been experienced across all topics before asking for full details.

Further consideration and potential research

Consideration should be given to understanding what an acceptable length for an online survey is in general and with regard to the particular emotional demand of the CSEW.

Data requirements should be reviewed before transforming the questionnaire.

Ways to reduce length by cutting content, such as dropping questions, modularising and short- or long-form versions could be considered.

Further development and testing of the impact of design might be required.

Government Digital Service (GDS) design principles and accessibility requirements should be considered (relating to first language, literacy, and IT skills or capability).

Notes for: Questionnaire content: survey length, burden and repetition

1. See also comments in earlier sections on question content or detail being thought appropriate.

Participants thought that when a victim-survivor lives with a perpetrator of the offences of interest, the following issues relating to control over responding to the survey invitation may arise:

• Perpetrators are known to control or intercept incoming post, so a victim-survivor may be unlikely to read the advance letter without the perpetrator being aware or have opportunity to respond without interference.

• A perpetrator could abuse a victim-survivor merely because of an invitation being received, even though letters are un-personalised, addressed to "the resident".

• The within-household subsampling method, to select one adult to respond following a set procedure, may not be followed correctly.

• A perpetrator may not allow a victim-survivor to respond, if selected.

• A perpetrator may pretend to be the victim-survivor and respond on their behalf.

• A perpetrator may influence a victim-survivor's responses (directly, if they know about the survey, or indirectly, if the victim-survivor fears potential consequences of answering accurately should the perpetrator become aware of the survey).

These scenarios would result in missing or inaccurate data about victim-survivors. There would be unit non-response, as the survey may not reach victim-survivors who are still living with a perpetrator or obtain unimpeded response from them, and there would be item non-response and/or inaccurate responses because of perpetrators responding on behalf of or influencing the victim-survivor.

Comparison of online or mixed-mode with traditional design

Control of post and potential consequences of the invitation would apply in any mode where letters are un-personalised (see Section 12: Mixed-mode survey options: following up initial online log-in or response regarding personalised contact). Potentially, this has a more substantial impact online, without the reference in the letter to an interviewer calling to act as deterrent to a perpetrator (though, conversely, that could be a greater provocation for some).

Interviewers can mitigate the impact of unread letters and have control over the process of selection within the household.

No evidence arose from the interviewer focus group of issues around the selection procedure (other than it reduces the likelihood of the advance letter having been read by the selected person).

This research suggests that it could be difficult for a victim-survivor to take part in an online survey while living with a perpetrator and if they did, they could face risk to their personal safety (see Section 10: Ability to respond online: risk to respondent's physical or psychological well-being posed by a perpetrator for more information).

Little is known from the interviewer focus group about whether and how non-response (refusals and non-contacts) is related to households involving an abuse situation, whether visits by interviewers enable victim-survivors to respond or affect perpetrator behaviours before, during or after interviews with victim-survivors.

Interviewers can control the survey environment to maintain respondent privacy, although some evidence arose from the interviewer focus group that this is not always the case and can impact on responses.

However, as noted in Section 3: Conducting the Crime Survey for England and Wales (CSEW) online in a mixed-mode design: summary and discussion of findings, there is no evidence from the CSEW's history that it has been implicated in abuse: there has never been contact by the police, social services, support services, or any other organisation or individual in relation to this.

Mitigation and design considerations for online or mixed-mode

It appears difficult to mitigate control over post, within-household selection and the self-completion environment. An impact on online response from victim-survivors is likely, as is potential for inaccurate data from those who do respond. Some mitigation might be possible, as follows:

• Emphasise in advance material that households are selected at random.

• Reduce risk by not mentioning the topics of domestic abuse, sexual victimisation and stalking in advance materials, which could provoke perpetrator behaviour; these topics are not mentioned in the traditional survey advance material, and interviewers reinforced this as being appropriate.

• Information about the topics and advice on completing in private can be given within the questionnaire itself rather than in the advance letter to avoid alerting perpetrators to the content of the survey; however, the corollary of this is that victim-survivors will not be aware that questions on abuse, sexual victimisation and stalking will be asked, information that could influence their decisions such as to whether to respond to the survey and how to manage the completion process and their own well-being (this information could be provided early in the questionnaire, as discussed in Section 4: Questionnaire content: emotional and psychological impacts).

• Following-up online non-response by an interviewer might mitigate some of these issues, though interviewer administration is not without its own issues - see Section 11: Mixed-mode survey options: following up online non-response by interviewer.

Further consideration and potential research

It would be helpful to increase understanding of the traditional face-to-face with computer-assisted self-interviewing (CASI) survey such as by conducting more research with interviewers (for example, focus groups or a survey), informed by these findings.

Even if a perpetrator is unaware of the survey invitation, other difficulties in responding faced by victim-survivors include lack of time or opportunity to be away from a perpetrator and their focus on the abuse situation to the exclusion of other things.

This would result in non-response and thus under-representation of victim survivors.

Comparison of online or mixed-mode with traditional design

Some victim-survivors would simply ignore the online survey invitation or refuse to respond. This is easier to do online than to an interviewer.

However, some victim-survivors would do what they could to try to respond, finding a way round their perpetrator. The survey would perhaps be easier to keep secret online, assuming they have got to the letter first and are the selected person.

There is potential for an interviewer to persuade and manage the situation and gain higher survey (unit) response, including among vulnerable groups; however, see Section 11: Mixed-mode survey options: following up online non-response by interviewer for consideration of interviewer administration. As was noted within the focus group with interviewers, human interaction can help to convince people to take part who perhaps otherwise would not. But we do not know about the nature or extent of refusals and non-contacts being related to, for example, domestic abuse occurring within a household.

Mitigation and design considerations for online or mixed-mode

Choice lies with the victim-survivor - they will often know if taking part online will put them at additional risk and have the autonomy to choose to take part or not. If there are consequences to them taking part in the survey, the responsibility lies with the perpetrator.

Consideration could be given to providing respondents an option at the outset for which mode they would prefer, for example, an opportunity to request an interviewer-led survey or to request an alternative location to the home (such as support service premises, though this could be logistically difficult).

Such an option could be provided within the advance material - but without referring to questionnaire content as the reason, as discussed earlier. However, this would potentially reduce the proportion of online response to the survey overall - people could request an interview for reasons not related to being victim-survivors of the offences of interest - reducing some of the intended benefits of the transformation programme in general.

Alternatively, the options and reason for offering could be included at the start of the questionnaire, along with content and trigger warnings.

Further consideration and potential research

It would be helpful to increase understanding of the traditional face-to-face with computer-assisted self-interviewing (CASI) survey such as by conducting more research with interviewers (for example, focus groups or a survey), informed by this research.

If a victim-survivor did manage to respond online, their physical or psychological well-being (and potentially that of others such as dependent children) could be at risk if a perpetrator became aware of their response to the survey, while self-completion was in progress or subsequently. Some concerns were expressed over data confidentiality and privacy. Perpetrators are known to monitor or control victim-survivors' use of IT equipment or devices.

Risks to well-being include physical and non-physical abuse.

Victim-survivors living with a perpetrator may not answer questions or may give inaccurate answers because of fear of repercussions - resulting in missing data or undercounting of offences.

Comparison of online or mixed-mode with traditional design

Victim-survivors are experts in their own safety and usually know whether it is safe to do something. For some, online response could be the safest option. But there are cases where they are being monitored without their knowledge, increasing the risk of discovery.

There is a risk that a perpetrator could find out regardless of the mode a respondent took part in. However, the consequences of discovery could be more serious online, without interviewer presence to manage the environment or deter perpetrator reaction.

The focus group with interviewers revealed that some respondents show concern over how confidential their answers are.

In face-to-face interviewing, questions in the screener and victimisation module might be overheard by a perpetrator (or the silent showcard questions observed). Computer-assisted self-interviewing (CASI) can be kept private to a large extent but probably not completely, given examples mentioned in the interviewer focus group.

However, there is no evidence from the history of the Crime Survey for England and Wales (CSEW) that it has been implicated in abuse: there has never been contact by the police, social services, support services, or any other organisation or individual in relation to this.

Mitigation and design considerations for online or mixed-mode

Choice lies with the respondent - they will often know if taking part will put them at additional risk and have the autonomy to choose to take part or not. If there are harmful consequences to them taking part in the survey, the responsibility for these lies with the perpetrator. Design features to mitigate effects include:

• clear statements of confidentiality and anonymity at the beginning of the questionnaire and at regular intervals could help to reassure respondents

• emphasising, in any CASI elements of the design, that answers cannot be seen by interviewer on return of the device

• including a "don't wish to answer" option on many questions, allowing respondents to skip over those they do not feel able to answer for safety reasons; however, it is possibly more likely a victim-survivor would give a safer answer instead, for example, to answer "no" about abusive behaviours that actually have been experienced

• allowing entire modules to be skipped (see Section 4: Questionnaire content: emotional and psychological impacts for more information) could enable respondents to avoid issues arising

• providing a quick escape button, on each page of the survey, which redirects to an innocuous website such as Google or the BBC; this could provide reassurance for those fearful they will be found taking part in the survey

• providing advice on how to cover online tracks or delete browser history within the survey questionnaire (but not in advance material)

• warning respondents, in the questionnaire, of risks and advise them not to take part if they are vulnerable to them

However, these mitigations might not always prevent discovery of a victim-survivor's participation by a perpetrator while they are responding or subsequently.

For some victim-survivors, no mitigation will be enough to reassure them that they would be safe in responding.

Further consideration and potential research

As some of these issues do not appear unique to the online mode, it would be helpful to use these insights to increase understanding of the traditional face-to-face with CASI survey by conducting more research with interviewers (for example, focus groups or a survey).

Various potential mixed-mode survey design options were explored with participants, described in this section and Sections 12 to 14.

The first option discussed was following up non-response to the online invitation by an interviewer. There was felt to be risk to victim-survivor safety if an interviewer calls at a household unexpectedly, which could be the effect if any invitation or reminder letter that mention a potential visit has not been read. A perpetrator could become suspicious of the victim-survivor, accusing them of arranging the visit, with risk of retaliation from the perpetrator.

This would increase the probability of non-response in such households.

Comparison of online or mixed-mode with traditional design

These concerns would apply to any design that involves an interviewer arriving at a household when not expected (even if a letter or reminder mentioning a potential visit is posted).

As mentioned previously, the experience of the traditional Crime Survey for England and Wales (CSEW) is of there being no known incidents implicating the survey in abuse.

Little is known about the extent of non-response because of such circumstances (as noted in Section 8: Ability to respond online: issues relating to control).

Mitigation and design considerations for online or mixed-mode

Notifying households in writing that an interviewer will be visiting because of no online response having been received, reiterating the random sample design and providing an option to decline a visit in advance could all help to mitigate the issues identified.

Further consideration and potential research

It would be helpful to increase understanding of the traditional face-to-face with computer-assisted self-interviewing (CASI) survey by conducting more research with interviewers (for example, focus groups or a survey), informed by the outcomes of this work.

There are also potential mixed-mode options whereby an initial online log-in could be followed up by interview, including:

• the respondent being given the opportunity to request an interview instead of online completion (as discussed earlier)

• data collection for an individual is split between modes: high-level questions to establish experience of abuse, sexual victimisation and stalking asked online, with victim-survivors asked full detail in later interview

• an integrated multi-wave survey model, which is a potential design for Office for National Statistics (ONS) social surveys collectively, involving an initial online registration stage collecting non-crime information and crime questions being asked at a follow-up wave (online or by interview)

Concerns were expressed around communications to arrange an interview or invite participation in the follow-up wave being intercepted or monitored by an abuser. Risks and fear of consequences are as for un-personalised communications, as discussed earlier.

Potential benefits arise in addressing communications to a named person (via them providing contact details) directly. However, personalised communication could be additionally risky if intercepted, as a perpetrator could suspect prior contact had occurred. Additional anxiety could arise as a result of knowing the communication would be arriving at some point.

There is potential for non-response to a follow-up interview or Wave 2 survey, even if a victim-survivor agrees to or initiates it.

Comparison of online or mixed-mode with traditional design

The findings and considerations apply to any online-first mixed-mode design options involving an interview or second wave.

Mitigation and design considerations for online or mixed-mode

When offering options or requesting a follow-up, ask for the respondent's preferred, safest method of communication and ensure it is used. This might be email, letter, phone or text message. Care should be exercised in the wording not to rouse suspicion in a perpetrator.

Some victim-survivors potentially will be unaware that their phone or computer is monitored by a perpetrator: consider whether to mention this possibility when asking for their preferred communication method. However, to do so might reduce likelihood of response from some.

A suggestion was made to allow people to opt-in to the Crime Survey for England and Wales (CSEW) (such as by promoting it through support services): this would not be appropriate as it deviates from standard random probability sampling methodology, risking low and biased response. As noted in Section 3: Conducting the CSEW online in a mixed-mode design: summary and discussion of findings, the CSEW is a general population survey and covers many question topics in addition to those explored in this research.

Another suggestion was made to allow respondents to initiate the re-contact, rather than the survey organisation contacting them: this too would be inappropriate as despite good intentions, people might not make the contact for one reason or other.

Offering an interview, or contact to arrange one, by telephone might reduce risk around an interviewer visit. However, it would still be subject to some degree of risk of discovery by a perpetrator and difficulty managing the interview environment. Furthermore, management of respondent well-being relating to the question content would be difficult over telephone.

Further consideration and potential research

Qualitative and/or quantitative research would be advisable, to explore issues and potential designs.

A potential design option would be to split data collection mode for an individual. High-level questions about experience of abuse, sexual victimisation and stalking would be asked of all respondents online, with victim-survivors asked the full details in a later interview. The intended purpose would be to reduce the risks to online respondents' emotional well-being, privacy and safety when living with a perpetrator, through minimising online content and taking advantage of interviewer involvement (even if using computer-assisted self-interviewing (CASI) to administer the survey).

Comparison of online or mixed-mode with traditional design

It would appear to be difficult to break the questions on the topics of interest into two stages of online collection then interview. In the traditional questionnaire, there are numerous questions to establish what respondents have experienced relating to domestic abuse, sexual victimisation, stalking and abuse in childhood.

To ask only these questions online could still act as a trigger of distress and/or raise concerns around privacy and safety, even without asking the full detail.

To ask them online would not leave much to cover in a follow-up interview, raising questions about efficiency.

However, to redesign the questioning to ask fewer, more abstract or high-level questions online to establish experiences (to the effect of, "have you ever experienced domestic abuse?" or similar), leaving a greater number of detailed questions for a follow-up, would probably be unfeasible: we learned that respondents may not understand and relate to concepts such as "domestic abuse" or "sexual assault" in the same way as intended. This could result in some false negative responses.

Mitigation and design considerations for online or mixed-mode

This would appear to be an unfeasible design option, for the reasons just given about questionnaire design and also because of likely practical difficulties around follow-up interviewing as discussed earlier, regarding obtaining respondent agreement and contact details and making arrangements safely. The need for the detailed questions to be administered by CASI might create an odd and awkward situation for interviewers. Cost effectiveness might be questionable.

It would seem that the data would be better collected through a full set of detailed questions, in one mode, unless data requirements are fundamentally changed.

Some victim-survivors think an interviewer can help them in their situation or report it to police or other authorities for them. To maintain confidentiality of the answers respondents provide, interviewers are not permitted to do this, unless a situation fell into a category covered by safeguarding protocols, whereby they may raise it with the office if they believe a respondent or child is at imminent risk of harm or report an emergency to emergency services. Respondent expectations about situations not covered by safeguarding potentially lead to disappointment or distress when it becomes apparent this cannot happen.

Comparison of online or mixed-mode with traditional design

Experience of respondents thinking field interviewers could tell authorities, such as the police, about crimes they had experienced was also reported at the interviewer focus group.

Mitigation and design considerations for online or mixed-mode

Set boundaries in the introductions to relevant topics and/or when providing information about support services.

"Crime" and "offence"

We use the words "crime" and "offence" in relation to behaviours asked about in the survey regardless of whether a victim-survivor has reported an experience to the police and/or would themselves consider it to be a crime or offence or anyone has been charged.

Satisficing

Satisficing is a respondent strategy that shortcuts the response process (comprehension of the question, retrieval of information from memory, formulation and provision of an answer) in some way and results in an adequate answer, meeting the basic requirement of providing one, rather than an optimal one.

Social desirability

Social desirability is the situation when a respondent provides an answer that conforms to social norms or the perceived expectations of others who may see the response, rather than the true value.

Victim-survivor

We refer to anyone who has experienced one or more of the behaviours asked about by the questions in the topics of interest to this research as a "victim-survivor". We intend the term to encompass people at whatever stage in their experience journey they are. We acknowledge that some people may identify as a victim, some as a survivor, and others still do not wish to be labelled in such ways.

The research was conducted in late 2019 and early 2020 by researchers from the Office for National Statistics' (ONS') Methodology and Social Survey divisions.

Ethical considerations in the qualitative research

The principles of informed consent and looking after the emotional and physical well-being of participants and researchers were paramount throughout the research, given the sensitivity of the topic matter and potential vulnerability of the study population. Approval to conduct the qualitative research was given by the National Statistician's Data Ethics Advisory Committee (NSDEC). To inform the NSDEC application¹ and the design of the research, an initial round of consultation with support agencies and experts in the field of research with victim-survivors was conducted in autumn 2019.

Research process

Topic guides were developed to explore the research questions. So that participants could give informed opinions, the potential design of an online survey and the question content of the topics of interest were described to them at summary level. Victim-survivor participants were probed about how they might personally react to the design and content of an online Crime Survey for England and Wales (CSEW) and behave if they were to be sampled for it. Participants from support agencies, charities and advocacy groups were asked to consider how the people they help would react and behave.

Purposive sample and recruitment

We employed a purposive sample design. The primary selection criterion for the sample of victim-survivors was type of offence, to ensure representation of people who had experienced domestic abuse, sexual victimisation, stalking and abuse in childhood. A number of secondary criteria were identified relating to the nature of their experience and socio-demographic characteristics, with the intention to include as much variety in the achieved sample as possible, given practical constraints of time, resources and the ability to find and recruit people.

Support agencies working with victim-survivors of these offences assisted with recruitment, by identifying and contacting individuals they have helped who met the selection criteria and/or distributing the ONS' recruitment material on social media or in blogs and newsletters.

Victim-survivor participants received a monetary token of appreciation. Face-to-face interviews took place in ONS offices, at the premises of support agencies and in hotel conference rooms. Interviews with agency participants were conducted either face-to-face or by telephone.

We interviewed 13 victim-survivors and 12 agency representatives and obtained findings of both breadth and depth. The achieved sample of victim-survivors contained a broad diversity of experiences but was smaller than intended because of complexities in recruiting via third parties and having to curtail fieldwork because of the coronavirus (COVID-19) pandemic.

Caveats

Most of the secondary sampling criteria were represented to at least some degree, but under-represented groups included men; transgender or non-binary people; older people; lesbian, gay and bisexual people; people in black ethnic minority groups; and people from certain religions.

Victim-survivor participants were all self-aware of what they had been through, had all received support at some time following their experiences, and were willing to take part in this research. Therefore, the findings might be somewhat biased away from the views of people currently experiencing the offences of interest or who were not fully aware of what they were experiencing or had not received support.

Focus group with CSEW field interviewers

A focus group with CSEW field interviewers discussed their experiences of administering the survey questions of interest and opinions about ethical issues relating to safeguarding and risks to respondents. This provided some background context for comparison with the findings from the interviews.

Analysis

We transcribed recordings of the interviews and focus group and conducted a thematic content analysis of the data. Note that in qualitative research of this kind, statistical inference is neither the purpose nor possible.

Acknowledgements

For their contributions to the research, we would like to thank:

• the victim-survivors who were interviewed, for sharing their views and being open about highly personal and sensitive experiences

• the representatives of the support agencies and other organisations who advised on the ethics of conducting research with this study population to inform our NSDEC application and the research design, commented on our research materials, helped with recruitment of victim-survivors, and/or took part in research interviews:

• Aurora New Dawn

• Centre for Gender and Violence Research at the University of Bristol (who also provided training to the ONS researchers)

• Galop

• Karma Nirvana

• Llamau

• The ManKind Initiative

• Men's Advice Line

• The National Association for People Abused in Childhood (NAPAC)

• National Society for the Prevention of Cruelty to Children (NSPCC)

• Rape Crisis England and Wales

• SafeLives

• SurvivorsUK

• Victim Support

• Women's Aid

• colleagues at KANTAR global consulting and research business for providing CSEW documentation and organising the interviewer focus group and the interviewers who took part

Participation and provision of assistance does not imply endorsement of our article.

Notes for: Data sources and quality

1. See agenda item four and appendix at NSDEC Minute, agendas and papers - October 2019.

The impact of the coronavirus (COVID-19) pandemic means the Crime Survey for England and Wales (CSEW) is currently administered by telephone, rather than face-to-face interview. The modules on the sensitive topics that were the subject of this research are not currently asked of respondents.

A work programme is being developed to look at the future design of the CSEW, which would address the issues raised in this article.