Redevelopment of domestic abuse statistics: research update November 2023

The way domestic abuse manifests is constantly changing. To ensure our statistics continue to provide the most accurate information and meet the needs of users, we have been undertaking a user engagement, research and testing programme to improve the collection of data on domestic abuse.

New survey questions to measure domestic abuse were added to the Crime Survey for England and Wales in April 2023 as part of a split-sample trial. Half of our respondents will answer the new questions and the other half will answer the old questions. Our previous Developing a new measure of domestic abuse: April 2023 article presented these questions alongside our plans for evaluating their success.

This update provides further information on our evaluation strategy as well as our future plans. It expands upon our evaluation criteria and provides details of the research questions we will use to assess the criteria.

Our evaluation criteria were developed in line with the five dimensions for measuring the quality of statistical outputs as set out in the UK Code of Practice for Statistics and Eurostat's European Statistics Code of Practice. The criteria are detailed in Section 4: Research questions and evaluation criteria to assess domestic abuse questions.

We will assess the criteria using a range of sources including, but not limited to, data collected from the split sample trial, our research work to date and user engagement. The sources will be used to answer research questions linked to each criterion. To date we have developed 29 research questions to assess the 12 evaluation criteria. The research questions are presented in Section 4: Research questions and evaluation criteria to assess domestic abuse questions. We will remain flexible as the evaluation continues and may add further research questions in the future.

This section expands on one of the research questions within each of the five quality dimensions explaining:

• why we are assessing the question

• the methods we will use

• the progress to date in answering the question

Answering each of the research questions will allow us to assess each of the evaluation criteria and ultimately evaluate the success of the new domestic abuse survey questions, and the future of domestic abuse measurement in the Crime Survey for England and Wales (CSEW).

Relevance

Within the relevance quality dimension there are 4 evaluation criteria and 10 associated research questions. More details on the criteria and questions can be found in Section 4: Research questions and evaluation criteria to assess domestic abuse questions. One of these questions is:

“Have the new questions been assessed against user requirements and do the proposed outputs meet priority requirements?”

Answering this question will provide evidence to assess the evaluation criterion:

“The new questions and proposed outputs meet the priority requirements of users”.

This research question will be assessed using both requirements and stakeholder feedback already collected through the redevelopment process, and data we will collect in the future from the split-sample trial. Before beginning the redevelopment, we asked users for their requirements for domestic abuse statistics. These requirements were collated and were central to the development of the new questions. We will assess whether the new statistics produced meet each requirement. However, we recognise that not all user requirements will be met because of the limitations of the CSEW. For this reason, we will measure success both on the proportion of user requirements met as well as user feedback.

In late 2024 we aim to publish our first trial outputs based on the new questions. Following this we will conduct a user survey which, as well as seeking feedback on the trial outputs, will also ask users whether they meet their priority needs. If the majority of users respond positively to the proposed outputs, then this will be deemed a success.

This research question will form an essential part of assessing the evaluation criterion. Providing evidence to demonstrate that the criterion has been met is one of many measures to determine the possible inclusion of the new survey questions permanently.

Work to answer this question has already begun and will continue as we engage further with users.

Accuracy and reliability

Within the accuracy and reliability quality dimension there are three evaluation criteria and eight associated research questions. One of these questions is:

“Are the widths of confidence intervals for headline domestic abuse prevalence estimates from the new questions a similar width to those of the current questions?”

Answering this question will provide evidence to assess the evaluation criterion:

“The new questions accurately measure lived experiences of victims of domestic abuse” within the accuracy and reliability quality dimension.

This research question focuses on comparing confidence intervals, rather than estimates. Comparisons between estimates from the old and new questions will not be made because of the large differences between questions and their structure. Confidence intervals are an indicator of the level of uncertainty around estimates. The levels of uncertainty between the estimates from the old and new questions can be compared and form an important quality measure for assessing the success of the new questions.

To assess this research question, we will use data collected from the split sample trial. For the headline prevalence estimates, confidence intervals will be compared with those for the estimates produced using the current questions. Although we will assess data quality on an ongoing basis, we will not answer this question until we have at least one year of data to allow for a large enough sample.

Although this research question will form an important part of assessing the evaluation criterion, it is not essential. We recognise the importance of data quality, and that assessing the confidence interval widths to understand uncertainty are a part of this. However, should their size differ from those of the current estimates, we will not immediately conclude that the evaluation criterion has not been met. Instead, we will conduct further analysis to ensure we understand any differences and can explain these to users.

Timeliness and punctuality

There is one evaluation criterion within the timeliness and punctuality quality dimension and two associated research questions. One of these questions is:

“Are estimates from the new questions produced within the same time period as those from the existing questions?”

Answering this question will provide evidence to assess the evaluation criterion:

“The new questions do not affect the timeliness and punctuality of our domestic abuse publications”.

Using the current survey questions, we publish headline domestic abuse estimates for the year ending March in our Crime in England and Wales: annual supplementary tables dataset in July. We then publish further breakdowns in our Domestic abuse in England and Wales statistical bulletins in November. With the new questions, we aim to produce estimates within the same time periods. By publishing trial estimates in late 2024, we will be able to assess the feasibility of this for future years, working to improve our processes, ensuring both accuracy and timeliness of data.

This research question is important because ensuring we publish timely estimates is an important user requirement for the new statistics and therefore inclusion of the new questions permanently. However, we accept it may initially take time to produce the estimates using the new data. If we expect the new estimates will take longer to produce than the current estimates in the future, we will seek user feedback to guide us on the trade-off between accuracy and timeliness.

Comparability and coherence

Within the comparability and coherence quality dimension there are two evaluation criteria and six associated research questions. One of these questions is:

“Do the new questions, and estimates produced from them, better align with the Domestic Abuse Act 2021?”

Answering this question will provide evidence to assess the evaluation criterion:

“Estimates produced using the new questions closely align with government definitions of domestic abuse, for example the Serious Crime Act 2015 and Domestic Abuse Act 2021, and any differences are clear to users”.

This research question will be assessed through comparison of the new survey questions and data against the legislation and statutory guidance, as well as using information that has already been collated on this during the redevelopment work. One of the main reasons for redeveloping the domestic abuse questions on the CSEW was because the existing questions did not align to the Domestic Abuse Act 2021. The research to design the new questions aimed to align the design of the questions and the new measures with the Act. You can read more about this in our Redevelopment of domestic abuse statistics: research update November 2022 article. However, a full assessment needs to be conducted to clearly show how the new questions and resulting estimates align to the Act.

Although this research question will form an important part of assessing the evaluation criterion, we acknowledge the questions will not perfectly align to the legislation. This is because the questions are limited by space on a wider crime survey and as such, cannot cover all aspects. Therefore, if we are to permanently include the new questions, the decision will not be based on exact alignment, but close alignment. Providing evidence to demonstrate this has been met will include a clear explanation to users of how and where the measures do and do not align with the Act. Work to answer this question has already begun and will continue over the coming months.

Accessibility and clarity

There are two evaluation criteria within the accessibility and clarity quality dimension and three associated research questions. One of these questions is:

“Do the majority of users think the outputs and data from the new questions are easily available and clear?”

Answering this question will provide evidence to assess the evaluation criterion:

“Estimates produced using the new questions are presented in a format that is easily available and clear to users”.

The survey we conduct following the publication of our trial outputs will ask users for feedback on the outputs, including their availability and clarity. Data received from this survey, alongside more detailed feedback on the estimates and outputs obtained from stakeholders, will be used to assess this question. We acknowledge that not all users may view the outputs as clear and accessible, however, if the majority of users respond positively then this will be deemed a success.

This research question is important because without easily available and clear outputs, the redevelopment of the survey questions and domestic abuse measures have limited benefits. User feedback will be vital when making a final decision on whether this research question has been met, along with understanding any improvements needed.

The new domestic abuse questions were introduced onto the Crime Survey for England and Wales (CSEW) on 1 April 2023 on a split-sample basis and will continue to be included until the provisional date of March 2025. Over the next year we will analyse the data received, using this information along with other sources, to answer the research questions and assess the evaluation criteria.

Our next update will be published in late 2024 where we aim to include trial outputs based on the new questions. Following this, we will conduct a user survey to seek feedback on the trial outputs.

In early 2025 we will make a final assessment of the evaluation criteria and use this to decide the next steps for the domestic abuse survey questions and prevalence measurements.

More details on the 29 research questions, their associated evaluation criteria and underpinning quality dimension are detailed in this section. We will remain flexible as the evaluation continues and may add further research questions in the future.

Relevance

Evaluation criteria

The new questions and proposed outputs meet the priority requirements of users.

Research questions

1.1. How have stakeholder requirements been assessed throughout the redevelopment process?

1.2. What are the priority requirements?

1.3. Have the new questions been assessed against user requirements and do the proposed outputs meet priority requirements?

1.4. Are there user requirements that have not been met from the new domestic abuse questions?

Evaluation criteria

Data collected from the new questions can produce prevalence estimates and breakdowns at the same level of detail as is currently produced.

Research questions

1.5. How do we define a victim based on the behaviours they have experienced?

1.6. Can we derive prevalence measures to the same level of detail as we currently do?

Evaluation criteria

The majority of users are satisfied with the outputs produced from the new questions.

Research questions

1.7. How has user feedback on the new questions during the redevelopment process been collected and does it demonstrate user satisfaction?

1.8. Are the majority of users satisfied with the proposed outputs published from the new questions trialled on the Crime Survey for England and Wales (CSEW)?

Evaluation criteria

Methods for producing abuse profiles have been tested and shared with users.

Research questions

1.9. Are we able to model domestic abuse profiles from the data we collect from the new questions?

1.10. Can we produce estimates from the abuse profiles?

Accuracy and reliability

Evaluation criteria

The new questions do not lead to a significant increase in non-response rates.

Research questions

2.1. What percentage of people do not respond to the new domestic abuse self-completion questions?

2.2. Are particular demographic groups not answering the new domestic abuse questions that were answering the current domestic abuse questions?

Evaluation criteria

The new questions accurately measure lived experiences of victims of domestic abuse.

Research questions

2.3. Has feedback been gathered from experts that the new questions cover all aspects of domestic abuse?

2.4. Has feedback been gathered from a range of respondents/victims/survivors about whether the questions accurately measure lived experiences of domestic abuse?

2.5. How do the estimates of overall domestic abuse prevalence from the new questions compare with prevalence measures derived using the current domestic abuse questions?

2.6. Are the widths of confidence intervals for headline domestic abuse prevalence estimates from the new questions a similar width to those of the current questions?

2.7. Have respondents answered the impact questions (what is the drop off of people who answered the behaviour questions but not the impact ones)?

2.8. How have respondents responded to the new domestic abuse questions that were highlighted in testing as being important to analyse?

Timeliness and punctuality

Evaluation criteria

The new questions do not affect the timeliness and punctuality of our domestic abuse publications.

Research questions

3.1. Are estimates from the new questions produced within the same time period as those from the existing questions (for the annual Crime publication in July and the domestic abuse publication in November)?

3.2. Can we produce more complex statistics (for example modelling for abuse profiles) within the same time period, and, if not, can they be published at a later date?

Comparability and coherence

Evaluation criteria

The comparability between estimates produced using the new questions and the existing time series is clear to users.

Research questions

4.1. Have we assessed the trade-offs of losing the existing domestic abuse time series due to developing new questions and do the benefits of the new questions and data outweigh losing the existing time series?

4.2. Has information about the comparability of new estimates with previous estimates been published and clearly explained to users?

4.3. How will the comparability of domestic abuse estimates be maintained in the future if the new questions are included on the Crime Survey for England and Wales permanently?

Evaluation criteria

Estimates produced using the new questions closely align with government definitions of domestic abuse for example, the Serious Crime Act and the Domestic Abuse Act 2021, and any differences are clear to users.

Research questions

4.4. Do the new questions, and estimates produced from them, better align with the Domestic Abuse Act 2021?

4.5. How have we measured controlling and coercive behaviour and are we measuring it in line with the Serious Crime Act?

4.6. Are there any areas where we do not align with government definitions and is this clearly explained to users?

Accessibility and clarity

Evaluation criteria

Estimates produced using the new questions are presented in a format that is easily available and clear to users.

Research questions

5.1. Are domestic abuse data from the new questions presented in a similar format to how they are currently and do the new data tables meet accessibility requirements?

5.2. Do the majority of users think the outputs and data from the new questions are easily available and clear?

Evaluation criteria

Supporting information about the changes to the questions, and estimates produced using them, is easily available and clear to users.

Research questions

5.3. Have we documented the changes to the new questions compared with the current questions and has this been made available to users?