Quality assurance of Continuous Working Patterns Survey data used in regional productivity statistics

Background

The Office for National Statistics (ONS) Productivity Team uses the Continuous Working Patterns (CWP) Survey from the Ministry of Defence (MoD). This data source is used in most of our productivity hours datasets and to produce two experimental statistics: Region by industry labour productivity and Quarterly regional productivity hours and jobs.

The Region by industry labour productivity statistics provide insight into regional productivity differences. The Quarterly regional productivity hours and jobs statistics is a main input series in estimating the changes in labour inputs by region.

The MoD data are used to help estimate total hours worked by armed forces personnel, because the household and business surveys that provide information about most industries do not cover the armed forces. Using data from the MoD increases the accuracy of the two experimental labour productivity statistics and other productivity hours datasets.

This report provides information on this MoD data source, describes how the ONS uses it, evaluates the strengths and limitations of the processes involved and outlines next steps to be taken.

UK Statistics Authority QAAD and survey data

The Quality Assurance of Administrative Data (QAAD) document (PDF, 298KB) describes the quality assurance arrangements required for a statistic compiled using administrative data to be compliant with the Code of Practice for Statistics and so to be certified as a National Statistic.

The Continuous Working Patterns (CWP) Survey data are collected primarily for research reasons, rather than administrative reasons, so in this regard it is not an administrative data source. However, the QAAD provides a wide-ranging approach to conducting quality assurance on data sources that are collected by organisations external to the statistical producer. This quality assurance report will therefore follow the structure and approach outlined by the QAAD document.

This report seeks to demonstrate that the ONS has undertaken the appropriate quality assurance actions on this Ministry of Defence data source. By doing so, this report will bring the two experimental labour productivity statistics closer to being accredited as National Statistics and so being regarded as high-quality, trustworthy statistics that meet users’ needs.

UK Statistics Authority Administrative Data Quality Assurance Toolkit

The QAAD document and the Administrative Data Quality Assurance Toolkit (PDF, 243KB) describe how, for each data source, the statistical producer should decide upon a level of assurance for each of the four practice areas associated with administrative data. The choice of level should be informed by the level of risk of quality concerns and the level of public interest in the statistic produced, as shown by Table 1. In Table 1, the levels of assurance are referred to as A1, A2 and A3 in ascending order of assurance level.

The Toolkit outlines four specific practice areas for assurance and the rest of this report will focus on these areas in turn. These are:

• operational context and administrative data collection

• communication with data supply partners

• quality assurance principles, standards and checks applied by data suppliers

• producer’s quality assurance investigations and documentation

Public interest profile of the statistics produced using the MoD data source

This section considers the public interest profile of the two experimental statistics produced by the Office for National Statistics (ONS) using the Ministry of Defence (MoD) Continuous Working Patterns (CWP) Survey data source. These experimental statistics are: Region by industry labour productivity and Quarterly regional productivity hours and jobs. The level of public interest in these statistics is determined by: the level of media interest, the level of political sensitivity, the type of decisions the statistics inform and the breadth of the user base.

Although these industry by regional labour productivity statistics have a lower user profile than the quarterly labour productivity National Statistic series, they are receiving increasing media and user interest as further analysis is released. These statistics have a low level of political sensitivity, as changes in productivity cannot be solely ascribed to specific government policies, so these statistics cannot be used to judge government policies directly.

The regions included within these statistics are relatively large, with 12 regions at the NUTS1 (Nomenclature of Territorial Units for Statistics 1) level in the UK, so it seems unlikely that these statistics will be granular enough to inform business decisions.

On this basis, there is a low level of public interest in these statistics.

Level of data quality concerns

The MoD Continuous Working Patterns (CWP) Survey contributes towards two statistical outputs produced by the ONS: Quarterly regional productivity hours and jobs and Region by industry labour productivity. The indicative calculations in this section show that this MoD data source makes a very small contribution towards these two statistics.

The approximate size of the CWP survey’s contribution towards the UK labour productivity estimates can be estimated by comparing total hours worked by armed forces personnel with the total number of hours worked across all industries. When calculated using the average hours from the 2018 to 2019 CWP Survey and the number of armed forces personnel on 1 April 2019 as found in the MoD Annual Location Statistics and the Quarter 1 (Jan to Mar) 2019 seasonally-adjusted productivity hours from Quarterly regional productivity hours and jobs, the armed forces hours worked were 0.6% of the all-industry UK total.

When this method is applied to individual regions, the lowest percentage was 0.07% in North West England and the highest was 1.79% in South West England.

The Region by industry labour productivity statistic uses labour inputs to calculate productivity estimates. These labour inputs are the same as those found in Quarterly regional productivity hours and jobs, except they are annual rather than quarterly and are broken down by industry. Although the contribution of the armed forces data to the final output will be similarly low across both statistics at the whole economy level, their main contribution is to inform productivity estimates for industry O, public administration and defence, across the NUTS1 regions.

The previous calculations provide an indication of the contribution of this MoD data source towards estimates of productivity levels and input levels. However, the most important statistics for many users are the estimates of productivity growth rates and input growth rates. As the average weekly hours worked by armed forces personnel changes little over time, this MoD data source’s contribution towards growth rates will be very small.

As Section 3 describes, the CWP Survey data are released alongside an annual Background Quality Report, which explains any changes in the methodology.

This suggests a low level of risk of quality concerns for this data source.

The CWP Survey has received the levels of assurance shown in Table 2 because of the low level of public interest and the low level of risk of quality concerns. The risk level for this data source is low because of its very small contribution towards the two experimental labour productivity statistics. The following sections describe how the ONS has met these levels of assurance for each of the four practice areas associated with the MoD’s CWP Survey dataset.

Operational context and data collection by the Ministry of Defence (QAAD matrix score A1)

Overview of the CWP Survey

The Continuous Working Patterns (CWP) Survey is produced annually by the Defence Statistics Group in the Ministry of Defence (MoD). The CWP Survey is a stratified random sample survey distributed to members of the UK armed forces who are trained, regular and work full-time. The survey is stratified by Service (Naval Service, Army and RAF), by rank group and by broad location (for example, UK and overseas).

The MoD Defence Statistics Team has produced information on the working patterns of UK armed forces personnel since 1987 to 1988. The CWP Survey was first released in its current form in 2014.

Purpose of the CWP Survey

The main users of the CWP Survey are the Chief of Defence People Policy Team and the Armed Forces Pay Review Body (AFPRB). The AFPRB advises the Prime Minister and the Secretary of State for Defence on the remuneration and charges for members of the UK armed forces. The CWP Survey data can also be used to answer Parliamentary questions and Freedom of Information requests.

There is no legal requirement for the MoD to publish the CWP Survey.

Production of the CWP Survey

Surveys are packed and distributed by external contractors then dispatched via the British Forces Post Office to Commanding Officers, who then distribute them to staff. The surveys are sent out in two waves, between Quarter 3 (July to Sept) the year before release and Quarter 2 (Apr to June) on the year of release.

Response to the CWP Survey is voluntary. Respondents are asked to provide administrative information and to complete a seven-day diary indicating the amount of time they spend at work, on call, on breaks and off duty. The respondents return their survey to their Unit Admin Office or return it directly by post. The data processing of the CWP questionnaires is conducted by external contractors.

All data are stored, accessed and analysed using the MoD’s restricted network and IT system, and all staff involved in producing the CWP adhere to data protection regulations from the MoD and Civil Service, as well as the codes of practice produced by the Government Statistical Service.

Once collected, the survey responses are weighted to correct for bias caused by over- or under-representation of responses for each group. The data then undergo automated data clearing rules. It takes approximately eight weeks for the collected data to be published as a report.

The CWP Survey results are published in July or August each year, containing data on questionnaires completed from between Quarter 3 the previous year and Quarter 2 on the year of release.

The CWP Survey results are released alongside a main report (PDF, 472KB) and a Background Quality Report (PDF, 140KB), which describes features such as the response rates and methodologies. For instance, in 2018 to 2019, out of a sample size of 24,470 personnel there were 2,724 valid responses, where a survey is regarded as valid if at least one day out of the seven is useable. Therefore, the response rate was 11%.

Sources of bias in the CWP Survey

The MoD’s Background Quality Report for the CWP Survey describes the potential for bias if non-response rate is directly related to individual working patterns. For instance, those who are busiest and work longer hours may be less likely to complete the survey. This would result in fewer responses from busier people, causing the reported average hours value to be below the true value. On the other hand, the most organised respondents may work longer hours and be more likely to complete the survey. As a result, there could be a positive or negative effect on the ONS estimates of hours worked and productivity.

Strengths and weaknesses

Strengths:

• the Survey has the same format across all three Services (Naval Service, Army, RAF)

Weaknesses:

• the survey is voluntary and is filled in manually, which can introduce inaccuracies

• the response rate was 11% (2,724 valid responses out of a 24,470 sample size) in 2018 to 2019

• characteristics affecting working hours may affect the response rate, potentially introducing bias

Next steps

The ONS will continue to check the CWP Survey’s Background Quality Report for any changes in methodology that may affect users, such as changes to the data collection process.

Communication with data supply partners (QAAD matrix score A1)

Release information for the CWP Survey

The Ministry of Defence (MoD) announces the release date for the Continuous Working Patterns (CWP) Survey data on the GOV.UK website’s statistical release calendar. The CWP Survey data are released alongside an overview report and copies of the survey forms, including the guidance given to respondents on how to complete the survey. The CWP Survey data are also released alongside a Background Quality Report (BQR), which describes the CWP Survey methodology and describes changes in the survey process, for example, changing the number of waves of surveys sent out.

The working relationship between the ONS and the MoD over the CWP Survey

The Office for National Statistics (ONS) does not contact the MoD directly. However, the ONS can email the MoD if there are any queries over data or quality assurance processes.

While there is no formal contract requiring the MoD to publish these data for the UK labour productivity statistics, the MoD has been forthcoming and helpful in providing further information about queries and revisions. The ONS continues to engage with the MoD in the writing of this report and we intend to maintain this level of communication.

MoD user engagement

The CWP Survey is an Official Statistic and so is covered by the MoD User Engagement Policy (PDF, 59KB), which sets out how the MoD consults users regarding proposed changes and how users can provide the MoD with feedback. This Policy states that any users of the MoD’s statistics may request to be kept informed via email of any proposed changes to those statistics or any other relevant issues.

The MoD’s Defence Statistics Group works closely with the survey’s sponsor, the Chief of Defence People, to ensure the relevance of the CWP report. The MoD has run consultations when proposing changes to publications. For instance, the MoD consulted on changes to the definition of army-trained strength in 2016. The MoD also ran a formal consultation on changes to the MoD Armed Forces Personnel Statistics in 2015.

Strengths and weaknesses

Strengths:

• the MoD announces the CWP Survey publication release date in advance on the GOV.UK website

• the MoD works closely with internal users, invites feedback from external users and will run consultations when proposing changes to publications

• the MoD informs users of methodology changes in the Background Quality Report released alongside the CWP Survey data

• the ONS maintains a current email address with which to contact the MoD

Weaknesses:

• there are no formal regular meetings between the ONS and the MoD, potentially leading to the ONS missing methodology changes

Next steps

The ONS will examine the potential for agreeing on an annual statement of data requirements with the MoD.

QA principles, standards and checks applied by data suppliers (QAAD matrix score A1)

The MoD quality assurance arrangements for the CWP Survey

The Ministry of Defence (MoD) designates the Continuous Working Patterns (CWP) Survey as an Official Statistic, so it is produced in accordance with the MoD Statement on Quality Guidelines Policy (PDF, 64KB). This Policy states that the MoD seeks to build quality into their Official Statistics by using methods such as validating and cleansing the raw data and using established quality assurance processes.

There are no comparable data sources with the CWP Survey, which limits the possibility for conducting validity checks.

The MoD applies automated data cleaning rules to ensure that all responses within a questionnaire are consistent. These rules are published in the CWP Survey main report. For example, if a respondent has filled in work hours in the diary when they are recorded as being on annual leave, the hours are ignored.

Revisions and amendments to the CWP Survey

The personnel location datasets are Official Statistics, so they are revised and amended in accordance with the MoD Revisions and Corrections Policy (PDF, 60KB). This Policy states that, if errors are found or figures are amended because of methodology changes, the MoD Head of Profession for Statistics will decide whether a minor or major error has occurred. Major errors cause the overall message to change and will be corrected as soon as possible. Minor errors are less substantial and will be corrected within 12 months.

The MoD informs users of its revisions and amendments to its Official Statistics by adding a note to the overview document and by including indications in affected spreadsheets. For spreadsheets where only a small number of values in a table are updated, an “r” is included above the affected columns. For spreadsheets where a more substantial number of figures are updated, the revisions and the reasons for them will be identified in commentary at the beginning of the relevant section and above the affected tables.

Strengths and weaknesses

Strengths:

• the MoD staff handling the CWP Survey data adhere to the relevant regulations and codes

• the MoD follows consistent automated data cleaning rules and publishes them in the report

Weaknesses:

• there are no comparable data sources with which the MoD can compare the CWP Survey to

• the MoD policy is to amend minor errors within 12 months, so inaccuracies may persist for up to 12 months

Next steps

The ONS will continue to engage with the MoD to consider developing wider validation checks on these data as part of our ongoing quality assurance procedure.

Producer's quality assurance investigations and documentation (QAAD matrix score A1)

ONS processing of the CWP Survey

The Office for National Statistics (ONS) maintains internal step-by-step written instructions explaining how to download the Continuous Working Patterns (CWP) Survey data from the GOV.UK website and extract the relevant values. These instructions are updated to account for changes to ONS processes.

Once the spreadsheet is downloaded, the figure for average weekly hours worked (across all Services and Ranks) is manually extracted and inserted into another spreadsheet. This new spreadsheet displays previous years’ values and enables any large discrepancies in the time series to be manually identified. If such discrepancies are identified, the Ministry of Defence (MoD) is contacted by email to seek clarification.

The average weekly working hours for military personnel is then multiplied by the number of MoD military personnel in each region to produce the number of military hours worked in each region. This, in turn, contributes to the calculation of the labour productivity estimate in industry O: public administration and defence, for each region.

There are 23 other industries in the UK economy besides industry O: public administration and defence. As discussed previously, the contribution of the CWP Survey towards the UK total number of hours worked across all industries is very small, at 0.6% in recent periods.

ONS user engagement

The ONS engaged with users following MoD data collection errors in the MoD’s Annual Location Statistics publication. These errors resulted in the MoD revising data from April 2016 to April 2018, which affected the ONS labour productivity statistics over that period. The ONS engaged with its users by describing these changes under the Revisions section of the Labour productivity publication. If a similar error occurred with the CWP Survey, users would be informed in a similar way.

The ONS attends an annual UK Productivity Forum event. At this event, the ONS introduces users to new statistics and any changes to existing statistics.

The ONS encourages feedback by including email and telephone contact details on each webpage containing a dataset.

Strengths and weaknesses

Strengths:

• the ONS manually compares the latest CWP Survey figures with previous figures to identify large discrepancies

• the ONS maintains records of previous years’ CWP Survey data

Weaknesses:

• ONS processing involves manual copying and pasting across Excel spreadsheets, which raises the possibility of errors

• the ONS relies on the MoD for these data because household and business surveys do not cover the armed forces

Next steps

The ONS will investigate the possibility of using an automated system to extract data, to reduce the possibility for error.

This report has detailed the investigations undertaken by the Office for National Statistics (ONS) into the Continuous Working Patterns (CWP) Survey data provided by the Ministry of Defence (MoD).

This report has identified the strengths and limitations of the MoD CWP Survey data source. This data source makes a very small contribution towards the ONS productivity estimates and has a low public interest profile, suggesting that it requires a low level of quality assurance.

The main strengths of this data source are:

• an annual MoD Background Quality Report describing changes in methodology

• detailed step-by-step instructions describing how the data are extracted and processed by the ONS

The main limitations of this data source are:

• the ONS manually extracts data from spreadsheets, potentially introducing errors

• no formal regular meetings between the ONS and the MoD

• no legal requirement for the MoD to publish the CWP survey data

In seeking to improve our data sources, we will be taking the next steps to address these limitations. These steps will be communicated to users in the future quality assurance of administrative data (QAAD) report on this area.

However, despite these limitations, this data source fulfils the requirements of an A1 assurance rating because of its strengths, its low public interest profile and its small contribution towards the Region by industry labour productivity and Quarterly regional productivity hours and jobs statistics.

Labour productivity, UK: July to September 2019

All data related to Labour productivity, UK: July to September 2019

Region by industry labour productivity

Quarterly regional productivity hours and jobs

Labour Productivity Quality and Methodology Information (QMI)