In my previous letter, I explained the challenge of lower response rates of the main sample of the Labour Force Survey (LFS), and the additional sample used to build the Annual Population Survey (APS). I committed to respond with our assessment of APS quality, including the feedback provided by key users at a workshop that was held on 2 October 2024.
The high-level assessment following the workshop on 2 October was that:
- the total sample size is still relatively large and at a national level where results remain reasonably robust with a small increase in variability compared with the pre-COVID-19 period
- headline data at a regional level is acceptable, but some cross-tabulations (such as unemployment by age band) cannot be published as sample sizes are too low
- the quality of data for local authorities has deteriorated and the majority of users are experiencing issues with producing results at this level from the APS
- the reduction in the achieved sample also impacts on the ability to publish estimates for other smaller segments of the population (for example, small age bands or specific demographics)
Our assessment considered evidence from across Office for National Statistics (ONS) outputs, including labour market, well-being, and population and demography. The evidence included the reduction in the reduction in the available sample for producing estimates at local levels and small domain levels:
- the increase in confidence intervals of published estimates
- the impact on APS estimates of its current weighting approach
External users also provided views related to their outputs, which were in line with the assessment of ONS output.
Overall, our view on the quality of the APS is that, while it is robust for national and headline regional estimates, there are concerns with the quality of estimates for smaller segments of the population, such as local authority geographies. In addition, we recognise the need to review and update the weighting approach used in the APS. Since analysis at the local area is one of the main uses of the APS data, we want to be clear with users about the extent to which different statistics are affected, and how and when APS data should be used in analysis. To help give that clarity to users, I would like to request that we suspend the accreditation of ONS outputs using the APS data while we undertake further work on APS quality. Changing the designation of the statistics will help to convey the uncertainty and clearly signal the limitations of the data to users to support their appropriate use. It also demonstrates our commitment to fully understanding, assessing and improving the quality of the APS data.
Our immediate next step is to arrange a follow-up workshop with users in early November. The aims of this workshop will be to:
- provide clarity on how APS data can currently be used and what the limitations are; the information provided by this discussion will be used to produce a technical note to be published by the ONS following the workshop, to provide further clarity to the wider set of users of the APS
- assess how quality concerns could best be addressed and any additional actions that would be necessary
We will update you further after the workshop and the publication of the technical note. The actions identified for the APS will build on the measures we are already taking to improve the quality of the LFS.
Alongside this letter, we understand the Welsh Government plan to request that the accreditation is suspended for Welsh Government APS outputs due to be published on 9 October.
We will communicate our decision on accreditation to Chief Statisticians at departments who publish outputs using the APS, encourage them to carry out their own assessment of quality and consider the designation of their APS outputs.
While we work to improve the quality of the APS and therefore the outputs produced from it, we are committed to keeping both you and our users informed of developments. In addition to the follow-up workshop with APS users, we will also include information about the APS in our regular updates on the LFS and the Transformed Labour Force Survey (TLFS).
Yours sincerely,
Michael Keoghan
Deputy National Statistician and Director General, Economic, Social and Environmental Statistics Group | Office for National Statistics