ONS Statistical Quality Improvement Strategy

Quality is fundamental to building trust in Office for National Statistics (ONS) statistics. Data and statistics are rapidly changing with the introduction of new methods and complex datasets. As our data increase in complexity, it becomes harder to assess quality, but producing high quality statistics is now more important than ever. The responsibility for producing high quality statistics lies with everyone working in the organisation.

This strategy aims to improve quality across the ONS in the production of statistics and analysis that serve the public good. It sets out what we will be doing to improve the quality of our statistics and analysis and how we will manage the processes surrounding their production. The ONS Quality Committee will oversee the implementation of this strategy and will work with divisions to embed the objectives across ONS.

Delivering the objectives in this strategy will ensure we comply with the quality requirements of the Code of Practice for Statistics. High quality statistics and sound knowledge about quality will lead to statistics that serve the public good, helping the ONS achieve the aims of the UK Statistics Authority’s Statistics for the public good strategy.

Sir Ian Diamond
National Statistician

The Office for National Statistics (ONS) Statistical Quality Improvement Strategy (SQIS) sets out our organisational commitment to quality and the actions we will take to improve the quality of ONS statistics and analysis. This strategy also acts as the ONS implementation of the Government Statistical Service (GSS) Quality Strategy, which is a two-year strategy to improve statistical quality across the GSS.

The quality of ONS statistics is important to ensure that they meet user needs and are fit for purpose. We strive to provide high quality statistics to our users to ensure they have trust in us as a statistics producer, and we make sure users have enough information on the quality of our statistics to support effective decision-making.

We have already taken steps to improve quality through our ONS quality champions network and rolled out quality training (run by the Government Data Quality Hub) to our staff. We are also using our expertise and emerging Data Access Platform to pilot the application of Reproducible Analytical Pipeline (RAP) techniques in statistical production, to improve production time and reduce the risk of errors. Further information on our quality improvement plans can be found on this page.

This strategy will be reviewed annually by the ONS Quality Committee, which provides assurance over the quality of ONS statistics and sets strategic direction on quality across the ONS.

According to the Code of Practice for Statistics, quality means that statistics fit their intended uses, are based on appropriate data and methods, and are not materially misleading. Quality is an important pillar within the Code that requires us to:

• be open about our commitment to quality and make clear our approach to quality management (practice T4.5)
• undertake systematic and periodic quality reviews (practice Q3.5)
• ensure that all staff involved in the production of statistics and data are provided with training on quality management (practice T5.4)
• ensure our statistics are based on data sources that are appropriate for the intended uses (practice Q1.1)
• ensure the strengths and limitations of our statistics and any uncertainty are clearly explained alongside the statistics (practices Q3.1 and Q3.3)

To determine whether outputs meet their needs, we measure quality in terms of the five quality dimensions of the European Statistical System (ESS):

• relevance
• accuracy and reliability
• timeliness and punctuality
• accessibility and clarity
• coherence and comparability

We use various measures to assess the quality of our statistical outputs such as standard errors, revision rates and user consultations.

In line with the Government Statistical Service (GSS) Quality Strategy, the Office for National Statistics (ONS) has four strategic objectives for improving statistical quality. In addition, we also have a fifth objective to ensure that ONS analysis is relevant, has impact and is fit for purpose:

• We will all understand the importance of our role in producing high quality statistics.
• We will ensure our data are of sufficient quality and communicate the quality implications to users.
• We will anticipate emerging trends and changes and prepare for them using innovative methods.
• We will implement automated processes to make our analysis reproducible.
• We will ensure that our analysis is relevant, has impact and is fit for purpose.

We now list specific actions against each objective, which together form our strategy. Achieving this will require commitment at leadership and working level to move towards a culture of quality improvement and sharing best practice across the organisation.

We will all understand the importance of our role in producing high quality statistics

Everyone in the Office for National Statistics (ONS) has a responsibility for the quality of the statistics that we produce. Quality needs to be built into the whole production process and at the forefront of analysts’ minds to produce high quality statistics. We must have clear quality management, structures and practices to monitor, measure and improve statistical quality and will ensure all those involved in producing ONS statistics understand how their role directly impacts on quality.

The following actions will be taken to achieve this objective:

• We will have clear governance on quality to ensure everyone across the organisation understands their role in producing high quality statistics. The ONS Quality Committee will provide assurance over the quality of ONS statistics and set strategic direction on quality across the ONS. The Quality Committee will monitor progress against the actions identified in this strategy on a bi-annual basis and report this to the National Statistics Executive Group (NSEG). The Quality Committee will be supported by the ONS Statistical Head of Profession who is responsible for ensuring the organisation complies with the Code of Practice for Statistics. Our Deputy Directors will have primary responsibility for quality in their divisions and will implement this strategy in their area. They will instil a culture of continuous quality improvement to our statistics where quality is seen as everyone’s responsibility. The Government Data Quality Hub will act as a centre of expertise for quality, providing advice and support to business areas in the ONS and across government.
• We will undertake periodic quality deep dives to look at statistical areas (for example, trade) and explore the extent to which the five dimensions of statistical quality are being met and propose improvements. We will also utilise our Regular Quality Review (RQR) process to assess and improve the quality of ONS statistical outputs as well as identifying systemic issues that need to be prioritised across the organisation. We will review the RQR process to ensure it is fit for purpose and is driving quality improvements.
• We will introduce mandatory training for new starters on the Code of Practice for Statistics. It is recommended that all staff, particularly those in statistical production and dissemination, also attend the training. This will ensure our staff are aware of the principles and practices on quality within the Code.
• We will ensure that all staff involved in statistics production use the quality statistics in government guidance and training offered by the Government Data Quality Hub. This will help ensure our staff understand the importance of quality and how it fits into their work as well as good practice in Quality Assurance.
• We will ensure sufficient time is built into our production processes for Quality Assurance and use Quality Assurance checklists across the organisation to record what checks have been undertaken in preparation for publication and by whom. Deputy Directors will ensure these checklists are used in their divisions and encourage them to be reviewed regularly to ensure they are fit for purpose.
• Quality management practices will be supported by a data architecture that enables us to trace errors to the root cause, even as data are transformed and processed. This will be achieved through the use of metadata to support data lineage techniques and practices, which will enable us to understand the journey data follow from collection to publication of our statistics. We will ensure that all ONS statistics have clear documentation of the data journey, including process maps.
• We will continue to use curiosity panels, chaired by a Deputy Director, to provide an opportunity for teams to sense-check results, agree the main messages and place the findings in context before our statistics are published.
• We will maintain and co-ordinate an ONS Quality Champions network to share best practice and lessons learnt on quality across the organisation. Our champions will work with Deputy Directors to identify improvements to quality that need to be made. We will also engage with the Government Statistical Service (GSS) through the GSS Quality Champions network.
• We will engage with wider networks (both cross-government and internationally) that are related to the quality of our statistics, such as the Population and Migration Statistics Advisory Board.

We will ensure our data are of sufficient quality and communicate the quality implications to users

The quality of our statistics is underpinned by the quality of the underlying data. High quality data are not sufficient to ensure high quality statistics, but they are a fundamental pillar of this. We will manage data quality through effective data management processes and encourage our analysts to be curious and question the data they are working with.

We will continue to deliver against our Data Strategy, which places high quality data and analysis at the centre of our mission to mobilise the power of data to help Britain make better decisions. Our Data Strategy is supported by a suite of Data and Statistical Policies, which define our data practices and enable us to adopt effective, proportionate and robust quality standards, including how we store and manage metadata.

The following actions will be taken to achieve this objective:

• We will follow the Quality Assurance of Administrative Data (QAAD) toolkit for our statistics that are produced using administrative data sources. The Government Data Quality Hub is currently developing similar toolkits for other data sources, and we will apply these to our statistics produced using other data sources (for example, survey data). We will ensure that we communicate effectively with our data suppliers to understand the quality of the underlying data and continuously seek ways to improve the quality. We will continue to publish QAAD reports to illustrate to our users how we have applied the QAAD toolkit to assess the quality of our administrative data sources. We will ensure staff attend QAAD workshops run by the Government Data Quality Hub so they can apply the toolkit to the data they work with. We will utilise automated data management processes to comply with the toolkit (for example, our data lineage techniques based on metadata (referenced earlier) will be used to understand the data journey and inform our QAAD reports).
• To increase transparency and trust in our statistics, we will include information on the quality of our statistics within our statistical bulletins. In addition, we will continue to publish Quality and Methodology Information (QMI) reports alongside our statistical bulletins (for example, Retail Sales Index QMI). These will help our users understand the strengths and limitations of the data, so that they can make the best decisions on how to use it. We will regularly review our QMIs to ensure they remain up to date and meet user needs.
• As set out in the Code of Practice for Statistics (practice T3.9), we will use the revisions section of the ONS website to be transparent to our users on how we handle revisions and errors to our statistics. We will ensure that lessons are learnt from any errors made in our statistics to improve our processes.
• We will continue to update our landscape of Data and Statistical Policies, ensuring that they remain relevant and fit for purpose, and place quality at the core of our data practices.
• We will continue to deliver against our strategic data objectives, including developing and implementing the capabilities needed to process and link datasets and to enhance our metadata capabilities.
• We will comply with ONS Data Principles, which support the definition of effective data management processes to manage the flow of data into the ONS and inform what rules, policies and standards are required to ensure its quality.
• We will contribute to the Cross-Government Data Architecture community and share best practice in data management across government. We will also support other initiatives across government aimed at increasing the quality and shareability of data (for example, Data Leaders Network and National Data Strategy) and will use opportunities arising from these platforms to promote best practices consistent with this strategy.
• We will work closely with the Data Standards Authority to support the creation and implementation of cross-government data standards and improve collaboration on data. The Government Data Quality Hub will lead the development of a single cross-government data maturity model. This will allow departments to consistently evaluate their data capability and measure the impact of data initiatives, and it will facilitate more effective targeting of funding and support.
• We will apply the principles outlined in the new cross-government Data Quality Framework to assessing the quality of our data. We will promote engagement with the framework across the organisation and organise training to ensure staff know how to apply it.

We will anticipate emerging trends and changes and prepare for them using innovative methods

As the world of data and technology change, new opportunities and challenges for our analysts will emerge. The detail, volume and frequency of data collected are rapidly increasing, as are the requirements for innovative methods, tools and techniques. We need to be prepared in this changeable environment to produce with efficiency high quality statistics that reflect user needs.

The following actions will be taken to achieve this objective:

• We will work collaboratively with experts such as Methodology, the Data Science Campus and academia to ensure the ONS uses innovative methods.
• We will engage with the development of National Statistician’s Quality Reviews (NSQRs) produced by the Government Data Quality Hub and take on board recommendations. NSQRs cover thematic topics of national importance and ensure that methods used by the GSS are keeping pace with changing data sources and technologies. Engaging with NSQRs will ensure that the methods used by the ONS remain up to date.
• We will continue to use the Admin Data Methods Research Programme to address some of the main challenges of administration and transaction data set out by Professor David Hand in his 2018 paper.
• We will review the findings and implement the recommendations identified by the systemic reviews produced by the Office for Statistics Regulation.
• We will continue to use our Data Science Campus and Data Science skills across the organisation to exploit alternative data sources. We will integrate quality into the decision-making on the suitability of these new data sources. We will work with the Government Data Quality Hub to ensure we understand the quality of these new data sources.
• As required by the Code of Practice for Statistics, we will continue to engage with users of our statistics through mechanisms such as StatsUserNet, Knowledge Hub and user groups to gather their views and ensure that our statistics meet their needs. The GSS Best Practice and Impact division (BPI) is leading a project to develop a GSS-wide user engagement strategy and implementation plan. Once the strategy has been published, we will develop an ONS action plan to implement the strategic goals across the organisation.
• We will support the GSS in improving the quality of their statistical methods by facilitating both the Methodology Advisory Committee (MAC) and Methodology Advisory Service (MAS). MAC is a free methodological advice service with access to a pool of experts spanning academia, the private sector, the GSS and National Statistical Institutes (NSIs). MAS is a free service providing methodological advice from ONS Methodology division and guidance. Both MAC and MAS are facilitated by the Government Data Quality Hub.
• We will work across the GSS to find innovative methods to make statistical data available to users of statistics with the goal to enhance user experience and the options available to explore statistical data.
• Our methodologists will continue to research and develop methods for the integration, analysis and production of official statistics from a variety of sources.

We will implement automated processes to make our analysis reproducible

We will apply software engineering techniques to statistical production to automate our processes and enable reproducible analysis. This will be achieved through the design and implementation of Reproducible Analytical Pipelines (RAP) principles and techniques to reduce the quality risks and broader organisational risks associated with legacy statistical production practices and systems.

We will adopt a strategic approach to the implementation of RAP principles and techniques, embedding this approach as one of several to develop Statistical Production Core Capability. This will help to ensure that:

• consistent criteria are applied to determine the suitability of RAP principles and techniques for a given statistical context
• implementation of RAP principles and techniques are co-ordinated and governed in a joined-up, consistent way through cross-cutting technology and service teams, and the ONS Data Service where appropriate, and delivered through existing statistical transformation programmes.

The following actions will be taken to achieve this objective:

• We will engage with the GSS RAP Champions network to support the implementation of RAP techniques across the organisation, and we will ensure best practices are embedded within programmes to develop Statistical Production Core Capability.
• We will ensure adequate training is developed and made available to staff to enable sustainable, business-led adoption of RAP principles and techniques. This will include Government Digital Service (GDS) training and the RAP companion, and it will be supported by the education and support stream of the ONS Data Service.
• We will use our expertise and emerging Data Access Platform to pilot the application of RAP techniques in statistical production, to improve production time and reduce the risk of errors. These pilot approaches will be used to upskill staff and inform the roll-out of RAP to support how we develop our Statistical Production Core Capability.
• We will ensure that RAP techniques are applied consistently and governed at a strategic level as an integral part of wider Statistical Production Core Capability through the Data Service Engagement Framework and associated processes and governance. This will ensure standard processes are applied and that sustainable data engineering processes are developed as much as possible, and it will help statistical producers understand the trade-offs of automation.

We will ensure that our analysis is relevant, has impact and is fit for purpose

The ONS Quality Committee will engage with ONS analytical Heads of Profession to ensure that analysis undertaken is consistent with the Aqua Book guidance for producing quality analysis for government and, where relevant, other guidance encapsulated in the analysis function standard (for example, Green Book and Magenta Book).

Modelling is essential to the work of the ONS and therefore it is important that the models we use are fit-for-purpose. We will embed best practice in modelling across the ONS and ensure we are complying with the Macpherson principles for business-critical models.

The following actions will be taken to achieve this objective:

• We will develop and maintain a list of “business-critical models” that we use – these are our most important models, in terms of high amounts of money involved and high sensitivity (influential and widely used).
• We will engage with ONS analytical Heads of Profession to ensure that ONS analysis is consistent with the Aqua Book guidance for producing quality analysis for government and, where relevant, other guidance encapsulated in the analysis function standard (for example, Green Book and Magenta Book).
• We will ensure that all ONS models have clear documentation of the methodology, including model maps (diagrams that summarise the main structure of a model, its inputs, its assumptions, its transformations and its outputs – see example) and sound descriptive commentary.
• We will create a culture of openness around errors and near misses in modelling to ensure lessons are learnt to improve processes.
• We will create a chain of accountability for all our models with a Senior Responsible Officer (SRO) assigned to each of our models.
• We will create a strong Quality Assurance culture for modelling in the ONS through representation at the cross-government expert modelling network (which covers all modelling activity across government), conducting regular internal peer review of our models and promoting an awareness of the Macpherson review of modelling across the organisation.
• We will increase transparency of code and provide additional resources to move to open source tools.
• We will work with the Government Data Quality Hub to review the state of modelling across the organisation and identify where improvements need to be made.

The Office for National Statistics (ONS) Quality Committee will monitor progress against the actions identified in this strategy on a bi-annual basis. Progress will be reported to the National Statistics Executive Group (NSEG).