Quality assurance of administrative data used in construction statistics

1.1 Background

National Accounts and Economic Statistics (NAES) within the Office for National Statistics (ONS) collects data from Barbour ABI on new orders in the construction industry. This data is used in our new orders dataset, and forms one source in the calculation of regional and type of work breakdowns for the index of construction (IoC).

This report outlines the process data take from initial collection through to the output of the release. It identifies potential risks in data quality and accuracy as well as details of how those risks are mitigated.

This report forms the latest in a series of quality assurance of administrative data (QAAD) reports produced by NAES to investigate the administrative data sources we use in the production of short-term economic output indicators as set out by the UK Statistics Authority. As such, this report specifically focuses on our administrative data use for new orders in the construction industry. Separate industries where we apply administrative data will be considered in other QAAD reports in the series.

Further information relating to quality and methodology for the new orders in construction data can be found in our report: New orders in construction QMI.

2.1 UK Statistics Authority QAAD toolkit

The assessment of our administrative data sources has been carried out in accordance with the UK Statistics Authority Administrative Data Quality Assurance Toolkit.

Each administrative data source investigated has been evaluated according to the toolkit’s risk and profile matrix (Table 1), reflecting the level of risk to data quality and the public interest profile of the statistics.

The toolkit outlines four specific areas for assurance and the rest of this report will focus on these areas in turn. These are:

• operational context and administrative data collection

• communication with data supply partners

• quality assurance principles, standards and checks applied by data suppliers

• producer’s quality assurance investigations and documentation

In the assurance of our data source, we have chosen to give a separate risk and profile matrix score (Table 1) for each of the four areas of assurance. This will allow us to focus our investigatory efforts on areas of particular risk or interest to our users (Table 2).

2.2 Assessment and justification against the QAAD risk and profile matrix

The risk of quality concern and public interest profile has been set as “low” due to the small contribution that new orders in construction statistics feed into the index of construction output (less than 0.01%). As such, a score of A1 is deemed appropriate for this data source.

All scoring was carried out by National Accounts and Economic Statistics (NAES) based on the level of risk of the data and interest of our users. Results for each area of assurance for construction are shown in Table 2. If you feel that this report does not adequately provide this level of assurance or you have any other feedback, please contact stoi.development@ons.gov.uk with your concerns.

3.1 Operational context and administrative data collection (QAAD matrix score A1)

This relates to the need for statistical producers to gain an understanding of the environment and processes in which the administrative data are being compiled and the factors that might increase the risks to the quality of the administrative data.

Barbour ABI is the current chosen partner of the Office for National Statistics (ONS) to collect new orders in construction data. Initially awarded this contract in 2013, it has again been renewed in 2017 following a further competitive tender. They are the UK market leader in the provision of construction data and they have a strong history of information acquisition and relationship building with important customers throughout the construction industry.

The service provided to National Accounts Economic Statistics (NAES) enables nearly complete coverage of UK construction projects and involved companies. Barbour ABI collect information on 100% of planning applications made to local authorities and also county councils, as well as many separate sources that are monitored daily, giving them a comprehensive picture of activity taking place currently or in the future. All collection of new orders data for NAES is supplied through these existing and well-established processes.

Barbour ABI also provides market reviews reporting on sectors in the construction industry that are experiencing growth and decline and to help their customers plan for the future. A fundamental part of Barbour ABI business is to track construction projects and involved parties across the public and private sector.

Barbour ABI has a large experienced in-house team that track every planning application and thousands of non-planning projects across the UK to deliver valuable insight and up to date construction leads and industry data to their customers. The data coverage tracks the activities of all local authorities and county councils in England, Scotland and Wales in addition to more than 10,500 contractors per annum. The value of the new orders data collected is that it measures main contractors by type of work and region.

The main users of this data are those working directly in the construction industry who require details of current and future activity, industry analysts requiring a leading indicator of the UK construction industry, trade associations making UK and international comparisons and other government departments such as Department for Communities and Local Government, Her Majesty’s Government. Barbour ABI are the Infrastructure and Projects Authority’s partner in providing the National Infrastructure and Construction Pipeline and trade associations such as the Construction Products Association.

3.1.1 Barbour ABI data collection for planned projects

The primary source for initially identifying construction projects is the planning process. Barbour ABI’s web crawling engine has been set up to link to all the local authorities in Great Britain allowing them to download all planning applications. Please note, at this stage contracts are not awarded and further investigation is needed through further telephone work. The data is collated daily and values are taken at current prices.

Once the data is collated, Barbour ABI’s quality assurance team will also look at the previous day and up to nine months of back data for any changes to be updated, for example, projects may change details such as funding during the process and any amendments to existing planning applications will be allocated a new application reference number.

Barbour ABI’s offshore business partner based in India acts as a data processor and is responsible for the initial data filtration following the data crawl. Each application is processed according to the available application details. The data are then sent back to Barbour ABI’s UK-based research team and allocated to the appropriate work stream, for example, smaller planning applications are processed but not researched whereas larger applications are prepared for further telephone research.

3.1.2 Barbour ABI data collection for projects outside of the planning process

The telephone research team makes over 1 million calls per year and speaks to more than 10,500 contractors about work they are bidding for or are already actively involved in. This enables the addition of contract data to planned schemes already being tracked, and also to identify further future projects at a very early stage in the construction life cycle. All the information is qualified on the telephone ensuring the data is correctly captured before data submission.

3.1.3 Barbour ABI project creation and tracking

Once projects have been identified and entered into the database, the system separates projects into

• schemes worth more or less than £100,000 for non-residential projects

• two houses or more for residential schemes

All projects are coded and categorised to identify different types of work taking place. Barbour ABI has mapped its in-house categorisation to the NAES taxonomy for 23 types-of-work examples, which include private and public-sector housing and infrastructure such as water, sewerage and electricity.

All planning applications of £100,000 and above are then passed to Barbour ABI’s in-house telephone research team. The team make more than 25,000 in-depth telephone calls per week to data providers and contacts across the specification chain – clients, design teams, consultants and contractors. The team is focussed on gathering important project data to provide their customers, including NAES, with as complete a picture as possible of activity taking place. This includes:

• contact details for relevant parties (name, job title, direct dial, mobile, email)

• contractor appointed as new order

• timings details (start, duration, finish and tender return dates)

• contract type

• construction value and funding

• consultant information

• material detail

All of these are collected for all projects with particular emphasis placed on project value, categorisation, funding and location for the purpose of NAES requirements.

To check the data that’s been collected, it is also gathered from other data sources, for example:

• Official Journal of the European Union, which includes details of public sector tenders and new orders as they are placed

• press and media channels, which are monitored daily to gather information on any project announcements and close links to media services within the construction industry such as building and design and property week digital magazines

• Google alerts, which are set up for the research team to capture further information

Additional research is carried out to identify future planned projects funded by local authorities, government departments, deregulated industry and the largest property owners in the private sector. Getting this information early helps Barbour ABI to gain valuable insight into up and coming new work.

The approach described in this section allows Barbour to collect 100% of planned project applications. However, for projects outside of the planning process there are no guarantees that some projects will not be missed from analysis NAES has concluded. Barbour ABI provides a virtual census of new orders data.

Barbour ABI estimates the market mark-up is approximately 80% on planned projects and 20% unplanned.

Data collected monthly by Barbour ABI means that provisional outputs by NAES are timely, but more reliable estimates are available in subsequent months. One of the benefits of the new orders in construction data is that contracts are tracked from order to completion so that any cancellations to contracts are reflected in revised data in future outputs.

The strengths of this method are:

• collects 100% of planned projects

• web crawling engine linked to all local authorities

• research team collates data from contractors and conducts further research

• data collected from other data sources to collate

• additional research for future planned projects

• all projects coded and categorised to Office for National Statistics taxonomy for 23 types of work

• timely monthly data tracked from order to completion

The weakness, however, is possible unplanned projects being missed. The next step for NAES is to investigate possible unplanned projects.

3.2 Communication with data supply partners (QAAD matrix score A1)

This relates to the need to maintain effective relationships with suppliers (through written agreements such as service level agreements or memoranda of understanding). This includes change management processes and the consideration of statistical needs when changes are being made to relevant administrative systems.

3.2.1 Barbour ABI communication with data providers

Barbour ABI research department has a dedicated team responsible for developing and maintaining close relationships with important contractors and information providers. Regular calls are made to gather further important information about contracts recently won and work currently out to tender. To ensure that the research team work to the highest possible standard, great emphasis is placed on data quality and the management team invest in telephone coaching and provide written scripts to help the team lead the conversation.

The scripts are adapted to include additional information as needed following customer feedback, industry rules and regulation changes. Important questions are always included, for example, tender type, cost, funding and consultants.

The team performance is monitored, measured and reviewed daily, weekly and monthly as required. Regular training and coaching is provided in-house and additional external training is provided by United Business Media (UBM) as a wider learning and development provider and the School of Built Environment at John Moores University, which delivers bespoke training for all employees.

It should be noted that although there are no formal agreements in place with their data suppliers, Barbour ABI has a strong history of information acquisition and relationship-building with the main stakeholders throughout the construction industry. This is achieved by face-to-face meetings with customers, relationships with main contractors, Barbour ABI tracking various channels such as the major tender portals (for example, the Official Journal of the European Union tenders, the central database for European public sector tender notices) as well as the planning authorities and county councils.

3.2.2 NAES communication with Barbour ABI

National Accounts and Economic Statistics (NAES) has described that is has a good working relationship with Barbour ABI with no barriers to communication. During the data supply, NAES can query certain projects or trends, to which Barbour ABI responds quickly with clarification and sometimes correction if the subsequent quality assurance identified an issue.

In the past, Barbour ABI has also been happy to provide a quote on the data for use in the statistical bulletin published by NAES. The team have a good understanding of the processes carried out by Barbour ABI due to the information they are provided with in briefings, which are accompanied with the data and regular visits to both offices for in-depth discussions. If there was a change to the methodology, NAES would expect to be notified by Barbour ABI if there were any implications for the data.

There is a legal contract in place since 1 January 2018 detailing NAES statement of requirements. The previous contract began in 2013, and data from Barbour ABI has been used from 2013 Quarter 2 (Apr to June) onwards. Barbour ABI is contractually obliged to supply initial new order estimates by the required period no later than 25 days after the quarter end. They are required to supply NAES with detailed management information covering data quality and commentary explaining significant movements in the data.

The strengths of this method are:

• strong communication links with data suppliers built up over the years

• regular telephone calls with data suppliers

• written scripts in place to ensure consistency and accuracy

• face-to-face meetings

• NAES has a good relationship with Barbour, which includes regular visits to both offices

• legal contract in place outlining NAES requirements to Barbour ABI

• system in place for NAES to query data

3.3 Quality assurance principles, standards and checks by data supplier (QAAD matrix score A1)

This relates to the validation checks and procedures undertaken by the data supplier, any process of audit of the operational system and any steps taken to determine the accuracy of the administrative data.

All new orders data are recorded through a secure Oracle database system. Department managers have access to management information monitoring more than 100 separate key performance indicators across every construction project report they create. Key performance indicators are used to ensure the important information required is targeted and measured. This includes tenders, timescales, cost, funding and materials. It also includes consultants and further down the line bidders, contractors and subcontractors.

To supply new orders data, records are extracted and all projects more than £50 million are fully re-researched plus a random 10% of all projects each quarter to ensure data accuracy and integrity. The team is supported by validation rules that include automated flags and warnings within their databases to seek out spikes and anomalies within data sectors and regions. A 10% tolerance change either way is in place on each sector and data is screened to identify significant movements against the previous quarter and against sector trends over previous years.

The web crawler previously mentioned in section 3.1.1 also has built-in validation checks. The system is run daily to produce logs for non-responders, duplication errors, high values and new applications.

The substantially experienced team analyses large volumes of construction project data, they carry out a number of quality assurance checks looking for error submissions, checking category codes are correctly asigned, council websites are operational and are fully crawled, and fortnightly volume checks.

All data is checked across disparate sources, for example, both companies and people working on a project. If there are disagreements the team will speak to design consultants, project managers and architects working on the same scheme to confirm the data.

Further checks are applied daily by importing the data into Excel. They spot check around 10% of projects valued at more than £100,000 and sense check large values, that categories are assigned correctly and revisit larger projects for consistency; they also check for outliers. This process is carried out by senior, experienced members of the department, including the data analysis team, overseen and signed off by the Group Content Director.

Additional quality checks are applied when needed; small teams can visit council sites twice a week if required to collect any missing information needed to approve applications into their system. The data are then manually updated; this allows them to keep an up-to-date record of past projects, associated people, previous relationships, contractors being used.

The strengths of this method are:

• comprehensive quality assurance checks in place

• managers monitor more than 100 key performance indicators

• fully research big projects

• further checks completed by senior, experienced members to ensure consistency overseen and signed off by the Group Content Director

However, Barbour ABI pays less attention to smaller projects, as it is difficult due to timeliness of outputs. Next steps for NAES is to investigate the possibility of improving capture of smaller projects.

3.4 Producers’ quality assurance investigations and documentation (QAAD matrix score A1)

This relates to the quality assurance conducted by the statistical producer, including corroboration against other data sources.

Upon receiving the data from Barbour ABI, National Accounts and Economic Statistics (NAES) transfer the data into a pre-prepared Excel spreadsheet where a macro file is run to convert the data into a format that can be run on our internal software system. This system incorporates the statistical processes of deflation and seasonal adjustment.

Once the data has been processed through the internal software, the output is checked for growth rates and the main trends. This is then checked against the original Excel file to ensure the new output has not changed because of the processing.

The data is supplied with detailed briefing information from Barbour ABI; using this and any trends picked up in the data, further questions are usually sent to Barbour ABI. They are quick to respond to queries, some of which result in further quality assurance work at their end. If the checks result in any revisions to the data, the data will be re-processed in the internal system.

The results from this process are then exported into another internal database where processing occurs in a table in the internal database’s software. This is then checked against a separate Excel spreadsheet that mimics the steps to ensure the internal processes have run consistently.

New orders data is a quarterly release within the monthly construction output publication. Once the report is written, it is checked by the construction team and sent for comments to senior members of staff. Prior to publication, the bulletin is uploaded to the preview site where the data can be further checked to ensure it is correct.

NAES can explain changes to the data because of the briefing provided by Barbour ABI along with the delivery of the data, which includes the details of all projects. Although these are not published for disclosure purposes, this helps NAES gain a deeper understanding of the industry. They are also able to gain further anecdotal evidence of changes from Barbour ABI if needed.

NAES have clear desk instructions in place; they are regularly reviewed and updated and in recent months various different members of staff have undertaken this process. In addition to this, Barbour ABI send the data file to at least two members of the team each round. Barbour ABI are also made aware if a specific member of staff should be sent the data to ensure receipt.

The strengths of this methods are:

• clear desk instructions

• comprehensive quality assurance checks in place including revisions

• detailed commentary provided by Barbour ABI

• data resupplied if errors are found

• final checks by senior managers

In investigating the administrative source for new orders in the construction industry, National Accounts and Economic Statistics (NAES) considers the main strengths of the data for its purposes to be:

• detailed knowledge and research of the subject area by Barbour ABI

• data tracked from order to completion

• 100% coverage of planned project applications

• comprehensive quality assurance checks carried out by Barbour ABI and NAES

• regular contact and briefings between construction suppliers, Barbour ABI and NAES

• detailed commentary provided by Barbour ABI

• legal contract in place detailing NAES requirements

We believe that the current limitations of this data source are:

• possible unplanned projects being missed

• less attention to smaller projects, due to timeliness pressure of outputs

In constantly seeking to improve our data sources we will be taking next steps to investigate these limitations and these will be communicated to users in the future quality assurance of administrative data (QAAD) report updates for this topic.

However, despite these slight limitations based on the low risk of quality concerns and small contribution that the new orders in construction statistics feed into the index of construction (less than 0.01%), NAES consider this data source to fulfil the requirements of an A1 assurance rating.