Quality assurance of administrative data report for the fishing industry

Background

The National Accounts and Economic Statistics (NAES) Group within the Office for National Statistics (ONS) receive quarterly data from the Marine Management Organisation (MMO) on landings for the fishing industry. These data form one source in the calculation of short-term economic output indicators, namely gross domestic product (GDP (O))for the UK.

This report outlines the process data take from initial collection through to the output of the release. It identifies potential risks in data quality and accuracy, as well as details of how those risks are mitigated.

This report forms the latest in a series of quality assurance of administrative data (QAAD) reports produced by NAES to investigate the administrative data sources we use in the production of short-term economic output indicators, as set out by the UK Statistics Authority (Administrative Data and Official Statistics – UK Statistics Authority). As such, this report specifically focuses on our administrative data use for the activities of the fishing industry (SIC 03.1) and does not aim to cover other aspects of the industry (for example, aquaculture – SIC 03.2). Separate industries where we use administrative data will be considered in other QAAD reports in the series.

Further information relating to quality and methodology for the short-term economic output indicators can be found in our Gross domestic product, preliminary estimate and Index of Services Quality and Methodology Information report.

Standard industrial classification (SIC) overview

The fishing industry covers all activities under UK Standard Industrial Classification 2007 (SIC 2007) division 03. This division includes marine fishing (SIC 03.1) and aquaculture (SIC 03.2).

According to the Inter-Departmental Business Register (IDBR¹) there were 3,930 enterprises classified under division 03 in March 2016 – this represents a slight increase of approximately 35 enterprises (0.9%) from the previous year (2015).

The majority of enterprises within division 03 are allocated to 0311 – marine fishing; and equate to 3,310 enterprises (84.2% of the total division). The annual turnover of 3,175 (80.8%) enterprises within division 03 was less than £250,000 whereas 205 (5.2%) enterprises had an annual turnover of £1 million or over. Within division 03, the vast majority of enterprises, (3550 (90.3%)) have fewer than five employees.

Notes for: Introduction

1. The Inter-Departmental Business Register (IDBR) is a comprehensive list of UK businesses that is used by government for statistical purposes. It provides the main sampling frame for business surveys carried out by both ONS and other government departments. It is also an important data source for analyses of business activity.

UK Statistics Authority QAAD toolkit

The assessment of our administrative data sources has been carried out in accordance with the UK Statistics Authority Administrative Data Quality Assurance toolkit.

Each administrative data source investigated has been evaluated according to the toolkits risk and profile matrix (Table 1) reflecting the level of risk to data quality and the public interest profile of the statistics.

The toolkit outlines four specific areas for assurance and the rest of this report will focus on each of these areas in turn. These are:

• operational context and administrative data collection
• communication with data supply partners
• quality assurance principles, standards and checks applied by data suppliers
• producer’s quality assurance investigations and documentation

In the assurance of our data source, we have chosen to give a separate risk and profile matrix scores (Table 1) for each of the four areas of assurance. This will allow us to focus our investigatory efforts on areas of particular risk or interest to our users. Scoring for the fishing and aquaculture data is shown below in Table 2.

Assessment and justification against the QAAD risk and profile matrix

The risk of quality concern and public interest profile has been set as “low” due to the small contribution that fishing statistics feed into gross domestic product (0.03%). As such, according to the QAAD risk and profile matrix a score of A1 (basic assurance) is deemed appropriate for this data source.

Please note, it is expected that Brexit will significantly impact the level of public interest on the fishing industry in the near future and this angle will be considered on the risk profile matrix in future editions of this report.

All scoring was carried out by National Accounts and Economic Statistics (NAES) Group based on the level of risk of the data and interest of our users. Results for each area of assurance for fishing are shown in Table 2 .If you feel that this report does not adequately provide this level of assurance or you have any other feedback, please contact us via email at stoi.development@ons.gov.uk.

3.1 Operational context and administrative data collection (QAAD matrix score: A1)

This relates to the need for statistical producers to gain an understanding of the environment and processes in which the administrative data are being compiled and the factors that might increase the risks to the quality of the administrative data.

The process detailed in section 3.1 is summarised graphically in Figure 1 of the annex.

3.1.1 Data providers

Statistics on fishing effort and landings from UK and foreign commercial fishing vessels landing into the UK are calculated using identical data collection and processing methods by various fisheries administrations around the UK. These administrations include:

• Marine Management Organisation (MMO)
• Marine Scotland (MS)
• Department of Agriculture and Rural Development (DARD(NI))
• Welsh Government (WG)
• Other departments in Jersey, Guernsey and the Isle of Man

The Marine Management Organisation (MMO) is an organisation that aims to license, regulate and plan marine activities in the seas around England so they are carried out in a sustainable way. The MMO assist the government in achieving its vision for clean, healthy, productive and biologically diverse oceans and seas.

The MMO holds information on all landings into England, Wales, Northern Ireland and the Isle of Man by UK vessels into the UK and UK vessels abroad, whilst Marine Scotland provides figures for landings into Scotland by all UK vessels and landings abroad by Scottish administered vessels.

In 2011, data for the Isle of Man became fully integrated within the data collection and recording systems operated by mainland fisheries administrations, which was previously unavailable. Landings for larger Channel Island vessels are also included on the data systems, however, a small amount of smaller vessels (less than 0.01%) are not included in the data. Jersey and Guernsey data are provided for quota management purposes, but are excluded from the monthly statistics publication.

3.1.2 Publications

The MMO produce three main National Statistics publications on fishery statistics that abide by the UK Statistics Authority Code of Practice. These publications include: Monthly UK sea fisheries statistics, Quota use statistics, and UK sea fisheries annual statistics.

Monthly UK sea fisheries statistics

A monthly report that compiles and presents statistics of the weights, landings and values of fish from UK vessels within the UK, UK vessels abroad, and by foreign vessels into the 20 major seaports across the UK. The MMO have stated that coverage is expected to reach 97% in the first publication and this figure increases to 99% coverage in the second month and 100% coverage is realised in the third month after release. The Monthly UK sea fisheries report also produces year-to-date figures, which presents statistics for the year running up to the month of publication. These figures are within 1% of final figures upon release.

MMO send National Accounts and Economic Statistics (NAES) Group quarterly extracts of data that are based on the monthly sea fisheries publication, which has undergone additional processes. The data have a 7 week time lag and are expected to be within 1% of final results.

Quota use statistics

These are the level of catches and landings of important quota species that are monitored throughout the year and compiled into a series of weekly and monthly spreadsheets. The Quota use statistics feed into the production of the Monthly UK sea fisheries statistics publication.

UK sea fisheries annual statistics

A publication produced every September that provides a general picture of the UK fishing industry based on the monthly landings publications.

Upon publication of the annual report, the MMO expect coverage of data to be 100% of all landings from UK vessels.

3.1.3 Data collection process

Vessels over 10 metres in overall length

Data collected on fishing effort by vessels over 10 metres come primarily from the fishing logbook, whilst other additional sources include landing declarations and sales notes. The fishing logbook captures data on fishing activity by individual vessels per trip, each day. This information includes catch, species, presentation and quantity of fish retained on board.

Supply of logbook data is mandated by legislation for all vessels over 10 metres and must be submitted within 48 hours of landing to UK authorities, including vessels that land into foreign ports. However, there are frequent instances where delays are experienced in receiving this documentation.

Vessels between 10 metres and 12 metres usually post their logbooks in paper form to port offices, whilst it is an expectation from UK fisheries administrations for vessels over 12 metres to submit their logbook data electronically. The phasing out of paper reporting for vessels over 12 metres was introduced vessel by vessel as new onboard systems were being installed, checked and thoroughly tested.

Since 2011, buyers and sellers with turnover of more than 200,000 Euros have been required to submit sales notes electronically. This is down from 400,000 in 2009.

Sales notes are required in respect of first sales of fish and fishery products and must also be submitted within 48 hours by the registered buyer of the fish (except at designated auction centres where the registered seller assumes responsibility); this reduces to 24 hours if they are submitted electronically.

Data collection for vessels over 10 metres in length aims to achieve full 100% coverage of activity in this sector of fleet as they are legally required to submit complete details of their activity every time they go out to sea.

Vessels 10 metres and under in overall length

For vessels under 10 metres long, there is no statutory requirement by the EU or national legislation for fishermen to declare their catches. Historically, this information has been comprised voluntarily through log sheets, landing declarations, sales notes, assessments of landings collected from market sources and by correspondents located in the ports.

During 2005 and 2006, UK fishery administrations introduced a system of restrictive licensing for activity targeted at shellfish. As part of this system, new requirements were introduced for fishermen with vessels under 10 metres in length to complete diaries of their daily activity, which are submitted on a monthly basis. Summaries of this information are used in Northern Ireland, and other parts of the UK use it alongside other data, however, the main source of activity data in the rest of the UK is sales notes data. This ensures consistency within activity data for other fishing activities that are taken from sales notes data.

For vessels 10 metres or under, data for landings are reported through sales notes from the buyer of the fish, whereas it is now a legal requirement for the buyer to provide sales notes for commercially sold fish or the data have been provided voluntarily. MMO still expect coverage to be 100%.

3.1.4 Data transfer process

A UK Electronic Reporting Systems (ERS) hub has been set-up by the MMO and is maintained by the Centre for Environment Fisheries and Aquaculture Science (CEFAS) in order to collect, process, and store the data from logbooks, sales notes and landing declarations provided by the vessels. Information supplied by fishermen in paper form is keyed into networked computers by government officials at the port offices.

The complete and validated fishing records are then transmitted to the central computer systems in accordance to Council Regulation (EC) No. 1224/2009 (the “Control Regulation”) and The Sea Fisheries Act 1967 (later expanded in the Sea Fish Conservation Act 1992). The central computer system is where final checks are carried out on the data before they are reflected in the main landings database (iFish) for analysis.

Once accepted as valid and complete, activity and landings data for the UK are compiled in a central database containing vital information from the systems managed by MMO and Marine Scotland.

The MMO are working on a system to replace iFish that will contain more up-to-date information, all landings at seas and reduce time lags between catching the fish and receiving the data.

3.1.5 Potential vulnerabilities

The reliability of the data collected is dependent on the information provided by fishermen.

The MMO state that the skipper of the vessel has an 8% to15% tolerance level when estimating the weight of the fish (dependent on the species) as it is typical for landings to not equal total caches. This is because of accidental discarding, breaches of quota limits, illegal fishing and catching damaged, less profitable or undersized fish, therefore causing potential vulnerability to estimates. As a result, the EU landing obligation was introduced in 2015 with the aim to reduce and eventually eliminate the discarding of marketable fish by 2019.

Overall, although there is a possibility of illegal fishing negatively influencing the amount of landings reported in the data, the electronic data collection process combined with very high levels of coverage shows that there is a fluid process in place, which allows a very good representation of the fishing industry. The strengths of this method are:

• identical data collection and processing methods by various Fisheries Administrations involved
• MMO receive an initial 97% of coverage in their first publication, which eventually reaches 100% coverage by the third month
• there are legal requirements to provide logbook data for vessels over 10 metres in length
• logbook data for vessels over 12 metres are provided through an electronic system
• extract received by NAES is within 1% of final results
• legislation also unpins the transfer of data from the ERS to the MMO

The weaknesses of this method are that reliability is dependent on the information provided by fishermen and illegal fishing causes an element of uncertainty to the data. Going forward, the MMO hope to maintain this level of knowledge of the area.

3.2 Communication with data supply partners (QAAD matrix score: A1)

The process detailed in section 3.2 is summarised graphically in Figure 2 of the annex.

This relates to the need to maintain effective relationships with suppliers (through written agreements such as service level agreements or memoranda of understanding), which include change management processes and the consideration of statistical needs when changes are being made to relevant administrative systems.

3.2.1 The Marine Management Organisation and their suppliers

The Marine Management Organisation (MMO) holds a working relationship with the fishing vessels through their various fisheries administrations around the coast. It was stated that port offices have the ability to co-operate with vessels without a need to communicate, as skippers would either post their logbooks or send them electronically. Upon finding a discrepancy within the data, the MMO would alert local port offices to contact the skipper via telephone or face-to-face meetings to discuss the issues and ask for a resubmission.

The MMO have described their co-operation with Marine Scotland as very good with no barriers to communication. The MMO submits data from the Fisheries Activity Database (FAD) to iFish, whilst Marine Scotland submits Fisheries Information Network (FIN) data both on a daily basis. If any discrepancies are found within the datasets, it was ensured that instant contact would be made between the MMO and Marine Scotland to resolve any issues.

3.2.2 The Marine Management Organisation and National Accounts Economic Statistics Group

The MMO is the producer of fishing statistics that the National Accounts Economic Statistics (NAES) Group feed into the Index of Services (IoS). Previously, there had been a service level agreement (SLA) between the two organisations for the transfer of data, however, this was not renewed in 2013 due to the mutual relationship between the MMO and NAES where data are provided through goodwill. This decision was also taken because the logistics of implementing and actively managing such formal arrangements are considered both prohibitive and unnecessary for this data source for the two parties, and there is no legal obligation for the MMO to share fishing data with NAES.

The MMO send NAES quarterly worksheets of data that includes accompanying commentary within the document that describes the data and if necessary, highlights any concerns or irregularities. Should an unidentified irregularity emerge, NAES holds up-to-date contacts and will contact the MMO when necessary, in which both parties are willing to communicate accordingly. No communication has been made regarding concerns over the data and there has yet to be any queries or questions arising from the quality of the statistics provided. Overall, this indicates a high level of confidence from both the user and producer regarding the data and that current methods of delivery satisfy essential quality criteria.

The strengths of this method are:

• MMO have communication processes in place to contact skippers to explain or resubmit discrepancies in their data
• Up to date contacts held between NAES and MMO
• Commentary is made from the MMO regarding the data provided

The weaknesses of this method are that it involves an out-of-date service level agreement and no formal betweens occur between the MMO and NAES. Next steps involve arranging a formal annual meeting between NAES and MMO.

3.3 Quality assurance principles, standards and checks by data supplier (QAAD matrix score: A1)

The process detailed in section 3.3 is summarised graphically in Figure 3 of the annex. This relates to the validation checks and procedures undertaken by the data supplier, any process of audit of the operational system and any steps taken to determine the accuracy of the administrative data.

EU legislation lays out a set of requirements that each UK fisheries administration must operate under to ensure the consistent treatment of data within the collection processes and to ensure that UK data are consistent with EU requirements for completeness and accuracy.

The vessel operators

The Electronics Reporting System (ERS) operated on the fishing vessels contains initial amount of validation checks, such as checking that the dates and units entered are entered correctly, to ensure the information is valid before submission to the ERS hub.

The Electronic Reporting System hub

The logbooks and landing declarations are received from the vessels in the ERS hub, in which paper forms are manually logged into the hub by government staff. The hub is maintained by database managers and reviewed by a team of statisticians on a regular basis.

In the hub there are a series of business rules, which includes the checking of individual submissions, to ensure their accuracy and completeness. When failing to comply with the business rules, failure messages are generated and sent to vessel operators, in which the skippers would be required to resubmit corrected data. In the scenario where a discrepancy has been identified, the appropriate corrections are made to the data by the skipper. Discrepancies that do not lie within the legislator requirements set out by the EU are investigated with the appropriate actions taken, which can range from a simple verbal instruction of how to correctly report information, up to a full prosecution of the master and/or owners of the vessels concerned if a serious offence has been committed.

The complete fishing records from the hub are then transmitted to the central computer systems where further checks are carried out on the data before they are reflected in the main landings database. These are more detailed sense checks that make sure of internal consistencies within the data, for example, making sure the date of the landings are after the date of departure for the trip.

In addition to the validation processes within the hub, the information reported by fishermen is run through cross checks with other sources of information regarding landings to ensure consistency and also to ensure producer organisations (PO) will not be affected, as errors, such as overstatements of catch, will prevent PO’s from fishing as their quota would have been reached.

These checks include:

• sending out a full breakdown of their quota statistics to other fishing organisations such as Marine Scotland and Cornwall IFCA Committee to make sure the information is correct
• checking information in logbooks against data reported by auction centres and fish merchants for fish sold; in particular, this is relevant for all vessels over 10 metres, as supplying logbook data is a legal requirement for catches of all species
• comparisons of information on sightings of vessels at sea by fisheries administrations

Quality assurance of evidence process

The MMO has a quality assurance process when undertaking their series of quality checks to assure confidence in a piece of evidence. The quality assurance process has main criteria evidence has to fulfil to obtain full approval for the data. The MMO seek written confirmation from main suppliers, external reviewers and assurances from corporate level that their processes and procedures are robust.

For a full breakdown of the criteria, please visit the MMO’s Quality assurance of evidence process.

The strengths of this method are:

• EU set of requirements that each UK fisheries administration must operate under
• Electronic Reporting System initial validation checks
• detailed sense checks that make sure of internal consistencies within the data
• cross checks alongside other data sources
• data resupplied if errors are found and discrepancies in data investigated
• the data have National Statistics badge status, which abides by the UK Code of Practice for Official Statistics
• data undergo a quality assurance evidence process

3.4 Producers quality assurance investigations and documentation (QAAD matrix score: A1)

The process detailed in section 3.4 is summarised graphically in Figure 4 of the annex.

This relates to the quality assurance conducted by the statistical producer, including corroboration against other data sources.

Upon receiving the data from the Marine Management Organisation (MMO), National Accounts and Economic Statistics (NAES) Group transfer the data into a pre-prepared Excel spreadsheet where seasonal adjustments and deflators are applied. NAES then proceed to apply quality assurance checks to ensure the data meets the statistical producer’s standards.

These quality assurance checks include:

• identifying outliers by graphing the data – upon finding an outlier, the necessary manual adjustments are made after seeking approval from MMO; this approach has never had to be implemented as there has not been data queries

• comparisons with a 5-year forecast of previous data to test if the data align with expectations and conform to historic trends; upon the failure to meet historic trends, NAES will contact the MMO to clear up any discrepancies

• sense checks from a senior manager and statisticians

NAES maintain clear desk instructions to allow team members to work on the analytical spreadsheets when necessary, and are currently in the process of developing a new checking tool which will provide more thorough quality assurance checks alongside the already extensive checks conducted by the MMO and fishing vessels. The new checking tool is estimated to be ready within the next 6 months.

The strengths of this method are that there are established quality assurance checks in place, as well as clear desk instructions. A new checking tool is also in development.

The overall process data take from initial collection through to the output is summarised graphically in Figure 5 of the annex.

In investigating the administrative source for the fishing industry, National Accounts and Economic Statistics (NAES) Group considers the main strengths of the data for our purpose to be:

• a dataset collected under legislation that reaches coverage levels initially of 97% and later 100%
• electronic submissions improve timeliness and accuracy of vessels over 12 metres in length
• data audited automatically by internal systems, reviewed and validated by statisticians and cross checked with other sources
• commentary made from the Marine Management Organisation (MMO) regarding the data provided
• National Statistics badge status, which abides by the UK Statistics Authority’s Code of Practice for Official Statistics

We believe the current limitation of this data source is that there are no direct or formal meetings between NAES and the MMO on a regular basis.

In constantly seeking to improve our data sources, we will be taking the next steps to address these limitations and these will be communicated to users in the future quality assurance of administrative data (QAAD) report updates for this topic.

However, despite these slight limitations, based on the low risk of quality concerns and small contribution the fishing industry feeds into gross domestic product (0.03%); NAES considers this data source to fulfil all requirements of an A1 assurance rating.