Quality assurance of administrative data used in unit trusts and investment trusts statistics

1.1 Background

National Accounts and Economic Statistics (NAES) within Office for National Statistics (ONS) collects data from the Investment Association (IA) and the Association of Investment Companies (AIC) for the activities of unit trust and investment trust funds. These two data sources form two separate sources in the calculation of short-term economic output indicators, namely gross domestic product (GDP(O)) for the UK.

This report outlines the process data take from initial collection through to the output of the release. It identifies potential risks in data quality and accuracy as well as details of how those risks are mitigated.

This report forms the latest in a series of quality assurance of administrative data (QAAD) reports produced by Economic Statistics Group to investigate the administrative data sources we use in the production of short-term economic output indicators as set out by the UK Statistics Authority. As such, this report specifically focuses on our administrative data use for investment and unit trusts, that is, Standard Industrial Classification (SIC) group 64.3. It does not aim to cover other aspects of SIC division 64 such as groups 64.1, 64.2 and 64.9 as these are collected by our Financial Corporations team who work closely with the Bank of England. The Financial Corporations team are the gateway for all the Bank of England-sourced data used by ONS. Separate industries where we apply administrative data will be considered in other QAAD reports in the series.

Further information relating to quality and methodology for the short-term economic output indicators can be found in our Gross domestic product, preliminary estimate report.

1.2 Standard Industrial Classification (SIC) overview

The activities of financial services cover all activities under UK SIC 2007 division 64. This includes activities of obtaining and redistributing funds other than for the purpose of insurance or pension funding, or compulsory social security.

Based on the UK Standard Industrial Classification (2007) the industry is classified to four groups:

• 64.1 – monetary intermediation

• 64.2 – activities of holding companies

• 64.3 – trusts, funds, and similar financial entities

• 64.9 – other financial service activities, except insurance and pension funding

According to the Inter-Department Business Register (IDBR)¹ there were 15,585 enterprises classified under division 64 in March 2017. This is an increase of approximately 1,275 enterprises (9.0%) from the previous year (March 2016).

Within division 64, there were 13,715 enterprises with fewer than five employees (88.0% of the division), compared with just 590 enterprises with 20 or more employees (3.7% of the division).

Within group 64.3, trusts, funds, and similar financial entities there were 1,455 total enterprises of which 1,160 enterprises have fewer than five employees (7.4% of the division), compared with just 45 enterprises with 20 or more employees (0.2% of the division).

Most of the enterprises within division 64 were allocated to group 64.2, activities of holding companies, which equates to 7,340 enterprises (47.0% of the total division).

Notes for: Introduction

1. The Inter-Departmental Business Register (IDBR) is a comprehensive list of UK businesses that is used by government for statistical purposes. It provides the main sampling frame for business surveys carried out by both ONS and other government departments. It is also an important data source for analyses of business activity.

2.1 UK Statistics Authority QAAD Toolkit

The assessment of our administrative data sources has been carried out in accordance with the UK Statistics Authority Quality Assurance of Administrative Data Toolkit.

Each administrative data source investigated has been evaluated according to the toolkit’s risk and profile matrix (Table 1) reflecting the level of risk to data quality and the public interest profile of the statistics.

The toolkit outlines four specific areas for assurance and the rest of this report will focus on these areas in turn.

These are:

• operational context and administrative data collection

• communication with data supply partners

• quality assurance principles, standards and checks applied by data suppliers

• producer’s quality assurance investigations and documentation

In the assurance of our data source, we have chosen to give a separate risk and profile matrix score (Table 1) for each of the four areas of assurance. This will allow us to focus our investigatory efforts on areas of particular risk or interest to our users (Table 2).

2.2 Assessment and justification against the QAAD risk and profile matrix

Investment Association – unit trust funds

The risk of quality concern and public interest has been set as “low” due to the small contribution that the investment trust statistics feed into gross domestic product (0.05%). As such, a score of A1 is deemed appropriate for this data source.

Association of Investments Company – investment trust funds

The risk of quality concern and public interest has been set as “low” due to the small contribution that the investment trust statistics feed into gross domestic product (0.01%). As such, a score of A1 is deemed appropriate for this data source.

All scoring was carried out by National Accounts and Economic Statistics (NAES) based on the level of risk of the data and interest of our users. Results for each area of assurance for investment and unit trusts are shown in Table 2. If you feel that this report does not adequately provide this level of assurance or you have any other feedback, please contact stoi.development@ons.gov.uk with your concerns.

3.1 Operational context and administrative data collection for unit trusts (QAAD matrix score A1)

This relates to the need for statistical producers to gain an understanding of the environment and processes in which the administrative data are being complied and the factors that might increase the risks to the quality of the administrative data.

3.1.1 Investment Association data collection process

The Investment Association (IA) is the data provider for National Accounts and Economic Statistics (NAES). The IA is the trade body that represents UK investment managers. Their 240 members collectively manage over £7.7 trillion on behalf of clients in the UK and around the world. They promote UK investment management, which is the largest industry of its kind in Europe and the second-largest in the world. They engage actively with policy-makers and other stakeholders to ensure that their industry remains among the world’s most competitive.

The IA has been collecting monthly data from its members for all authorised and recognised funds since 1992. At the outset members upload spreadsheets into a secure statistics database. It is a requirement for members to provide monthly data under the terms of the IA’s articles of association (PDF, 317KB), which all members sign when they join.

Safeguards are in place, such as validation checks, to minimise the sources of uncertainty bias or error. Further information on safeguards is detailed in Section 3.3.1. The members have a good track record for timeliness of supply and the IA estimate that their data coverage is more than 90% of the UK retail fund market at the point of collection. They estimate that their members manage more than 90% of the UK asset management industry by comparing their members’ funds data with Morningstar data.

IA data are used widely by many stakeholders across the asset management community. Scottish Widows, regulators, Bank of England and government departments such as the Department for Business, Energy and Industrial Strategy are all examples of stakeholders. They make a significant amount of data publicly available to all users, of which NAES is also a user. In the production of gross domestic product (GDP), NAES download the funds under management data table, allowing sufficient time for publication within the preliminary estimate of the gross domestic product output measure (GDP(O)).

Revisions to the data are published with a note confirming small revisions to the figures may have been made since the previous release reflecting additional information received.

Results from the latest IA asset management 2017 to 2018 survey measure the size of the industry as well as looking at the main drivers and trends behind its evolution. The survey covers the structure of the industry, ownership of assets managed, asset allocation, profitability, the prevalence of soft commissions and important trends.

3.1.2 Potential challenges measuring the scale of the wider industry

In the latest asset management survey report (PDF, 8.5MB), the IA reported total assets managed in the UK by their members as £7.7 trillion of client money as at December 2017.

They also reported, while the IA’s members represent most of the UK asset management industry in asset terms (85%), a significant number of firms contributing to the industry activity lie outside the IA membership and are not collected. These can be broadly categorised into the following groups:

• hedge funds – £320 billion

• private equity funds – £270 billion

• UK commercial property management – £540 billion

• discretionary private client management – £550 billion

• UK commercial property management – £540 billion

• firms who are not members of the IA for reasons not stated previously (no consistent third-party data available to estimate amount)

3.2 Operational context and administrative data collection for investment trusts (QAAD matrix score A1)

3.2.1 Association of Investment Companies

The Association of Investment Companies (AIC) is the data provider for National Accounts and Economic Statistics (NAES). The AIC is the trade body of the investment trust industry and was formed in 1932.

It speaks on behalf of investment trusts in five main ways:

• lobbying policy-makers on behalf of the investment trust sector to provide the best regulatory, tax and legislative environment for investment trusts

• working with member companies to promote and maintain high standards within the industry

• raising awareness and communicating the benefits of investment trusts to consumers

• providing statistical information to existing and prospective shareholders and to the industry

• providing education, information and training on investment trusts to financial advisers and stockbrokers

However, AIC outsources the data collection to Morningstar. Morningstar has the most complete database of UK investment trust data available to the market. It collects data from AIC members where membership is open to investment companies. To be eligible, a company must be closed-ended (fixed number of shares), invest in a diversified portfolio of assets and have its shares traded on a stock exchange.

Morningstar collects 100% coverage of all AIC membership data. AIC members report the latest monthly data via the Monthly Information Release (MIR). This is a monthly data collection process by which data is submitted using a secure file transfer by email into a secure web-based interface.

AIC and Morningstar encourage members to submit data in line with industry standards and best practices. There is no legal obligation to do so, however, members do adhere to these rules and are co-operative, but a small risk of inconsistent methodologies exists. Morningstar mitigates this risk by calculating derived statistics such as returns, portfolio exposure from the raw data (unclean) sourced from asset managers as opposed to using the asset managers’ calculated statistics.

More details on number of funds or assets under management can be found in the Management group rankings (XLS, 52KB) file on the AIC industry overview page.

Morningstar has a change management process in place to keep pace with industry standard, best practices and regulatory changes to the industry. Any changes are reviewed and put through a robust change in process and applied consistently across the applicable fund universe. All changes by Morningstar are clearly described to the AIC members. In addition, if AIC members instigate the change, Morningstar will work with them to accommodate such changes wherever possible.

In 2018, Morningstar reported more than 98% of investment trust companies (for example, Blackrock investment UK) in the UK industry consistently sent portfolio information directly to them on a monthly and quarterly basis. To expedite data collection and processing, their exclusive data management system assures there is virtually no backlog for uploading data into their systems.

Strengths:

• requirement for IA members to supply monthly data under the terms article of association
• IA estimate market coverage is more than 90% at the point of collection
• AIC data source is collected monthly by Morningstar through a secure web base
• Morningstar have a change management process in place
• Morningstar collects 100% coverage of all AIC membership data

Weaknesses:

• IA only receives data from clients that are subscribed to IA membership

• IA do not capture hedge funds, private equity funds, commercial property management, private client management and firms who are not members of the IA

• IA reported a significant number of firms contributing to the unit trusts industry lie outside their membership

• AIC only receives data from clients that are subscribed to AIC membership

Next steps:

NAES to consider the impact of investigating other data sources such as Morningstar who reported in 2018 they collect more than 98% of investment trust data for the UK industry.

3.3 Communication with data supply partners for unit trusts (QAAD matrix score A1)

This relates to the need to maintain effective relationships with suppliers (through written agreements such as service level agreements or memoranda of understanding) which include change management processes and the consideration of statistical needs when changes are being made to relevant administrative systems.

3.3.1 Investment Association communication with their members

The Investment Association (IA) publishes a guide on joining the association including a list of membership benefits, such as:

• monthly interaction with data suppliers’ analysis team, providing assistance when needed and dealing with collection or data queries

• privileged access to the members’ only section of the IA website, which includes knowledge, documentation, industry standards and statistics

• access to expert staff on hand to answer specific technical queries and offer advice on important initiatives and policy issues such as Brexit

• opportunities to be represented on 70 committees, working groups as well as member meetings, briefings, forums and discussion groups addressing the latest developments and action planning

IA has a wide range of fund operations documentation on its website, which sets out standardisation and efficiency within the industry. In addition, they publish a series of guidelines and processing templates in which they set out member expectations on issues such as corporate governance, share capital management and other issues relating to capital markets.

It has a large established team on hand providing guidance through telephone and secure email. In addition, they provide and update a frequently asked questions page readily assessable on their propriety internet site. Amongst the staff are experts on regulation, tax, operations, investment and market structure issues, corporate governance, communication, corporate affairs, research and statistics, training and education.

A committee is in place to provide guidance to members on the appropriate classification of funds to keep abreast of market developments. The committee is made up of representatives of a cross-section of member companies and the major data suppliers. For further information please read the terms of reference.

The fund management company representatives on the committee provide a wide range of industry expertise, with the ability to address complex issues that may arise. The support in place allows members to contact the IA by email and telephone and in addition, access to a web page covering industry guidance.

3.4 Communication with data supply partners for investment trusts (QAAD matrix score A1)

3.4.1 Morningstar communication with Association of Investment Companies members

Morningstar digitally connect with the Association of Investment Companies (AIC) members through:

• relevant articles on the web
• videos
• reports
• web tools
• dedicated customer team
• technical support team

Their dedicated teams assist members throughout the process of selecting, planning and integrated web tools solutions. The level of involvement streamlines development time and ensures that each member gets a solution that meets business goals and technical requirements. Ongoing communication allows members to maximise the value of their investment to meet their needs.

In addition, the technical support team are product specialists who understand the issues and can meet members’ needs. They can rely on the team to answer data questions on many financial products.

Morningstar provide AIC with a dedicated relationship manager and provide:

• face-to-face meeting every two weeks

• telephone contact two to three times a week for general business

• support mechanism to answer data queries

Morningstar plan official quarterly service reviews where they meet members of investment trusts (including brokers and asset managers) to discuss all data and product issues. They meet the Statistics Committee (industry experts) twice a year and the Asset Managers Committee four times a year to discuss trends and methodologies within the industry.

3.4.2 Association of Investment Companies’ communication with their members

The Association of Investment Companies (AIC) offers its members privileged access to a selection of web pages giving a wide range of information such as a policy and technical library, updates to thousands of investors, financial advisers and media. Members can access online all their guidance on important technical and commercial issues with an opportunity to discuss further.

Members are invited to attend regular free conferences, seminars and roundtables on the latest issues affecting investment companies. In addition, they are invited to participate in committees and forums where they can learn more about current issues and provide input to their work.

3.4.3 NAES communication with Investment Association and Association of Investment Companies

National Accounts and Economic Statistics (NAES) downloads the latest data from the Investment Association (IA) and the Association of Investment Companies (AIC) websites. Supporting documentation is available to view on the web page however, NAES does not have a formal service level agreement with IA and AIC.

The data used for the output measure of gross domestic product (GDP(O)) are publicly available on the website and the logistics of implementing and actively managing formal arrangements are considered both prohibitive and unnecessary for these data, considering the low weight they comprise.

No formal contact is in place, however, in the past whenever there has been a query about any of the data there has been a contact email address accompanying each dataset and both companies have been forthcoming and helpful in providing further information. Further contact has been established through the writing of this report and we intend to maintain this level of communication.

Strengths:

• IA has a wide range of information available for their members to access online
• IA offer additional help for members to get answers to complex queries
• Morningstar has a dedicated team to assist data providers
• Morningstar provides AIC with a dedicated relationship manager
• AIC offers members privileged access to a members only section on their website with a wide range of help and information

Weaknesses:

• No direct, regular meetings or contact between NAES, IA and AIC
• Changes to methodologies and policies are not communicated directly to NAES

Next steps:

• NAES to maintain the newly-established communication links developed during the writing of this report
• NAES would benefit from being informed in advance of methodology and policy changes.

3.5 Quality assurance principles, standards and checks by data supplier for unit trusts (QAAD matrix score A1)

This relates to the validation checks and procedures undertaken by the data supplier, any process of audit of the operational system and any steps taken to determine the accuracy of the administrative data.

3.5.1 Investment Association quality assurance processes

The Investment Association (IA) carries out many checks on the consistency of information reported by fund companies and sales through a reconciliation of monthly fund figures with net sales and third-party data from Morningstar relating to fund performance. Where necessary queries are raised directly with companies to resolve any issues.

System checks

These checks are applied through the statistical database, which enables members to upload data files. The system will carry out integrity checks on the file to ensure accuracy and consistency of data stored by checking the structure of the data is correct. The price tolerance levels for errors are flagged in the database. The two common issues highlighted from these checks are reported at share price requiring further contact with members to confirm validity.

Transaction checks

The system checks for large transactions, which report values over £20 million. The system will highlight this by flagging errors for further investigation. Out of the ordinary transactions are checked with members to ascertain why the funds have experienced a large increase or decrease. If necessary further checks are made directly with fund managers and agents for further clarification.

Funds with over £20 billion in assets are also flagged, though in practice IA say they know which funds are genuine.

Performance checks

Performance checks are calculated by the system using the previous month’s data and adding in the current month’s net sales to get an implied performance of the fund for the current month. The system will then compare this figure with the reported fund performance third-party data they source from Morningstar.

For example, if fund X had shares of £10,000,000 in May and net sales of £5,000,000 in June, the implied fund at the end of June would be £15,000,000. If the performance of the fund was 5% in June according to Morningstar (third-party data) the system would expect the fund at the end of June to be £15,000,000 multiplied by 1.05 equalling £15,750,000. The system checks all transactions to see if there are any suspicious values, including values that appear to be too large or too small, and highlights as being invalid.

The IA prides itself on having good quality data and ensures checks are carried out by more than one person to minimise errors. This involves further quality checks at an aggregated industry level prior to publication of final figures.

3.6 Quality assurance principles, standards and checks by data supplier for investment trusts (QAAD matrix score A1)

3.6.1 Morningstar quality assurance processes

Quality investment data are the core of Morningstar’s business. They use propriety screens and quality assurance tests at each stage from collection to delivery to provide the cleanest information to their clients.

Morningstar data analysts use a database to view, edit and update data. Quality assurance tests are performed at each level including raw and calculated data production and output. The system flags irregularities such as incorrect formatting, outdated or incomplete data and performance discrepancies so that issues can be resolved before the data are delivered to the Association of Investment Companies (AIC).

Morningstar is continuously striving to provide the highest quality data. It regularly benchmarks its data against other leading data sources such as the Financial Times Stock Exchange All-share Index. In addition, annual institutional client surveys provide direct insight into how data are used and where changes can improve the value of data feeds for clients.

Morningstar has a formal quality programme, which it has developed specifically to meet the needs of their company and clients. The goal of their programme is to drive client-centricity, operational excellence and continuous improvements across Morningstar.

3.6.2 Association of Investment Companies’ quality assurance processes

After receiving the initial data from Morningstar, the statistics team apply further quality assurance checks to ensure data accuracy prior to publication. These are:

• sense checks – data are checked to ensure they are in line with previous trends
• percentage differences – data are checked for large differences
• monitor companies’ announcements for unexpected changes
• compare performance with Morningstar data
• data resupplied if errors are found

All data changes are confirmed with Morningstar and AIC members prior to publication. AIC tells its users about instances where data have not been provided to them, in a published note informing users that estimated data are used and which funds are most affected.

Strengths:

• IA carry out comprehensive quality assurance checks by more than one person to minimise errors
• Morningstar has a formal quality programme developed for the needs of their company and clients
• Morningstar has a quality assurance system in place to monitor and track errors
• Morningstar regularly benchmark data with other sources
• AIC carry out further quality checks to ensure data accuracy prior to publication

Weaknesses:

• none

Next steps:

• none

3.7 Producers’ quality assurance investigations and documentation for unit trusts and investment trusts (QAAD matrix score A1)

This relates to the quality assurance conducted by the statistical producer, including corroboration against other data sources.

National Accounts and Economic Statistics (NAES) receives automated email alerts allowing them sufficient time to download the latest data from the Investment Association (IA) and the Association of Investment Companies (AIC) websites. This ensures NAES branch can investigate, analyse and query the data in time for publication within the preliminary estimate of gross domestic product output approach (GDP(O)).

NAES has validation checks in place built into Excel spreadsheets to carry out summary checks including raw data (unclean) and growth rates. It also investigates historical trends including revisions. Data are then loaded into internal systems, which use one calculation to aggregate for the whole of Standard Industrial Classification (SIC) division 64.

The final output is quality assured by two senior managers within the team. There are clear desk instructions in place; these are reviewed and updated on a regular basis. The systems are run each month for final output.

User engagement is continual and the feedback tends to relate to the overall impact of the statistics rather than individual data sources used. To date, no specific feedback on the use of the investment and unit trusts data has been provided.

While writing this report, a system error resulting in the incorrect decimal point being used for the AIC upload was identified, with £ million and not £ billion feeding into the results. However, the AIC data source is being used to create a baseline index resulting in no change to the growth rates. Calculating the index using figures in £ million or £ billion will give the same result.

Strengths:

• established quality assurance checks in place
• data are compared by looking at historical trends including revisions
• desk instructions regularly reviewed
• final output quality assured by two senior managers

Weaknesses:

• Excel-based system used for validation checks

Next steps:

• NAES to improve automated procedures to populate internal systems with updates

In investigating the administrative data sources for the activities of investment and unit trusts, National Accounts and Economic Statistics (NAES) consider the main strengths of the data sources for our purpose to be:

• the Investment Association has been collecting monthly data from its members for funds since 1992

• Morningstar collects 100% coverage of all Association of Investment Companies membership data

• comprehensive quality assurance principles in place by the Investment Association and Association of Investment Companies

• the Investment Association has a wide range of information available for their members to access online

• Morningstar has a dedicated team to assist Association of Investment Companies data providers

• Morningstar provides Association of Investment Companies with a dedicated relationship manager

We believe current limitations of the data sources are:

• data only taken from clients that are subscribed to membership

• a large number of firms contributing to the industry lie outside the Investment Association membership

• no direct, formal, regular contact between NAES, the Investment Association and the Association of Investment Companies

• Excel-based system used for validation checks

In constantly seeking to improve our data sources we will be taking next steps to investigate these limitations and these will be communicated to users in the future quality assurance of administrative data (QAAD) report updates for this topic.

However, despite these slight limitations based on the low risk of quality concerns and small contribution that the investment trusts (0.01%) and unit trusts (0.05%) feed into gross domestic product, NAES consider this data source to fulfil the requirements of an A1 assurance rating.