Table of contents
1. Introduction
The UK’s Foreign Direct Investment (FDI) survey has undergone a National Statistics Quality Review (NSQR) as part of the reinstatement of the rolling programme of NSQRs in December 2012. The review was started in January 2015 and reported its findings in July 2016. The review was led by Gareth James, a current Office for National Statistics (ONS) employee responsible for the Sample Design and Estimation Centre within our Methodology group and Stuart Brown, an international expert and former ONS employee who previously headed our Balance of Payments (BOP) team.
The ONS-commissioned review had a broad scope, which covered many aspects of the FDI survey and the International Transactions branch responsible for the statistics; from questionnaire and sample design through to data collection and results production. The review provides us with a wide range of recommendations and suggestions for improving current processes and, ultimately, the overall quality of FDI estimates. These recommendations will be used as part of our ongoing commitment to improving the way we measure and disseminate official statistics.
This article provides a formal public response to the NSQR and provides details on which measures have been adopted to address some of the recommendations raised in the NSQR, whilst also outlining the steps being taken to address the remaining recommendations.
Back to table of contents2. ONS response
2.1 Benefits of the NSQR
The broad scope of the National Statistics Quality Review (NSQR) allowed the reviewers to consider a wide range of aspects in relation to how the Foreign Direct Investment (FDI) survey is conducted. Having an experienced methodologist and former Balance of Payments (BOP) compiler conduct the review offered a detailed insight into the production of FDI statistics.
The NSQR followed a programme of work that had already been completed to ensure that the FDI survey was compliant with the latest international standards, as defined in the Balance of Payments manual version 6 (BPM6). The work to ensure compliance with BPM6 began in 2009, when the International Transactions development team was formed comprising staff with technical and methodological skills to ensure successful implementation of BPM6 requirements. The project drew to a close shortly before the revised FDI questionnaires were issued to respondents for the reference period of 2013. The timing of the NSQR enables us to further review the FDI survey and processes and has identified further enhancements that can be made to improve the overall quality of the estimates.
2.2 Overview response
The NSQR makes clear that measuring FDI statistics is a complicated process; however, despite these complexities, the report acknowledges that official FDI estimates produced by the Office for National Statistics (ONS) are fit for purpose.
We are constantly seeking to improve the way official statistics are measured and communicated to users; therefore, we welcome the thorough review of how FDI statistics are measured and the subsequent recommendations made. In particular, we are keen to take forward improvements surrounding FDI sampling methods and applied methodology, as improvements in each these areas will further enhance the quality of outputs and increase user confidence.
We agree with the recommendation made that the FDI teams would benefit from additional resource. We believe that the best way to improve processes and quality is through increasing the analytical capability of the teams rather than headcount.
The International Transactions branch responsible for producing FDI statistics has recruited statisticians in recent years to develop production systems and methods that have automated a number of the manual processes, which has provided the results teams with additional valuable time to focus efforts on quality assurance. Economics capability has also been significantly increased within the branch, with economists filling roles including Assistant Deputy Director, Branch Head and economics analyst roles. The increased analytical capability has also allowed us to meet growing user demand for analytical publications to help explain developments in UK FDI statistics. We will continue to review the makeup of the FDI team to ensure it is appropriately resourced.
We would like to thank the authors of the NSQR for acknowledging the vast amounts of development work that took place prior to the NSQR being published, including references made to the new IT systems developed in-house to produce both annual and quarterly FDI estimates.
We welcome the recommendations and the suggestions made in the NSQR, which will form a major part of current and future FDI development work plans – although progress against some of the recommendations will depend on availability of resources outside the FDI team. All 20 recommendations have been accepted either wholly or in part. Progress against a number of the recommendations had been made prior to the release of the NSQR in July 2016, a number of which will soon be fully implemented.
2.3 Summary of recommendations
The NSQR includes a range of recommendations. We have reviewed these and feel that they can be categorised into 4 main areas:
sampling
data collection
survey methodology
general improvements (including staffing arrangements)
Of these, the reviewers identified 10 high priority recommendations:
review FDI sample design
implement improvements to the sampling process
review FDI sample allocation
re-introduce sample rotation on annual FDI using permanent random numbers (PRNs)
consider use and benefits of electronic data collection once the Data Collection Transformation programme (DCTP) has proven to deliver a reliable and flexible platform
implement measures to improve quality of data collected for quarterly FDI
cease using the Chancellor’s initiative data in annual inward FDI data compilation and report on the impact of its removal
implement production of standard errors for headline annual FDI estimates without delay
conduct a detailed review of benchmarking methods
increase the size of the FDI results and Publication and Development team
Overall there were 20 recommendations made in the NSQR. Of the remaining 10 not outlined in this section, 8 were deemed medium priority and 2 low priority. A response to each of the recommendations is outlined in Table 1.
2.4 High priority recommendations
Recommendations made in the NSQR should not be considered in isolation, as many rely on securing resource from in-house specialists and wider ONS initiatives such as the Data Collection Transformation programme – both of which are outside the immediate control of the International Transactions branch. Each of the recommendations will be considered and prioritised against competing priorities within the International Transactions branch and wider ONS. This will impact upon the speed at which we are able to implement the recommendations.
It should be noted that work had already begun on some of the recommendations made in the NSQR prior to its publication on 22 July 2016 – with some already implemented. An example of this was the recommendation made to conduct a review of FDI benchmarking methods, which has already taken place and updated methods have been implemented. Benchmarked estimates produced using the updated methodology were published in June 2016’s edition of the Balance of Payments.
Table 1: Detailed responses to each of the NSQR’s recommendations
| Recommendation | Description (abbreviated) | Priority | Accepted (A) Partially accepted (P) |
ONS response |
|---|---|---|---|---|
| R1 (Linked to recommendations R3 / R2 / R4) | Optimise the process of adding to the FDI database of known investors. | Low | A | Sampling for FDI utilises databases known internally to ONS as world base and non-world base. The non-world base database is a list of known traders in FDI and has been built up over a number of years utilising survey responses, whereas the world base database is compiled using data from multiple sources of possible businesses involved in FDI. We accept the recommendation that the process for moving companies between the UK business database (world base) and the FDI known traders database (non-world base) needs to be optimised. The thresholds currently used to identify companies that move between the populations need to be reviewed. It is envisaged that this would be achieved as part of the wider methodological review that will be conducted in response to recommendations R2 and R4. |
| R2 | Commission a methodological review of the FDI surveys sample design. | High | A | The NSQR paid particular attention to current sampling techniques, which we acknowledge as an area of FDI that has not been reviewed recently; therefore, the recommendation of a full methodological review is welcomed. Prior to the publication of the NSQR, we had designated this as a high priority piece of work to be completed by in-house methodologists ideally during this financial year. Once the review is complete, we will assess the recommendations made and develop a plan for how and when the changes can be implemented with the objective of improving both the FDI sample design and allocation. It is not envisaged that any significant changes to FDI sampling will be implemented in time for the sampling of the 2017 quarterly surveys and the 2016 annual surveys. |
| R3 | Improve the implementation of the sample selection mechanisms. | High | A | We have already implemented procedures aimed at improving sampling mechanisms without the need for input from in-house specialists. Systematic processes were introduced during the sampling process in January 2016 that will have benefited the sampling process for both the annual 2015 and quarterly 2016 FDI surveys. We envisage that further recommendations regarding the FDI sample design will be made in the review. We will consider each recommendation in turn and implement the changes accordingly. |
| R4 | Review the sample size allocation implementation procedures, whilst ensuring sampling fractions and sample sizes are constant over time. | High | A | The NSQR highlights that the approach used for FDI sample allocation has varied in recent years and deviated slightly from the preferred sample allocation method specified by in-house methodologists. We acknowledge that current methods have resulted in sampling fractions being maintained over time rather than fluctuating to reflect changes in the population. We have prioritised this recommendation and will ensure that sample allocations are reviewed and changes implemented either in isolation or as part of the wider FDI sample review depending on which is more timely. |
| R5 | Re-introduce sample rotation with the correct use of PRNs. | High | A | We agree with the recommendation to re-introduce the use of permanent random numbers (PRNs) to improve how the random element of the FDI sample is rotated. We have already put in place a solution to start recording PRN s over time, which is necessary before sample rotation based on PRNs can be fully re-introduced. This information will be utilised in the in-depth methodological review focusing on FDI sampling and how the use of PRNs is to be re-instated. |
| R6 | Questionnaire development (including addressing typographical errors). | Medium | A | We agree that there is a need to improve the FDI questionnaires, particularly the question definition guidance provided to users on what they should include and exclude from their returns. We also acknowledge that typographical errors that exist on the survey need to be rectified. We will commit to updating both the quarterly and annual surveys where applicable and will use the revised questionnaires for data collection from the Quarter 1 (January to March) 2017 and 2016 reference periods onwards. |
| R7 | Consider carefully the use of electronic questionnaires for FDI and adopt these only once ONSs electronic data collection platform has been delivered and proven to be reliable for FDI. | High | A | The FDI survey will not be migrated to the new platform before we are confident that the functionality is able to cater for the surveys complexities. The data collection transformation platform currently under development at ONS will facilitate collection of data electronically. For business surveys such as FDI, the goal is to have electronic questionnaires that respondents can complete online and use to communicate directly with us any explanations for large data variations. We welcome this initiative and recognise the benefits that collecting data in this manner will have to FDI survey data. In the interim, we will continue to use a method known as secure electronic file transfer (SEFT) to send FDI questionnaires electronically. This has proven to be a valuable tool for capturing outward FDI data returns as a large proportion of the respondents are based outside of the UK. This method supports timely dispatch and return of questionnaires to overseas respondents. |
| R8 | Introduce data processing measures aimed at improving quality of data collected on the quarterly surveys | High | P | The NSQR recommended measures should be implemented to improve data quality with particular attention paid to improving response and clearance rates one quarter after the reference period. We acknowledge response and clearance rates achieved for FDI, particularly on the outward survey, are low; however, this is largely due to overseas respondents struggling to return survey data in time for the tight national accounts production round deadlines FDI deliveries must adhere to. FDI data deliveries do adhere to the UKs National Accounts Revisions Policy, however; therefore, initial estimates do get revised in later periods when response and clearance rates are much higher. Taking all factors into consideration, we believe that there are no obvious steps that can be taken in the short-term to improve response and clearance rates of initial estimates. We do, however, envisage that the roll-out of the electronic questionnaires for FDI will help improve the overall response and clearance rates in the long-term. FDI will begin utilising such methods in 2017, with the roll-out expected to continue into 2018. The NSQR also identified perceived inconsistencies between validation rules coded into the data collection platform and rules specified by methodological experts a factor that may unnecessarily contribute to FDI form failure rates. We will carry out investigations to identify validation rules that have either been coded incorrectly or omitted and take steps to ensure that, under the Data Collection Transformation programme, any future platform for FDI data collection has validation rules that align with those specified by our methodologists. |
| R9 | Continue work under way evaluating the benefits of migrating the FDI data take-on system to the latest version. | Medium | P | We have actively participated in work streams that focused on the functionality required for FDI to migrate to the latest version of the current data collection platform. Due to the volume and complexity of the functionality required, coupled with the loss of IT developers, a decision has been taken by the wider organisation to halt the migration exercise. Users will not experience a loss in data quality as a result of this decision. We welcome the initiation of the wider data collection transformation platform, a goal of which is to replace the current data collection platform used by FDI and other business surveys. We acknowledge the benefits this will bring from a strategic perspective, such as having all our business surveys using a common system and also the improvements likely to be seen in the quality of FDI outputs. We will engage fully in this project to ensure its successful implementation. |
| R10 | Improve the industry coding used on FDI and its presentation for respondents and users on this basis. | Low | P | FDI data is compiled using a 3-digit standard industrial classification (SIC) code, which FDI survey respondents select from a pre-defined list included in the FDI questionnaire. For a number of SIC codes, there is no correlation to the lower level 5-digit SIC code held on the Inter-Departmental Business Register. The reasoning behind this slight deviation in classifications is to ensure that respondent data is classified correctly at source to ensure it feeds into the correct industry totals. Failure to classify businesses in this manner and to utilise SIC codes held on the IDBR could introduce volatility to the FDI estimates as SIC codes on the IDBR at business level can change. We do accept that this may not be implicitly clear to users, so will commit to updating publically available FDI documentation to better communicate the use of 3-digit SIC codes to users. |
| R11 | Discontinue the use of the Chancellors initiative data and report on impact of its removal from historical estimates. | High | A | The Chancellors initiative data has historically been used in the production of annual inward FDI estimates to ensure imputation classes have sufficient data to produce robust imputation links. In line with the recently implemented BPM6 manual, the increase in the size of the FDI sample size has resulted in the use of the Chancellors initiative no longer being required. In addition, the data had not been updated for some time; therefore, the NSQR recommended that the use of the Chancellors initiative data be discontinued. We have taken steps to implement this recommendation ahead of the next FDI publication in December 2016, in which the Chancellors initiative data will be removed from the revised 2014 estimates and provisional 2015 estimates. To ensure consistency with balance of payments, data for periods prior to 2014 will remain un-revised until the next opportunity arises for historical revisions to be taken on. All associated publication tables will be clearly footnoted to ensure users are aware of the change and a paper presenting the impact on estimates will be published alongside the December 2016 FDI publication. |
| R12 | Investigate the practical need for an adjustment for clustering and identify the best approach for robustly estimating values used in the formula for prediction estimation. | Medium | P | Work is currently underway to specify a model to assess the impact of this change on FDI estimates. We will conduct a full impact assessment and, if clustering is deemed to improve the quality of the estimates, then we will permanently implement the change. |
| R13 | Investigate, report, and implement the switch to the winsorisation method for outliering. | Medium | P | We agree with the recommendation that switching to the winsorisation method for outliering should be explored with a view to implement the use of this technique in the longer-term. However, due to the volatility of the FDI data, particularly intercompany loans, a cautious approach to adopting the winsorisation method will be taken. A full assessment will be carried out to study the impact of the new outliering methods on FDI estimates and the time series before a decision is made on whether to adopt the new methods. |
| R14 | Re-introduce standard error calculation and publish without delay. | High | A | FDI standard errors have not been published since the 2010 reference period, when the implementation of the latest international standards (BPM6) affected existing estimation methodology. We acknowledge the importance of standard errors as a measure of accuracy and their use to users. With this in mind, we agree with the recommendation made in the NSQR that they should be re-instated. System development is currently under way and near completion, however, the methods used are subject to review by in-house methodologists. Once this methodology has been deemed as fit for purpose, we will re-commence publishing FDI standard errors. |
| R15 | Expand the standard errors work to measure the precision of lower level domains and quarterly estimates. | Medium | A | Once suitable methods have been developed to produce standard errors for annual FDI data, we will investigate whether the methods can be adapted to produce standard errors for lower level domains and quarterly estimates. No stand-alone bulletin for quarterly FDI data is currently published; therefore, we will also need to decide on the most appropriate method for communicating these standard errors to users once they have been produced. |
| R16 | Commission an investigation to determine appropriate sample sizes for principal FDI output. | Medium | P | We agree that a review to determine appropriate sample sizes for FDI would be beneficial and believe that the analysis needed to inform sample sizes should be part of the methodological review due to be conducted for recommendations R1 to R5. We will review the findings of the methodological report and will ensure that priority is given to addressing recommendations R1 to R5. After a period of stability, we will look to implement any changes specified in relation to FDI sample size.In the interim, work will continue to develop standard errors for FDI, which will help inform the review into the appropriate sample sizes. |
| R17 | Improve the disclosure control process and transparency of FDI. | Medium | A | We agree with the recommendation made that the applied FDI statistical disclosure methodology should be reviewed. FDI disclosure rules were reviewed as part of a wider ONS initiative on disclosure methods, with recommendations provided on how to enhance methods and reduce levels of suppression. We will assess the feasibility and impact of implementing these recommendations and, if appropriate, will look to utilise the new method in the 2017 edition of the FDI publication. We will continue to investigate software solutions that can be used to automate the application of secondary disclosure control, in line with our wider initiatives. |
| R18 | Conduct a detailed review of benchmarking methods. | High | A | The FDI benchmarking process has been enhanced in recent years to reflect changes in the applied methodology and to automate processes. These developments have improved the overall quality of the FDI benchmarked estimates. The NSQR has recommended a detailed review of the benchmarking methods be conducted, which we welcome given the extensive work undertaken recently. A detailed review with in-house methodologists is expected to further enhance the quality of the benchmarked estimates. |
| R19 | Increase the size of the FDI and International Transactions teams. | High | P | The NSQR recommended increasing the size of the FDI team, which comprises both the FDI Results and Publication (RAP) team and the Development team, of which there are 9 staff in total. We partially agree with the recommendation, as we believe that increasing the analytical capability of the teams rather than solely focusing on headcount, is the best approach to improving quality and understanding of the data. Over the last 12 months, we have changed the management structure of the FDI team considerably; the analytical capability of the team has increased with the recruitment of a skilled economist taking up the position of Branch Head of the International Transactions branch, overseen by a Deputy Director who is also a skilled economist. Recruiting in this manner has enabled a number of major analytical pieces to be published, which have aided user understanding of FDI statistics and improved the quality of estimates. Efforts have also been made to ensure that FDI knowledge is shared amongst existing members to mitigate the loss of survey knowledge in response to a recent period of high staff turnover. This has been aided with the rollout of the newly developed SAS systems, which have provided the RAP teams with much needed system stability and, crucially, time to conduct training exercises. We are committed to reviewing the structure of the FDI teams on an on-going basis and, where possible, explore options to increase staff numbers if business needs exist. |
| R20 | Review and increase FDI documentation that is available to both internal and external users. | Medium | A | FDI has an up-to-date quality and methodology information (QMI) document that is available via our website. We acknowledge that FDI has recently undergone significant amounts of change, such as migrating between production systems. It is worth noting, however, that no deviations in methods or sampling techniques have occurred to make the existing documentation redundant. We are committed to continuing to aid users understanding of FDI and have begun publishing in-depth articles drawing on analysis from the FDI micro data, which has been well received and will continue. Supporting documentation such as the QMI will continue to be updated as changes in methods occur. |
3. Conclusion and next steps
We welcome the recent review of the FDI survey and are grateful to the reviewers, Gareth James and Stuart Brown, for the wide range of recommendations made with the aim of improving the quality and interpretation of FDI statistics. These recommendations will be used as part of our ongoing commitment to improve the way official statistics are measured and disseminated. Details presented in this response are based on an internal plan that has prioritised each of the recommendations and outlined how they will be taken forward. A number of solutions have already been implemented to address some of the recommendations raised in the NSQR, while work on others is already under way. We will continue to keep you informed of the methodological improvements made to our FDI statistics that have arisen from both this NSQR and other transformation programmes via our published methodological notes and documentation found within the annual FDI publication.
Authors
Daniel Robinson, Claire Dobbins and Sami Hamroush – Office for National Statistics
October 2016