Quantifying uncertainty in headline international migration estimates

In November 2023, measures of uncertainty with our administrative based migration estimates (ABMEs) were published alongside our provisional long-term international migration (LTIM) estimates bulletin for the first time. Uncertainty measures are indicators of the quality of the estimates.

The Office for Statistics Regulation's (OSR's) Code of Practice for Statistics (PDF, 577KB) recognises that data sources and methods have potential uncertainty and that the extent and nature of any uncertainty in the estimates should be clearly explained.

The published uncertainty measures are the result of our simulation-based approach for measuring uncertainty in LTIM estimates. However, it is important to recognise that these uncertainty measures only quantify some of the uncertainty in the ABME estimation process.

In this research update, we outline our approach for producing a composite measure of uncertainty for the headline international migration estimates, covering emigration, immigration, and net migration. The composite measure in the November 2023 publication covers the quantification of uncertainty around adjustments, modelling and survey-based estimates, but it does not include the quantification of uncertainty associated with administrative data sources.

The results from the current composite measures suggest that, as might be expected, provisional international migration estimates have larger relative uncertainty intervals than revised estimates. The reduction in uncertainty comes from the revised estimates having more length of stay information for people. It is less driven by assumptions, or modelling, in the estimation process.

We also present findings from a case study using expert judgement to estimate uncertainty in the unadjusted estimates produced from administrative data. The unadjusted estimates are the estimates from data sources before any modelling and adjustments are used for estimation purposes. We provided experts with evidence about the quality of the data for migration estimation, and asked them to come to an uncertainty estimation for both emigration and immigration. We gathered their uncertainty estimates and brought the judgements together to quantify their collective assessment of the uncertainty with unadjusted administrative based estimates.

Our results from the expert judgement showed it is a feasible option for quantifying uncertainty with data, with potential to offer a holistic evidence-based view of uncertainty across many areas of non-sampling error. However, further research and consideration of alternative estimation methods, beyond expert judgement, is required before it could be used in our LTIM uncertainty estimates.

We are keen to receive feedback on our methods for measuring uncertainty, both from those who find it useful, and from those who think it needs further thought and refinement. We are also interested to receive feedback on how we are communicating uncertainty. Please contact us at demographic.methods@ons.gov.uk.

Aims

Our paper provides a research update on our work to quantify uncertainty in LTIM estimates and builds on our previous research.

In this paper we will:

• briefly outline differences, and challenges, with quantifying uncertainty for administrative based estimates in comparison with sample based estimates

• provide a short comparison with other national statistical institutes (NSIs) for quantifying uncertainty in international migration, enabling us to consider our progress with respect to measuring uncertainty in administrative-data based estimates

• illustrate how we produce a composite measure of uncertainty for headline LTIM estimates: immigration, emigration, and net migration

• outline method and results from an exercise aiming to incorporate expert judgement for estimating the uncertainty associated with administrative data, with a focus on migration data

• outline future developments to progress with our work on quantifying uncertainty in LTIM estimates

Measuring uncertainty in statistical estimates

All statistical estimates have uncertainty or error, which relates to how they have been compiled, as well as the concept they are trying to measure. This error is defined as the difference between the final estimate, and the underlying unknown "true" value you would measure if you had a complete, true, dataset.

Historically, many official statistics have been based on surveys, in which data are collected as a random sample from a wider population. Estimates based on such data contain uncertainty (total survey error), which is made up of both sampling and non-sampling error.

Sampling error is a result of sample size, population variability, survey sample design, and the estimation method used. Non-sampling error is a result of specification, nonresponse, frame, measurement, and data processing errors.

Using administrative data for statistical estimates gives more prominence to non-sampling error. Administrative data-based statistics are the product of operational processes and as such are not set up in the same way as sample-based statistics (for example, survey statistics). Therefore, assumptions, processes, and quality measures applicable to sample-based statistics are not always applicable to administrative data-based statistics and alternative methods are required.

For example, administrative data are usually not a random sample and so estimates are not subject to sampling error; however, the data may suffer incompleteness and the processes are often not designed for statistical purposes. Therefore, as well as differences in the estimates themselves, the usual measures of uncertainty associated with sample-based statistics may not be applicable to administrative data-based statistics.

In comparison with quantifying sample-based uncertainty, methods for quantifying uncertainty with estimates derived from administrative data are undeveloped. To represent uncertainty in administrative data we need to consider other types of error. The Office for National Statistics (ONS) has classified these as representative, measurement, and processing error. More information about these classifications is available in our Cataloguing errors in administrative and alternative data sources methodology.

Representative errors (for example coverage and selection) and measurement errors (for example, validity and reliability) are mainly aligned with data uncertainties. Processing error is mainly aligned with uncertainties from methods applied to the data (for example, adjustments, modelling and data linkage).

Overall, quantification of uncertainty from administrative based estimates requires accounting for coverage, measurement, and processing errors if the measures of uncertainty are to quantify total administrative error.

International comparison

Administrative based migration estimation with uncertainty measures is a developing field for national statistical institutes (NSIs). Here we present a brief international comparison to understand progress by other NSIs and what, if any, measures of uncertainty they provide with LTIM estimates. The comparison only covers some NSIs that use administrative data for their international migration estimates, rather than surveys or population registers being the main data source.

Statistics New Zealand, Australian Bureau of Statistics, and Statistics Canada use administrative data as the main source for their international migration estimates.

Stats New Zealand uses statistical modelling to produce international migration estimates. The provisional estimates have associated confidence intervals that reflect model uncertainty, but not the extent of future revisions to provisional data.

The Australian Bureau of Statistics produce overseas migration estimates from administrative data from the Australian Government Department of Home Affairs, which currently do not have accompanying measures of uncertainty.

Statistics Canada produce quarterly demographic estimates, with migration estimates, that are extracted from administrative files and derived from other Statistics Canada surveys or other sources. Precocity error is a main quality indicator, defined as the difference between provisional estimates and final estimates because of revisions.   

Overall, the comparison with other NSIs suggests that the ONS is either comparable with the provision of measures of uncertainty for administrative based migration estimates, or providing more quantification of uncertainty than other NSIs. The Office for Statistical Regulation (OSR) has also recognised that we have demonstrated our commitment to providing users with a clear and comprehensive understanding of uncertainty. This paper is continuing the commitment through the provision of a research update.

Limitations

There are limitations to consider when interpreting the results presented in this paper.

• The expert judgement was an experimental attempt.

• Currently, we do not plan to integrate expert judgement into measuring uncertainty with LTIM estimates without further research and consideration of other methods.

• We advise not to combine results from this paper with other published measures of uncertainty for LTIM estimates; the results would be inaccurate as there are limitations with the expert judgement method.

• The expert judgement and composite measures are currently distinct.

• Our reported results for the composite measure are partial and only measure uncertainty around adjustments, modelling, and survey-based estimates.

Our Measuring uncertainty in international migration estimates working paper provides more information about our methods for these components of uncertainty in the ABME process.

Data

Multiple data sources were used to generate a composite measure of uncertainty for long term international migration, including:

• Home Office Border and Immigration data (HOBI)

• Registration and Population Interaction Database (RAPID)

• International Passenger Survey (IPS)

Some additional data from the Higher Education Statistics Agency (HESA) and Pay As You Earn (PAYE) Real Time Information (RTI) (for the EU national student adjustment) are also incorporated for modelling adjustments.

For more information on data sources and methods for producing the published administrative based migration estimates (ABMEs) see our Methods to produce provisional long-term international migration estimates methodology.

Methods

The ABME process is a complex statistical system with multiple steps needed to produce the provisional long-term international migration (LTIM) estimates. To measure uncertainty, we quantify uncertainty associated with the individual steps in this statistical system, for example, adjustments, modelling, and survey-based components. This uses a simulation methodology that is outlined in our Measuring uncertainty in international migration estimates working paper.

The final stage of the methodology is to bring together all the simulated samples to create an uncertainty interval for overall immigration, emigration, and net migration. We produce simulated samples for the three main national groupings for LTIM estimates: EU, non-EU, and British. Our simulation samples for each national grouping currently quantify some, but not all, sources of uncertainty.

Non-EU nationals

Quantified uncertainty

• Final-year immigration adjustment (excluding British National (Oversees) (BNO) and Ukraine).

• Emigration re-arrivals adjustment.

Unquantified uncertainty

• Admin data source (Home Office Borders and Immigration Systems data).

• Irregular migration or asylum migration.

• Indefinite leave to remain.

• Visa transitions adjustment.

EU nationals

Quantified uncertainty

• Student adjustment.

• People aged under 16 years adjustment.

• Inflow adjustment.

• Outflow adjustment.

Unquantified uncertainty

• Admin data source (for example, RAPID).

• EU settlement scheme.

British nationals

Quantified uncertainty

• IPS sampling error.

Unquantified uncertainty

• IPS non-sampling error.

The immigration and emigration simulated estimates for British, EU and non-EU nationals are all summed together in the order returned from the simulation results, rather than ordering the results ascending or descending. This adds some uncertainty to the composite measure. This approach is largely adopted from our Mid-Year Estimate uncertainty methodology in which the separate components of uncertainty are all summed together to provide an overall estimate.

Our composite method could potentially be improved. An assumption made when summing the components is that they are independent, and therefore the uncertainty is also a simple summation. This means that the three national estimates of migration (that is, EU, non-EU, and British nationals) that are brought together cannot have a strong, either positive or negative, correlation between them. Correlations between EU, non-EU and GB migration have not yet been investigated, but in theory there is likely to be some association, and this will vary over time.

This, coupled with how the three nationality groups are summed together can contribute towards producing asymmetrical uncertainty intervals. There could also be unaccounted biases in the data that are contributing to this trend.

Using this method to produce LTIM uncertainty intervals for headline international estimates means it is likely that we will produce asymmetrical and non-centered uncertainty intervals. Some of the asymmetry is explained by the composite method. It is also affected by limits and approaches for quantifying uncertainty in individual ABME processes. For example, non-EU adjustments and temporal disaggregation uncertainty estimations contribute to the asymmetry of the uncertainty intervals.

Non-EU adjustments for early leavers (immigration) and re-arrivals (emigration) can contribute to asymmetry as there are lower and upper limits. If the current trend is close to these limits, then the simulated results are likely to be skewed. For example, if more people with a certain type of visa are more likely to remain for 12 months or more, and are less likely to be early leavers, then the range of results to the left of the median will be denser, and the range of the results to the right of the median will be more spread out.

For temporal disaggregation of EU migration estimates our method is informed by the confidence intervals from the International Passenger Survey (IPS). While the confidence intervals for the IPS are symmetrical, there is potential for the range of the monthly confidence intervals for EU immigration and emigration to go below zero. In this situation, we use a truncated normal distribution, which can produce non-symmetrical results for our simulation approach.

The non-centered uncertainty means that the median of the uncertainty interval will generally differ from the LTIM point estimates for immigration, emigration, and net migration. The difference is largely attributable to method differences in the uncertainty estimation from the LTIM estimates. The uncertainty estimation incorporates a simulation-based approach, which is not used for the published point-estimates. Some of the data used for our uncertainty estimation differ to the data used in the point estimate approach (for example, adjustments and temporal disaggregation).

Results

The results for our composite measure of uncertainty for LTIM estimates presented here are partial, only covering some of the ABME process. They do not include any quantification with unadjusted administrative based estimates. They also do not include any uncertainty estimation from the expert judgement approach.

Our results cover three years of estimates:

• year ending (YE) June 2023 (provisional)

• YE December 2022 (revised)

• YE June 2022 (revised)

Tables 1 to 3 show the percentiles from the simulated results for total immigration, emigration, and net migration for each period of interest. Figures 1 to 3 show the relative percentage widths of the percentiles from the median (50th percentile).

To note, the revised periods have narrower intervals than the provisional periods. This is likely to be because of the actual data catching up to the periods of interest, meaning adjustments are either no longer needed or are more accurate.

In this situation, we apply a 12-month UN definition to classify a long-term migrant, so the revised data hold more information if someone is a long-term migrant. However, the provisional estimates require more assumptions for applying the 12-month definition.

There is also more up-to-date data used in the temporal disaggregation method for EU migration, which will also contribute to the narrower intervals. As more up-to-date data eventually become available for YE June 2023, we would expect these intervals to become narrower as well.

Data

For quantifying the uncertainty with unadjusted administrative-based estimates using expert judgement, we used Home Office Border and Immigration (HOBI) data from 2017 to 2022. The data combine visa and travel information to link an individual's travel movements into and out of the country using passport information. The data also assign long-term international migration status based on the derived length of stay in the UK.

We selected HOBI data for the expert judgement for two main reasons. First, HOBI data are used for visa-based migration estimation, which is proportionally the main contributor to long-term international migration (LTIM) estimates. Second, HOBI data are being considered by the Office for National Statistics (ONS) for the future estimation of EU national migration.

Methods

Expert judgement is one option that can be used for the estimation of statistical uncertainty in estimates based on administrative data, especially if alternative options are few or not well defined. Expert judgement relies on asking experts on a particular topic to draw on their expertise and provide reasonable estimates of some unknown quantity in the absence of direct quantitative measurement.

The ONS has previously used expert judgement for international migration estimates during the coronavirus (COVID-19) pandemic when the International Passenger Survey (IPS) was suspended. The Delphi process (as explained on the Corporate Finance Institute's website) was applied to obtain consensus from experts on the assumptions required for modelling international migration estimates during the pandemic (as explained in our working paper).

Our method for expert judgement has some components of the Delphi process and is influenced by how the International Panel on Climate Change (IPCC) consider quantifying uncertainties in practice.

In our semi-Delphi process we identified the issue and objective, selected a group of experts, and then held a questionnaire round. We did not have any follow-up questionnaire rounds as used in the full Delphi process. We considered the IPCC's methods for encoding expert judgements and how to align with experts' familiarity with probability distributions.

The issue we identified is that LTIM measures of uncertainty do not quantify the uncertainty with unadjusted estimates from the administrative data. The objective for the expert judgement was to provide an estimate of uncertainty for HOBI data as one of the main data sources for LTIM estimates.

The selected experts were both internal and external to the ONS. We required experts to have knowledge and experience in either demographic or migration statistics. We recruited six experts that were a combination of practitioners and academics. The internal experts had the required expertise in producing demographic statistics, but not involved in producing the ONS' LTIM estimates.

The experts were asked to produce their uncertainty estimates from evidence we provided them on the data source, combined with their expertise. We produced an evidence document to assist experts with their estimation of the uncertainty associated with HOBI data used for visa-based migration estimates, prior to any adjustments or modelling. The evidence document was produced through a combination of qualitative and quantitative analysis.

We structured the evidence document to align with the typology in cataloguing of errors in administrative data, as shown in our Cataloguing errors in administrative and alternative data sources working paper, (that is, representative, measurement, and processing error).

We also made clear to the experts that the scope of the exercise was specifically to focus on uncertainty associated with HOBI data, prior to adjustments and modelling. The uncertainty judgement was not to consider anything beyond visa-based migration that would need to consider uncertainty from different data sources (for example, Registration and Population Interaction Database (RAPID) and the International Passenger Survey (IPS)).

When designing the data collection methods for this expert judgement exercise, we considered the framing of the questions on uncertainty about the unknown quantity (the encoding method). The IPCC (PDF, 189KB) offers four methods - fixed value, fixed probability, interval methods, and graphing. The method selected should be based on the experts' familiarity with probability distributions. When recruiting the experts, we also enquired about their preference for encoding method, with fixed probabilities being selected as the option with greatest preference.

For the fixed probabilities, the experts were asked to independently complete two tables - one for immigration and one for emigration. The tables had rows with associated chance and probabilities, with experts being asked to complete their lower and upper bound estimates to align with the associated chance and probability. Table 4 shows the structure of the table that experts completed and the range of associated probabilities. In the lower and upper columns, the expert would enter their upper and lower estimates for a hypothetical estimate of 600,000 for both emigration and immigration.

The instructions asked experts to provide a lower and upper range where the chance of the "actual" unadjusted estimate falling within the provided bounds was represented by the fixed chance values. For example, if the unadjusted estimate was 600,000 for emigration and the fixed chance value was 1 out of 2 (50% probability), the expert could decide, based on the evidence, on ranges of 595,000 and 620,000 where the "actual" unadjusted estimate would fall within that range 1 time out of 2.

The fixed probabilities question and table were designed without the requirement for producing symmetrical results for the lower and upper bounds for each probability. This was an intentional design to give experts the capacity to assign more certainty to potential for overestimation or underestimation of immigration and emigration.

We also requested that experts provide some supporting rationale and reasoning for their uncertainty estimations for immigration and emigration. The supporting information provides useful context, enables us to evaluate if the expert has stayed within the scope of the request, and if there could be any inconsistencies with the uncertainty estimates.

From the experts' responses to the fixed probability questions, we applied a range-based probability resampling approach to generate uncertainty intervals for the unadjusted hypothetical estimate - both for immigration and emigration. There are four main steps to this resampling approach.

Step one: define the ranges that align with all the fixed probabilities

Define the ranges that align with all the fixed probabilities for each expert (for example, the range for 25% probability (1 in 4 chance) for an expert could be between 590,900 and 610,000 for emigration, while the range for the 50% probability (1 in 2 chance) for the same expert could be 570,000 and 625,000).

Step two: resample with replacement from these ranges

Resample with replacement from these ranges with the fixed probabilities defining the chances of a value within the range being selected. This is completed for each expert for immigration and emigration. Within the defined ranges for each fixed probability, each value had an equal chance of being selected (that is, uniform distribution). However, the probability of a value being selected between different ranges were not equal and aligned to the fixed probabilities.

Step three: conduct a final resampling with replacement

Conduct a final resampling with replacement from combining all the experts into one pool - we then resample from this pool for immigration and emigration, and the values within the pool all have an equal chance of being selected.

Step four: generate uncertainty intervals

The values from the final resampling with replacement are used as the uncertainty estimate for the unadjusted administrative-based estimate; we generate uncertainty intervals from using the observed percentiles, which are reported in the results.

Results

Here, we show uncertainty estimates from our expert judgement exercise for unadjusted administrative-based migration estimates (ABMEs) for visa-based migration using HOBI data. The results presented are based on providing the experts with hypothetical unadjusted immigration and emigration estimates. We used the hypothetical estimate of 600,000 for both immigration and emigration. The experts would use the hypothetical estimates to inform their answers for the fixed probabilities questions with respect to the range of uncertainty (that is, non-sampling error) with the 600,000-point estimate.

The results have some interesting findings. First, the collective results from the experts show more uncertainty (that is, wider ranges) associated with immigration, rather than emigration. For immigration, 50% of the simulation results ranged from 530,000 to 670,000 and 95% of the results ranged from 400,000 to 780,000. Whereas for emigration, 50% of the simulation results ranged from 560,000 to 660,000 and 95% of the results ranged from 430,000 to 780,000.

Second, the collective results are asymmetrical when it comes to underestimation and overestimation, especially at the 95% percentile range. The asymmetry distribution of results reflects that collectively for the experts, at the 95% level, there is a wider range of uncertainty associated with underestimation than overestimation - both for immigration and emigration.

Our exercise of incorporating expert judgement for estimating uncertainty associated with administrative data used for migration has been informative. However, our method would need some further consideration for improvement, based on limitations of the current design. The methodological considerations include the following steps.

Increase number of experts

our exercise included six experts and could benefit by expanding the number of experts. It would also be worth including more experts with direct experience of working with administrative data used for demographic estimation. In particular, consider experts with direct experience of working with HOBI data.

Revise the fixed probability questions

We would expect that expert judgements would produce uni-modal distribution. In actuality, the distributions simulated from the range-based probability sampling returned bi-modal and multi-modal distributions. The fixed probabilities selected in the question may have been the cause. Including more fixed probabilities, especially at the lower scale (for example, 1 in 5 chance, 1 in 10 chance), would likely produce uni-modal distributions from the experts.

Include feedback round for experts to revise their estimates

A verification step in the expert judgement, prior to the range-based simulation, would give an opportunity to provide the expert with feedback. This would identify if there were any inconsistencies in their estimates, or if the expert is basing their judgement on reasons that are out-of-scope. The verification step would give the opportunity for the expert to revise their estimates based on feedback.

Review resampling method

Consider alternatives to our range-based resampling method, with some alternatives assuming an underlying probability distribution (for example, normal, gamma).

Our research for measuring uncertainty in international migration estimates will continue to develop. Some areas we are considering for future development.

Data uncertainty - quantification method

As a case study, using expert judgement for estimating administrative data-based estimation uncertainty has been an invaluable exercise. Future research will continue with regards to how data uncertainty is quantified. Two areas could be considered for further research.

First, build upon using expert judgement with future refinements and amendments to the method. Some considerations could align the method more closely with the Delphi method, establish bounded limits (aligned with visa issued data), refine the approach for soliciting probabilities from experts, and test alternative methods for generating the overall uncertainty with the data based on the expert judgement. We can also consider providing the experts with more time to learn about specific data sources, with the opportunity to learn about data collection and processing from the data owners.

Second, test alternative methods to the expert judgement for the quantification of data uncertainty. Here we would consider if there were other appropriate methods suitable for quantifying data uncertainty with administrative data used for migration estimates. Ideally, we would look for methods that can be repeatable with each long-term international migration (LTIM) estimate.

Decide on the coverage of uncertainty intervals

Our comparison with other national statistical institutes (NSIs) demonstrated that there is variance on what is covered by the uncertainty measures for the migration estimates. In practice, when estimating using a sample-based estimation process, it is common to quantify the sampling error (for example, confidence intervals, standard error), but not quantify the non-sampling error.  

With administrative data, a decision is needed on what uncertainty should be quantified and captured in the uncertainty intervals. For example, do the uncertainty intervals need to include the quantification of doubt associated with representative error, measurement error, and processing error; or would it be appropriate to produce uncertainty intervals that only cover some areas of possible error associated with administrative based estimates (for example, modelling error within processing error). The latter approach recognises that quantifying uncertainty for all parts of the statistical process might not be practical or produce timely estimates.