Estimates of total factor productivity from the Annual Business Survey, Great Britain: 1998 to 2019: August 2022

• Calculating labour productivity growth using data from business surveys, overall productivity growth fell from 2.22% per year from 1998 to 2007 to 1.12% per year from 2011 to 2019, a fall often referred to as the "productivity puzzle".

• Falling growth from reallocation across firms inside industries and falling total factor productivity growth of the most productive workers accounts substantially for the productivity puzzle.

• Capital deepening and shallowing only has a small role in the productivity puzzle and varies across industries.

• When explaining variation in productivity across businesses, having a higher capital stock, being foreign owned, being an international trader, being located in the Greater South East, using IT more intensively and using advertising and market research services more intensively, are all factors correlated with higher labour productivity.

• Controlling for the capital stock using a total factor productivity measure, these correlations are lower or not statistically significantly different from zero.

Labour productivity is a simple measure of productivity, dividing an output by a measure of the labour input, usually either hours worked or number of workers.

A total factor productivity (TFP) or multi-factor productivity (MFP) measure tries to account for multiple inputs into production. Usually these include a labour input measure and a capital input measure, but there is significant variation in the types of input measures used by researchers, often because of the availability of different data sources. TFP measures may be more informative, because capital plays an important role in production. However, TFP numbers are also more difficult to interpret. Hours and workers are well-defined quantities. TFP has to be defined relative to a benchmark, such as growth from one year to the next.

The Office for National Statistics (ONS) has long published the multi-factor productivity experimental statistics for the market sector at the UK level. Capital input is accounted for by the Volume Index of Capital Services, and labour input is accounted for by the Compositionally Adjusted Labour Input index. Our simple guide to multi-factor productivity provides more detail on this statistics series. This article will use TFP to differentiate from the MFP statistics series.

For this article, TFP measures were calculated for each business recorded in the ONS Annual Business Survey (ABS), approximately one million datapoints over twenty years. TFP was calculated using regressions to calculate production functions, to control for number of workers and total capital stock at each business, and then taking the residual error term in the regression as the estimate. As the production function regressions were estimated in log terms, the regression residual can be treated as a ratio between actual output and the amount of output accounted for by number of workers and total capital stock in the regression.

Table 1 shows six regression models examining which high level variables correlate with labour productivity and total factor productivity (TFP).

Trade in goods data was only captured on the Annual Business Survey (ABS) from 2013 onwards, however other information on a business is available for longer.

Table 1 shows correlations. Almost certainly, there will be different causal effects driving the correlations, and many factors that are not observed. More spending on IT may cause some of the increase in productivity. However, a business might have a higher quality of managers who are more likely to invest in IT, but also in other factors that increase productivity. Higher spending on advertising and market research may be more indicative of a management team making wider strategic investments than having a direct relationship to making production more efficient.

Intensity of IT and advertising and market research spending are included as these are often used as examples of intangible capital, and data for this expenditure are available on the ABS. For example, the value of a brand behaves in a capital-like way even though it does not exist physically and is not currently measured as part of the national accounts capital stock. In both cases, a high intangible capital firm is more productive than a firm whose spending is fully concentrated on other items.

There are several variables which correlate with labour productivity but not TFP. There is a productivity premium for EU-owned companies for labour productivity, but not TFP. EU-owned companies have higher capital per worker, which accounts for the higher labour productivity. There is no premium in TFP.

Similarly, the effects of many of the other coefficients are diminished. Higher capital per worker explains some of the London premium and some of the non-EU ownership premium, as well as reducing the correlation between productivity and intensity of spending on IT and advertising.

A business engaging in international trade in services is likely to be more productive than a business engaging in trade in goods. However, the goods trader is still likely to be more productive than a non-trader. These correlations are higher for labour productivity than TFP.

In specifications 5 and 6 the services TFP premium is the same as for labour productivity. In specification 6, there is no correlation between higher TFP and trading goods internationally. Much of the productivity premia of international traders is accounted for by having higher capital, but this does not indicate causation. It may be that a firm intending to export makes more valuable investments, or that firms with higher TFP are also better at overcoming trade barriers, or it may be both in addition to other factors.   

The variables for trade are binaries, and do not account for the size of the trading relationship. Previous research work on UK trade in goods and productivity found that there is a productivity premium increasing in the intensive margin of a trading relationship for goods. This research used a separate data source for trade in goods, looked at a separate time period, and found higher labour productivity goods premia.

Table 2 repeats the regression specifications for the manufacturing sector specifically. Manufacturing has strong returns to scale. Even in TFP terms there are strong returns to scale in workforce size, as shown by the strong positive correlation between number of workers and TFP. IT expenditure is more important for manufacturing than services. Moving 1.0% of spending into IT from other spending is correlated with a 1.3% increase in TFP, while for the whole sample the effect was only barely significant.

This section provides statistics on how different businesses contribute to overall productivity growth. Productivity varies greatly across businesses. In 2019, for the same level of inputs, a worker in the top 10% by productivity produced over 3.5 times as much output as a median productivity worker.

Changes in the composition of businesses can contribute to productivity growth, as well as the businesses themselves. If a high-productivity business takes on extra staff, and they contribute to the economy at that high productivity, the overall productivity of the economy grows even though the productivity of the underlying business does not grow at all.

This reallocation of workers is likely to be an important channel for aggregate productivity growth. A high-productivity business should be able to afford to hire workers away from low-productivity businesses and grow its profits. Unless there are other factors, workers should move to higher-productivity businesses over time.

Table 3 shows labour productivity growth decomposed into:

• within-firm growth in total factor productivity (TFP)

• within-firm growth because of changes in capital per worker

• productivity growth because of reallocation of workers between firms

There are two sides to the reallocation of workers -- firms can compete against firms in the same industry, but industries themselves have growing and shrinking sizes of their workforce. Table 3 splits out productivity growth because of reallocation inside and across industries.

Reallocation across firms inside industries and TFP growth in large firms account for the productivity puzzle for the non-financial business economy. The other contributions are small or, in fact, contribute more to overall labour productivity growth after than before.

Table 4 shows, where within-firm labour productivity growth is driving aggregate productivity growth, it is the most productive 10% of workers. These workers are already starting with a much higher labour productivity level, so any growth rate has a larger weight on the overall average, if it is coming from TFP or investment.

Low-productivity companies often have falling within-firm productivity, or at least the effect of falling within-firm productivity outweighs any within-firm productivity growth for below-median productivity firms. In fact, below-median firms have slower within-firm productivity decline in the 2010s, compared with before 2007.

Most of the contribution to aggregate productivity growth from below the 90th percentile was because of reallocation. Below the 90th percentile, relatively higher productivity firms grew faster than relatively lower productivity firms. However, this growth slowed down substantially comparing the 2010s with the period to 2007.

The growth contribution from capital deepening is relatively small. Estimating production functions from firm-level microdata usually gives a lower value for the parameter for capital than in the national accounts-based multi-factor productivity growth accounting model. In the model, the income share of gross operating surplus is used as the parameter for capital. The production function parameter estimates are also fixed over the whole time period, while the national accounts-based estimates vary over time. Retail and wholesale and transport and storage have higher capital per worker in the 2010s than in the 1999 to 2017 period, and contribute the most to this effect.

Total employment in manufacturing fell 31% between 1998 and 2007 in this dataset. While these jobs were low productivity for manufacturing, they were high productivity relative to the rest of the economy. For this reason, manufacturing's within-firm contribution to growth is strong and positive, while its contribution from reallocation across industries is negative. These two effects are very large in size, for a services dominated economy, but offset each other.

The trends for manufacturing were seen in most of the two-digit industries within the sector. There was no growth from reallocation inside two-digit manufacturing industries, while the reallocation across industries is mainly caused by the difference between manufacturing and services industries. Part of the overall fall in trend growth between 1998 and 2007 is because of manufacturing. However, this is because of the structural change in the size of the sector, and could not be repeated without causing further losses to reallocation across industries.

As manufacturing accounts for a disproportionate share of the fall in within-firm TFP growth, most of the remainder of the productivity puzzle is because of falling reallocation inside services. No one services industry stands out here. Falling growth because of reallocation is present in most of the two-digit industries in services, and all services industry sections (with the exception of real estate services).

Combined, the pre-2007 trends lead to a fall in the overall dispersion of labour productivity, but lower growth enables a higher dispersion in productivity in the 2010s. Post 2010, the economy is more dependent on the smaller fraction of high-productivity workers to produce gross domestic product (GDP), leading to overall productivity growth.

Total factor productivity

Productivity accounting for labour and capital inputs, whereas labour productivity measures only account for labour input.

Perpetual Inventory Method

The Perpetual Inventory Method is a common way of estimating capital stocks. It assumes that the stock of assets at any given point in time is the cumulative sum of all investment flows, minus any losses arising from depreciation.

Production function

The production function is an equation that represents how a firm, industry or economy transforms inputs (usually capital and labour) into output. The exact shape of the production function depends on the available technology and organisational arrangements employed in the production of a particular good or goods.

Unbalanced panel

A dataset of units over time, where not all units are present in all years. Because the data in this article is an unbalanced panel, there are sample error terms in the productivity decomposition.

Data sources

This article is limited to the data available from the Annual Business Survey (ABS) and the Annual Business Inquiry Part 2. More details on these data sources can be found in our Annual Business Survey technical report and our Estimates of markups, market power and business dynamism from the Annual Business Survey, UK: 1997 to 2019 article.

The surveys aim to capture the largest 20,000 firms every year, then represent the rest of firms using a stratified sample. Firm level capital investment is estimated for firms not sampled, using average investment per worker of sampled firms in the same sizeband and industry. Firm level capital stocks are estimated using the Perpetual Inventory Method. Past reported capital expenditure is added and depreciated where it exists, and the estimated capital expenditure when the firm is not sampled. When a firm starts, it is allocated an initial capital stock from the productive capital stock from the national accounts productivity series, and depreciation rates are taken per industry from the national accounts.

Methods

The productivity decomposition is adapted from decompositions used in the economics literature, primarily chapters 3 and 4 of Price and Productivity Measurement Vol 6 (2010), by Diewert and Fox. The relative prices term accounts for biases where there is high productivity growth. This is because while total output in nominal prices terms is the same, price per unit of output drops, and the real terms value of output grows. In this case, going backwards in time, the real terms value falls, but in an arbitrary way because of the selection of the base year. To avoid this bias, national accounts gross value added (GVA) and gross domestic product (GDP) are calculated in chain volume terms, where growth is only weighted by contemporaneous nominal GVA and GDP. The relative prices term applies this correction.

To examine changes in productivity between one year's sample and the next, new survey design weights are derived post-hoc, and additional outlier filtering is applied. We have to link firms between years, but this is only possible for roughly 60% of the sample. These 60% will each have to represent more firms in the total population to make up for the other 40%. For firms with employment of less than 50, the sample does not reach that far. We would need survey weights that are too high to give reliable data. With the available data, we cannot take any strong stance on reallocation or within-firm growth for these micro and small firms. The dataset is not a balanced panel even for large firms where the ABS aims for full coverage. Large firms undergo mergers or restructurings, and the continuity breaks. These factors all contribute to the total sample error term in Tables 3 and 4.

As price deflators are applied at the industry level, the productivity decomposition is arranged into an industry level decomposition, which includes a within-industry decomposition.

The full specification for the aggregate productivity decomposition is available in the accompanying dataset.

For consistency, the aggregate GVA deflator is calculated using ABS data, and has minor differences to the GDP deflator.

TFP is calculated by Ordinary Least Squares (OLS), weighting each respondent by the GVA that the respondent represents -- otherwise the small number of large firms would have too little weight in the calibration of the production function parameters. A value-added translog production function with second order polynomials is used for section 3. Section 4 uses a Cobb-Douglas production function, because there are occasional edge cases in year-to-year changes in capital stock which create outliers (usually when a large firm undergoes a merger, selloff, acquisition or other restructuring).

A firm's TFP number is defined as the ratio between realised output and the fitted value from the production function, as this is analogous to the definition of labour productivity. We apply outlier filters, to the highest positive 1% and highest negative 1% of observations by labour productivity for the labour productivity specifications, and TFP in the TFP specifications in section 3, within each year and two-digit industry. In section 4, these filters are applied by:

• labour productivity

• changes in labour productivity

• changes in capital stock

• contribution to the sample error term

Section 3 only uses firms that have responded to the ABS long-form questionnaire, as it asks for more detail on spending. To reduce unnecessary burden on business respondents, many businesses are sent a shorter questionnaire only asking for vital information. It is slightly biased towards larger firms, but the larger firms have less imputed capital expenditure estimates.

The results presented here only cover productivity growth to 2019. In future work, we aim to cover productivity growth over the most recent years.

We will also publish further details on the data, methods and analysis undertaken in future articles. There are many factors that relate to growth and business dynamism and in our accompanying article Estimates of markups, market power and business dynamism from the Annual Business Survey, UK: 1997 to 2019, we present initial results on how different factors relate to business growth.