Improved methods for total public service productivity: total, UK, 2017

Annual estimates of public service productivity, as published by the Office for National Statistics, are National Statistics. We measure productivity in nine service areas, four of which are adjusted for quality in some way. We produce estimates of productivity growth (and inputs and output growth) by service area and as a final aggregated index, total public service productivity.

This article updates users and invites views on methodological improvements that will be implemented in the forthcoming article, Public service productivity: total, UK, 2017, to be published on 8 January 2020. Comments can be sent to productivity@ons.gov.uk.

We have published similar articles in the past to pre-announce methods changes to users, in accordance with the Code of Practice for official statistics. In October 2017, we published our proposal on a new quality adjustment for public order and safety (which was later incorporated into our annual estimate) and in June 2018 we published an explanation of changes from a methods update to adult social care.

This article covers improvements across four service areas, to varying degrees of impact: education, healthcare, police, and public order and safety. The other service areas will only be affected by data changes and improved processing as outlined in Section 7. We also have changed our source for National Accounts data, which will affect all nine service areas. These changes include:

• updated data sources for education attainment at GCSE (or equivalent) for all four countries in the UK
• the application of the attainment data using a “cohort split” approach, where we attribute the GCSE (or equivalent) grades achieved across all five years of secondary schooling, rather than solely in Year 11 (or equivalent)
• improvements to inputs and output in healthcare
• minor changes to the public order and safety quality adjustment and a change to the deflator used for public order and safety labour inputs
• an update to police labour inputs
• a change to use Blue Book 2019 National Accounts data to include statistical improvements associated with Blue Book, particularly around consumption of fixed capital

Some of these updates relate to quality adjustment, where the output of a service area is adjusted to account for changes in quality in the final productivity estimate. For more details, see our guide to quality adjustment.

The education service area is one of the largest of the nine measured in public service productivity. We estimate growth in the quality of the services provided for four service areas. The education service area has contributed most to year-on-year changes in the overall quality adjustment. This reflects both its large weight (second only to healthcare) and volatility of the quality measures. This is shown in Figure 1, taken from last year's annual estimates.

The improvements described in this article will reduce the extent to which education quality fluctuates, which we expect to have a noticeable effect on the overall quality adjustment for total public service productivity.

In addition, policy changes are frequent in education provision. We regularly look to update the data sources used for the education adjustment accordingly.

Education was one of the first service areas for which a quality adjustment was developed. In 2007 an early methods article (PDF, 650KB) was published, detailing the first education productivity estimates that also considered various options for quality change. This followed the recommendations related to education from the Atkinson Review (PDF, 1.1MB), which is the foundation of our public service productivity estimates.

From then, other developments included the switch from the original average points score adjustment to using the threshold measure, the methods update being published in 2015 (PDF, 183KB). We also published an updated sources and methods document for education in 2017. 

Data for England were the priority in previous work as England is the largest country in the UK, in terms of both population and education expenditure share. In this article, we detail updates to all four countries in the UK. Noting the differences between the countries is especially important for education services, where devolved administrations have significant control, and so country-specific data is desirable.

England

We currently use the threshold measure from Department for Education (DfE) data: the proportion of students at the end of Key Stage 4 achieving five or more passes at GSCE (grades A* to C). We first used these data for the 2013 annual publication, incorporating backwards revisions using the threshold measure back to academic year 2008 to 2009. Before this, an average points score series was used.

After recent policy changes, this threshold measure is no longer being produced as a headline measure of GCSE attainment. The structure of the qualifications has changed and so has the marking system, with a shift from lettered to numbered grades. These changes were phased in over the past several years. Details on Key Stage 4 headline measures (PDF, 799KB) have been published by DfE.

To account for this change, we will use the DfE's new headline measure, Attainment 8 (PDF, 274KB), from the academic year 2016 to 2017. It counts performance in eight slots, two of which are reserved for English and maths, which are double weighted to demonstrate their importance. Three of the other slots are for English Baccalaureate subjects and the last three are for any GCSE or equivalent subject.

To reflect the importance of English and maths for students, we also intend to use the threshold measure inclusive of attainment in these two subjects from the academic years ending 2009 to 2017. This importance is reflected in the design of Attainment 8, where English and maths are double-weighted. Slots are also reserved for English (or Welsh) and maths in the new Welsh attainment data, called Capped 9 (more detail later in this section). Also, there are numerous academic resources (PDF, 296KB) on the importance of literacy and numeracy proficiency, and students achieving a good pass at GCSE English (or Welsh) and maths acts as a proxy for this.

The Department for Education recognises the importance of English and maths (or literacy and numeracy) in various papers including Improving engagement and attainment in maths and English course: insights from behavioural research (PDF, 3.8MB). This was behind a policy change in 2014 that required students to sit GCSEs in English and/or maths at college if they did not pass at secondary school, which has seen positive results. It was also recognised in the Wolf Review (PDF, 2.9MB), which was part of the rationale for previously switching from the Average Points Score to the threshold measure for English attainment.

Table 1: Data sources for attainment at GCSE or equivalent for England, current and new method

Source: Office for National Statistics

Notes

1. Dates are given by academic year, in accordance with the publication of the attainment data.

Scotland

For Scotland, we currently use a forecast of the previous average point score series, which was provided by Scottish Government until academic year 2013 to 2014. The discontinuation of this dataset may coincide with the educational reforms in Scotland, where Standard Grade qualifications were replaced with National 5 qualifications.

The Scottish education system is notably different from those in the other three countries of the UK. As such, the data produced by the Scottish Government to measure student attainment are not consistent with those produced by other countries. We do not make any direct comparisons between the countries because of the differences in how they collect and present data. However, the more similar the attainment data is, the more similar the proxy for quality is, increasing the validity of the aggregation of the four series to the total UK index.

We will now use data from the Scottish Qualifications Authority (SQA). SQA publish attainment by qualification type on their website. Different qualifications are for different SCQF (Scottish Common Qualifications Framework) Levels. In the SCQF, Level 5 is considered equivalent to GCSE passes of grade C or above.

The three Level 5 National qualifications we consider are: National 5s (the equivalent to GCSEs); Skills for Work and Personal Development (SWPD); and Intermediate 2s. For the latter, these have been phased out, with the last exams for them sat in the academic year 2014 to 2015. The majority of students sit National 5s.

Data on the average attainment per student for each type of qualification are provided by SQA - for example, for National 5s, the average number of grades A to C per students. We use the number of students sitting each type of qualification (also provided by SQA) to weight attainment together. This means that an overall Level 5 attainment index can be constructed. Data are available from 2013 to 2014, necessitating the use of one year of the forecast average points score value.

Table 2: Data sources for attainment at GCSE or equivalent for Scotland, current and new method

Source: Office for National Statistics

Notes

1. Dates are given by academic year, in accordance with the publication of the attainment data.

Wales

Welsh attainment data is currently the "average wider¹ points score" for Year 11 students. It has seen large growth from the academic year 2010 to 2011 onwards, before a steep decline in the academic year 2016 2017. As attainment is a proxy for quality, the conclusions drawn from these data seem unrealistic and the Welsh Government is currently reviewing its attainment measures.

We will now use the "average capped wider points score" from academic years ending 2011 to 2017. Both these data series and other Key Stage 4 indicators are available online. The capped version of the average wider points score considers the students' best eight qualifications, whereas as the original measure has no restriction on the number of qualifications obtained. The Wolf Review (PDF, 2.9MB) discussed the growing contribution to average points score measures in England from vocational qualifications, as more students were studying for and passing them. There were concerns that not all the qualifications were as useful as the attainment data suggested. It appears likely that this same issue is been observed in Wales, resulting in the Welsh Government's decision to review their attainment data.

For the academic year 2017 to 2018, we will use a new headline measure, as recommended by the Welsh Government. This is called "Capped 9"².

More detail on the methodology behind Capped 9 and recent changes to Key Stage 4 indicators are available here (PDF, 799KB). Broadly speaking, Capped 9 is more similar to Attainment 8, England's new headline measure, than the average capped wider points score. It counts the students' best nine qualifications, where one slot is reserved for English or Welsh, two for maths qualifications and two for science, and the remaining four are for the best GCSE or equivalents achieved.

Table 3: Data sources for attainment at GCSE or equivalent for Wales, current and new method

Source: Office for National Statistics

Notes

1. Dates are given by academic year, in accordance with the publication of the attainment data.

Northern Ireland

In all previous estimates of public service productivity, Northern Ireland attainment has been assumed equal to England, as a result of the absence of known data sources.

We will now implement a threshold measure for Northern Ireland which is publicly available since the academic year 2005 to 2006. Through collaboration with the Northern Ireland Department of Education, we are now able to use data from as early as the academic year 1996 to 1997. This means that we have an attainment adjustment specifically for Northern Ireland, helping to ensure good representation for all four countries in the UK. The measure is the percentage of Year 12s achieving five or more GCSEs at grade C or above and it is available online.

As for England, we will now use the threshold measure inclusive of English and maths beginning from the academic year 2009 to 2010 (also available online).

Table 4: Data sources for attainment at GCSE or equivalent for Northern Ireland, current and new method

Source: Office for National Statistics

Notes

1. Dates are given by academic year, in accordance with the publication of the attainment data.

Notes for Improvements to data sources for the education quality adjustment

1. The "wider" in "average wider points score" simply refers to the inclusion of various non-GSCE qualifications in the average points score measure, to differentiate it from the standard "average points score" measures used elsewhere.

2. There are two versions, the original and the interim, both of which will be produced for the foreseeable future. We plan to use the original measure as the interim Capped 9 is only available for academic year 2018 to 2019.

Currently, when we use attainment data as a proxy for change in quality in education, the data are attributed solely to the final year of teaching (Year 11 or equivalent). However, when a new attainment measure is published, this is not just a reflection of that year's quality of teaching, but of the secondary education provision over the entire five years (or equivalent).

For example, GCSE attainment data published for the academic year 2017 to 2018 reflects the effectiveness of the teaching from Year 7 to Year 11. As such, the new "cohort split" approach will apply certain percentages of the new attainment data back to previous years, subject to contributions deemed appropriate.

This approach is supported by the Atkinson Review (PDF, 1.1MB) which recognised that "The GCSE results are the outcome of 11 years of compulsory schooling" but also that there would be a large time lag in fully measuring attainment. Chapter 9 of the review is dedicated to education, with Recommendation 9.2 being "we recommend that ONS should update and revise the quality adjustment factor for schools, using later information about GCSE results, and if possible also information from all parts of the United Kingdom" (page 131). The updates presented in this article are in this spirit.

There is limited evidence on the contribution of different years of schooling to attainment. Based on the structure of secondary education and the available literature, outlined later in the section, we have chosen weights as shown in Table 5.

In the above scenario, when the academic year 2017 to 2018 results are released, 10% would be retrospectively applied to the academic year 2013 to 2014 to account for Year 7 quality of teaching, 10% to the academic year 2014 to 2015 for Year 8, and so on.

In England and Wales, secondary schooling is undertaken from ages 11 to 16 years, with children progressing from Year 7 to Year 11. GSCE (and equivalent) exams are typically taken in Years 10 and 11 (Key Stage 4), while earlier years of secondary schooling build on the education in primary school (ages 4 to 11 years). In Northern Ireland Year 12 is equivalent to Year 11 in England and Wales. This is because Reception in England and Wales is called Year 1 in Northern Ireland. Scotland has a differing system, with a more blended approach to when students take qualifications. S4 and S5 are equivalent to Years 10 and 11 in England and Wales, and the majority of GCSE-equivalent exams are taken in these years.

We have given a 20% weighting to Year 9 in our chosen specification. Year 9 has been identified as an important period in some literature, and there is evidence that some students sit final exams in Year 10 and even Year 9, not just Year 11. Additionally, the teaching of the courses for these exams does not necessarily take place over the year in which the final exam is set; many schools start teaching for GCSEs in Year 9.

More information on the importance of Year 9 is available in Ofsted's Key Stage 3: The wasted years? (PDF, 148KB), which explains how this stage is essential to transition students from primary school to the exam stage in secondary school, as well as Influences on students' GCSE attainment and progress at age 16 (PDF, 4.52MB) published by the Department for Education. In this report, surveys of Year 9 students were held, and they found that "attending a secondary school which placed a greater 'emphasis on learning' and had a more positive 'behaviour climate' - as they were perceived by students - showed positive effects on total GCSE score" (page 105).

Figure 2 shows the application of the new "cohort split" approach, using the weights in Table 5. This is contrasted with the current method, where GSCE results are applied only to the year in which they are sat, using the new data sources for England as outlined in Section 2.

The "cohort split" approach has smoothed the series and shifted it left (as attainment is partly attributed to earlier years of schooling, the higher attainment is shown to be attributed further back in the time series, rather than only in the year immediately before the grades were obtained). 

This method is robust to changes to the weighting specification. We replicated the process using other specifications such as:

• using equal weighting (20%) across all year groups
• using incremental increases from year to year, by 5 percentage points, and starting at 10% for Year 7 ( 15% for Year 8 and so on)
• using a Key Stage 4 weighting pattern, with 50% each on Year 10 and Year 11

Using these different specifications had a limited effect on the series. They tend to track closely, although we do observe some variation, depending on how heavily the earlier years of schooling are weighted. As the weights given in Table 2 are those most supported by the available literature (and considering the lack of specific literature on this topic), we will use this weighting specification.

The upcoming annual estimates are for calendar year 2017. The latest attainment data we would typically use are for the academic year 2017 to 18 (to be splined to convert to calendar year 2017). However, this would mean that we are only using one year's worth of attainment data for the latest year, with the weightings rescaled so that all weight is placed on Year 11. To introduce the "cohort split" approach for 2017, we will use academic year 2018 to 2019 attainment data where available, attributing the relevant proportion back to the academic year 2017 to 2018 (that is, Year 10 for those students sitting the exams in the academic year 2018 to 2019).

Future improvements to education quality adjustment

In addition to the changes outlined in this methods article, we are planning on exploring further developments to the education quality adjustment in the future. Attainment data shows the effectiveness of the education system at facilitating the achievement of qualifications by its students. Obtaining these qualifications is an important desirable outcome of education service provision. However, it is not the only desirable outcome. For example, schooling should help students develop social skills, support their wellbeing, and prepare them for the workforce. The extent to which schools are responsible for these outcomes will warrant thorough investigation.

This may include assessing data from Ofsted, employment-related indicators, well-being and mental health data, and others. For example, Ofsted data could show the standards of schools as an environment in which to learn, while well-being data could show the potential negative effects of education on students' health. It is unlikely that any one indicator provides a full sense of education quality - attainment data are the current best option, but combining these data with one or more additional indicators of education quality would help to develop a more holistic quality adjustment.

For the "cohort split" approach described in Section 3, it should be noted that we currently apply the attainment adjustment to both secondary and primary school output. We will use the new attainment data sources and the "cohort split" approach across the five years of secondary school to calculate the quality index in upcoming publications, and apply this index to secondary and primary output. We will investigate alternative options for primary education output in the future.

We will make a few changes to the measurement of public service healthcare productivity:

• the introduction of a "number of days adjustment" to output to account for the effects of leap years and year-to-year changes in the number of working days
• the introduction of a new deflator for intermediate goods and services consumption
• the incorporation of expenditure on NHS "bank" staff into labour inputs

Number of days adjustment

As healthcare output is calculated using a cost-weighted activity index, annual healthcare output may vary according to the number of days available to carry out activities during any year. The total annual number of days varies with leap years, while the number of bank holidays and how weekends fall during the year influence the annual number of working days.

Both the total number of days and the number of working days can affect the annual output of different parts of the healthcare service. The effect is particularly notable when using financial year (April to March) data, such as that which form the basis of our healthcare productivity measure, as some financial years may contain four Easter bank holidays and others none.

We therefore intend to introduce an adjustment to output to remove the effect of changes in the annual number of total days and number of working days on healthcare output.

New deflator for intermediate goods and services consumption

In common with inputs for other services sectors in public service productivity, healthcare inputs are produced in volume terms by deflating expenditure by the most relevant available deflator. In previous editions of the Office for National Statistics (ONS) public service healthcare productivity publication, the main deflator used to calculate goods and services inputs was the Health Service Cost Index (HSCI). The HSCI was produced by the Department of Health, but was discontinued in 2017, with a final data point of March 2017.

To replace the HSCI, a new NHS Cost Inflation Index has been developed by the Department of Health and Social Care (DHSC), in conjunction with NHS England and NHS Improvement, the Centre for Health Economics at the University of York, and the ONS. The NHS Cost Inflation Index uses a range of NHS and ONS data and different components of the index will be used to deflate appropriate elements of our healthcare productivity inputs.

We plan to publish the NHS Cost Inflation Index in an annex alongside the public service healthcare productivity release and more information will be provided in a separate article.

NHS "bank" staff

In the NHS, "bank" staff fulfil a similar role to agency staff, working variable hours in response to demand. However, unlike agency staff, bank staff are NHS employees. In the next publication of Public service productivity: healthcare, we will include bank staff in our labour inputs for England using deflated expenditure data from the NHS, starting from financial year ending 2016.

There are two small changes to the quality adjustment for public order and safety (POS) for the forthcoming release, outlined in this section. There is also an improvement to the labour deflator.

Quality adjustment

Prisons are adjusted for quality using three metrics: recidivism (severity adjusted re-offending rates), escapes from prison, and safety in prison. For safety, we construct an index from growth rates of the number of occurrences each year of slight incidents, serious incidents, and fatalities. The source data, from the Ministry of Justice, is split by self-inflicted and assaults for the two incidents types. We have been able to extend the series for the assaults data back to 2000, from the currently used year of 2004.

For the upcoming publication, the Quarter 4 (Oct to Dec) 2017 re-offending source data are unavailable from the Ministry of Justice as a result of data collection updates. We are forecasting the final quarter based on the data since 2012 which were collected and reported on a consistent basis. Sensitivity checks suggest that the quality adjustment is robust to various forecasting approaches, and that the method we have used produces sensible results.

Labour deflator

We have also improved how we deflate POS labour inputs. We currently use the Index of Labour Costs per Hour (ILCH) for the public sector to deflate expenditure on labour in POS. The POS service area covers activities that employ a unique set of occupations, which makes a more specific and accurate deflator possible. For example, a large component of POS is the Fire and Rescue Services. Firefighters might be expected to have wage growth that differs to the public sector as a whole, given their employment conditions.

To improve the POS labour deflators, we have used published data from the Home Office, Ministry of Justice and HM Prisons and Probation Services on the workforces of the POS service area. The breakdown of employment by occupation, role, grade or job title have been used to weight the growth rates in wages of specific occupations, as recorded in the Annual Survey of Hours and Earnings (ASHE). We will implement these new deflators for the fire and prisons elements of POS, which collectively make up around 60% of labour expenditure in POS.

Analysis indicates that the new deflators better reflect the trends in wages and the volume of labour in these areas, and thus will produce more accurate estimates of productivity. The remainder of POS, covering courts and other legal activities, will continue to be deflated by the ILCH public sector index.

Currently, labour expenditure on local government and central government policing is deflated to approximate the volume of labour inputs. For local government expenditure, ASHE (Annual Survey of Hours and Earnings) data and police workforce data are used to calculate an appropriate deflator. 

We will now estimate the volume of local government labour directly from data on full-time equivalent employees (FTEs) and relative salaries for different groups. The main advantage of this change is to incorporate more recent data from Her Majesty's Inspectorate of Constabulary and Fire & Rescue Services (HMICFRS), which disaggregates FTEs by policing function (for example, neighbourhood policing, road policing or investigations). This is consistent with principles set out in the Atkinson Review (PDF, 1.1MB), where it is recommended that the measurement of inputs should be as comprehensive as possible.

In the forthcoming release, we will be using government expenditure data consistent with Blue Book 2019. Previously we have used the European System of Accounts (ESA) Table 11 (General government annual expenditure) data. ESA Table 11 is published earlier in the year but is based on the previous year's Blue Book, and has some slight accounting differences to the National Accounts.

The Office for National Statistics (ONS) has, along with other developments, improved its estimation of capital consumption and of government purchases of goods and services. These updates have increased the consistency and accuracy of inputs data. These and other methodological changes to Blue Book 2019 are detailed in an impact article and Blue Book 2019 itself was published in October 2019.

For capital consumption, asset lives have been reviewed (and found to be shorter in most cases, increasing consumption of fixed capital). For goods and services, the scope of the method to estimate the cost of Value Added Tax (which the government does not pay) for consistency with private purchases in the National Accounts has been widened.

We have also made some more general systems improvements to maintain best practice and improve consistency across the different aspects of our processing. In particular, we have improved the consistency of the construction of our deflators in our processing system, leading to minor revisions to other deflators (as well as the deflator development described elsewhere in this methods article).

These improvements described in this methods note will be incorporated into the upcoming publication Public service productivity: total, UK, 2017, to be published on 8 January 2020.

Any feedback or comments are welcome and can be sent to productivity@ons.gov.uk.

Leah Harris and Josh Martin, Office for National Statistics.

We are grateful to colleagues at the Welsh Government, Scottish Government, and Northern Ireland Department of Education for assistance with data sources. We also gratefully acknowledge support from the Department of Health and Social Care, NHS England and NHS Improvement, and the Centre for Health Economics at the University of York.