Second meeting, 13 March 2002
The following work in progress papers were considered:
Paper 1: The role of quantitative social surveys in evaluation research
Recently, the Government has sought to carry out social experiments which use quantitative social surveys in their evaluation. This paper describes how the Social Survey Division in the Office for National Statistics (ONS) has responded to this new demand.
Committee conclusions:
-
Members agreed that there is a need for an area centralising guidance on and providing standards for evaluation research across government. The Centre for Management and Policy Studies and other government departments such as the Cabinet Office, Her Majesty's Treasury, the Department for Work and Pensions and the Home Office already have some expertise in this area
-
It was noted that longitudinal research and panel surveys may have a lot to contribute in this area. General and longitudinal surveys can provide a reference point, enabling comparisons of data about the general population with data about evaluation respondents
-
There is a potential to learn from the experience of medical researchers in conducting 'evaluations' into medical and pharmaceutical procedures
Paper 2: Approach to measuring public confidence in NS outputs and services
New measures of public confidence in National Statistics outputs and services are under development. This paper outlines the proposed approach to measuring public confidence.
Committee conclusions:
The Methodology Advisory Committee agreed that the measure should be multidimensional, and endorsed the use of the seven European Statistical System quality attributes. They discussed the issues of:
-
users: user education is vital; non-users could provide useful feedback; international users provide a broader perspective
-
NS website: questions for users could be put on the website; the logging of clicks and user testing could be used to analyse the website
-
the academic community: topic-based user groups are effective; transparency of methodologies and problems is vital for public confidence
Paper 3: Methodological issues arising for the New Earnings Survey
The aim of the paper is to present a summary of the recent methodological issues which have arisen in the New Earnings Survey (NES), including the weighting scheme, target variable, and survey design. Some potential solutions are also presented.
Committee conclusions:
The following points were raised:
-
Weighting: a highly skewed distribution may affect the raking approach used; low paid jobs need to be weighted with care as the coverage of the NES is less representative at this end of the pay distribution
-
Target variable: outliers should be considered carefully; a range of measures including the mean and median should be published, but the headline rate should be easily understood by users and ideally unadjusted
-
Survey design: a better survey design may reduce the need to poststratify; the use of the Inter-Departmental Business Register and not the Labour Force Survey as a standard for employment figures should be reviewed; future data may help us to understand the characteristics of 'Exempt Category 3's (those likely to work elsewhere) in a given year.
Paper 4: Quality measures for time series
A project is underway in ONS to develop a comprehensive framework for quality measurement. This paper gives a preliminary list of quality measures for time series.
Committee conclusions:
The Methodology Advisory Committee discussed:
-
real world discontinuities: it was concluded that only discontinuities caused by methodologies or changes in classifications should be included, whilst briefing on real world effects should be provided, but not as a quality measure
-
standard errors for seasonally adjusted series: no National Statistical Institute has introduced regular publication of such standard errors, and given resource pressures arising from providing standard errors for unadjusted data, this is a longer-term goal. It was concluded that the provision for unadjusted data, plus a measure of revisions due to seasonal adjustment would be a major step forward
-
Turning points and trends: although the identification of turning points and the use of trends may be a subjective area, it was generally agreed that, since these concepts are of interest to users who may not be in a position to estimate trends themselves, ONS may want to undertake some analysis of the statistics, in the hope of meeting the needs of a broad range of users
