Rapid Evaluation for health and social care innovations: Challenges for “quick wins” using interrupted time series.

McCarthy, Andrew, McMeekin, Peter, Haining, Shona, Bainbridge, Lesley, Laing, Claire and Gray, Joanne (2019) Rapid Evaluation for health and social care innovations: Challenges for “quick wins” using interrupted time series. BMC Health Services Research, 19. p. 964. ISSN 1472-6963

Preview	Text s12913-019-4821-7.pdf - Published Version Available under License Creative Commons Attribution 4.0. Download (792kB) | Preview
Preview	Text Rapid Evaluation for health and social care innovations v2.pdf - Accepted Version Download (972kB) | Preview

Abstract

Background: Rapid evaluation was at the heart of National Health Service England’s evaluation strategy of the new
models of care vanguard programme. This was to facilitate the scale and spread of successful models of care
throughout the health & social care system. The aim of this paper is to compare the findings of the two evaluations
of the Enhanced health in Care Homes (EHCH) vanguard in Gateshead, one using a smaller data set for rapidity and
one using a larger longitudinal data set and to investigate the implications of the use of rapid evaluations using
interrupted time series (ITS) methods.
Methods: A quasi-experimental design study in the form of an ITS was used to evaluate the impact of the
vanguard on secondary care use. Two different models are presented differing by timeframes only. The short-term
model consisted of data for 11 months data pre and 20 months post vanguard. The long-term model consisted of
data for 23 months pre and 34 months post vanguard.
Results: The cost consequences, including the cost of running the EHCH vanguard, were estimated using both a
single tariff non-elective admissions methodology and a tariff per bed day methodology. The short-term model
estimated a monthly cost increase of £73,408 using a single tariff methodology. When using a tariff per bed day,
there was an estimated monthly cost increase of £14,315.
The long-term model had, using a single tariff for non-elective admissions, an overall cost increase of £7576 per
month. However, when using a tariff per bed-days, there was an estimated monthly cost reduction of £57,168.
Conclusions: Although it is acknowledged that there is often a need for rapid evaluations in order to identify
“quick wins” and to expedite learning within health and social care systems, we conclude that this may not be
appropriate for quasi-experimental designs estimating effect using ITS for complex interventions. Our analyses
suggests that care must be taken when conducting and interpreting the results of short-term evaluations using ITS
methods, as they may produce misleading results and may lead to a misallocation of resources.

Actions (login required)

View Item

Downloads