Rapid Evaluation for health and social care innovations: Challenges for “quick wins” using interrupted time series.

McCarthy, Andrew, McMeekin, Peter, Haining, Shona, Bainbridge, Lesley, Laing, Claire and Gray, Joanne (2019) Rapid Evaluation for health and social care innovations: Challenges for “quick wins” using interrupted time series. BMC Health Services Research, 19. p. 964. ISSN 1472-6963

s12913-019-4821-7.pdf - Published Version
Available under License Creative Commons Attribution 4.0.

Download (792kB) | Preview
Rapid Evaluation for health and social care innovations v2.pdf - Accepted Version

Download (972kB) | Preview
Official URL: https://doi.org/10.1186/s12913-019-4821-7


Background: Rapid evaluation was at the heart of National Health Service England’s evaluation strategy of the new
models of care vanguard programme. This was to facilitate the scale and spread of successful models of care
throughout the health & social care system. The aim of this paper is to compare the findings of the two evaluations
of the Enhanced health in Care Homes (EHCH) vanguard in Gateshead, one using a smaller data set for rapidity and
one using a larger longitudinal data set and to investigate the implications of the use of rapid evaluations using
interrupted time series (ITS) methods.
Methods: A quasi-experimental design study in the form of an ITS was used to evaluate the impact of the
vanguard on secondary care use. Two different models are presented differing by timeframes only. The short-term
model consisted of data for 11 months data pre and 20 months post vanguard. The long-term model consisted of
data for 23 months pre and 34 months post vanguard.
Results: The cost consequences, including the cost of running the EHCH vanguard, were estimated using both a
single tariff non-elective admissions methodology and a tariff per bed day methodology. The short-term model
estimated a monthly cost increase of £73,408 using a single tariff methodology. When using a tariff per bed day,
there was an estimated monthly cost increase of £14,315.
The long-term model had, using a single tariff for non-elective admissions, an overall cost increase of £7576 per
month. However, when using a tariff per bed-days, there was an estimated monthly cost reduction of £57,168.
Conclusions: Although it is acknowledged that there is often a need for rapid evaluations in order to identify
“quick wins” and to expedite learning within health and social care systems, we conclude that this may not be
appropriate for quasi-experimental designs estimating effect using ITS for complex interventions. Our analyses
suggests that care must be taken when conducting and interpreting the results of short-term evaluations using ITS
methods, as they may produce misleading results and may lead to a misallocation of resources.

Item Type: Article
Uncontrolled Keywords: Interrupted time series, Complex interventions, Health and social care, Vanguards, Service evaluation
Subjects: B700 Nursing
B900 Others in Subjects allied to Medicine
Department: Faculties > Health and Life Sciences > Nursing, Midwifery and Health
Depositing User: Elena Carlaw
Date Deposited: 12 Dec 2019 16:54
Last Modified: 31 Jul 2021 20:08
URI: http://nrl.northumbria.ac.uk/id/eprint/41734

Actions (login required)

View Item View Item


Downloads per month over past year

View more statistics