Guide to evaluation design: primciples and practice

Guide to Evaluation Design, Principles and Practice
Midlands Decision Support Centre [December 2020]

• This guide has been produced to take you through the evaluation process, and considerations, in designing and delivering a high quality service evaluation. It begins with an introduction to evaluation and how it should be used, followed by sections on: who to involve in developing the evaluation design; governance and ethics; resourcing; the main qualitative and quantitative approaches; and how to report your findings.

The road to renewal: five priorities for health and care

The road to renewal: five priorities for health and care
Kings Fund 16 July 2020

• Five priorities to help guide the approach to renewal across health and care.

1. A step change on inequalities and population health
2. Lasting reform for social care
3. Putting the workforce centre stage
4. Embedding and accelerating digital change
5. Reshaping the relationship between communities and public services

Rebuilding the NHS Improving medical pathways for acute care

Rebuilding the NHS Improving medical pathways for acute care
Royal College of Emergency Medicine, Royal College of General Practitioners, Royal College of Physicians, and the Society for Acute Medicine 13 July 2020

• A call for transformation of the urgent and emergency care pathway in light of the COVID-19 pandemic based on the key principle is that patients and carers should only have to attend hospital when it is essential or the added clinical value outweighs the risk

Evaluating social prescribing (IRISS)

Evaluating social prescribing
IRISS 22 June 2020

• A review of the research methods and approaches used to evaluate UK-based social prescribing interventions in recent years aims to inform healthcare and social services professionals, as well as organisations delivering social prescribing interventions and those conducting evaluations of them.

Understanding the provision of occupational health and work-related musculoskeletal services

Understanding the provision of occupational health and work-related musculoskeletal services
Department for Work and Pensions and Department of Health and Social Care, 28 May 2020

• This research maps the existing provision of occupational health across the UK and NHS musculoskeletal services available across England.

Clinical service accreditation - statement of intent

Clinical service accreditation - statement of intent
BSI, HQIP, UKAS October 2019

• BSI, Healthcare Quality Improvement Partnership (HQIP) and the United Kingdom Accreditation Service (UKAS) have published a statement setting out a shared intent to improve clinical services through accreditation.
• Their statement outlines their commitment to working together to:
• improve the quality and effectiveness of healthcare by providing clinical services with an infrastructure and a set of requirements that enables a structured approach to quality improvement against which they can be assessed
• encourage further development of certification schemes in clinical services and to have the providers of those schemes accredited against national standard
• integrate these initiatives with regulatory bodies (such as the Care Quality Commission (CQC) and other inspection and improvement processes so that commissioners, service providers and service users have a mechanism that supports quality improvement and provides assurance.

Evaluation of New Care Models in Mental Health

Evaluation of New Care Models in Mental Health
Niche Consult September 2019

• An independent evaluation of the 17 sites in waves 1 and 2 of the mental health New Care Models programme, including Low and medium secure services for adults, Inpatient services for children and young people (CYP) and Eating disorder services for adults. The report is based on both visits and financial and activity data and demonstrates mixed results. The report includes topic summaries for each of children and young people, and adult secure services.

Better evaluations to support the needs of older people in the UK

Better evaluations to support the needs of older people in the UK
RAND Research Report 2019

• How can evaluations of complex interventions, such as services for older people in the UK, be improved? Findings from workshops with evaluators, commissioners of evaluations and services, and those delivering services:
• Evaluations were too often not focused on major challenges and consequently findings were 'unsurprising' or even trivial
• Evaluation findings were often not available when decisions had to be made.
• Evaluations were often designed as stand-alone pieces rather than building on previous evaluations and contributing to future evaluations
• Competition, a lack of clarity in communication, and poor knowledge management/mobilisation led to a reluctance to share information, pool data, or drew on the evaluations of potential competitors.
• Evaluations often failed to use routine data and ignored the costs imposed on service providers and users

Hyperacute neurology at a regional neurosciences centre: a 1-year experience of an innovative service model.

Hyperacute neurology at a regional neurosciences centre: a 1-year experience of an innovative service model.
Clin Med (Lond). 2019 Mar;19(2):119-126. doi: 10.7861/clinmedicine.19-2-119.
This paper reports the results from the first year of the St George's Hospital hyperacute neurology service (HANS). The outcomes include admission avoidance from ED, and a reduction in length of stay for non stroke patients and reduction in occupancy of stroke beds by non-stroke cases.

Abstract

Devolving healthcare services redesign to local clinical leaders: does it work in practice?

Devolving healthcare services redesign to local clinical leaders: does it work in practice?
J Health Organ Manag. 2019 Mar 28;33(2):188-203.

• Investigation into the impact of policy changes resulting from the 2012 Health and Social Care Act on instances of service redesign by local leadership.

Abstract

Measuring general practice productivity: Development and evaluation of the GPET

Measuring general practice productivity: Development and evaluation of the general practice effectiveness tool
Kings Fund, 21 February 2019

• This study developed and tested a measure of effectiveness in general practice known as the general practice effectiveness tool (GPET), in consultation with GPs, practice staff, and members of the public. This summary outlines the findings and offers learning for practices on how they think about their effectiveness.
• GPET comprises 4 performance areas (clinical care, practice management, patient focus and external focus), which are covered by 11 objectives, measured by 52 indicators, gathered mainly from data that is routinely collected by practices.

Accelerating the Uptake of Value-Based Reforms in NHS England

Accelerating the Uptake of Value-Based Reforms in NHS England
Health Foundation 28 November 2018

• Analysis of progress and challenges of the new care models by researchers from the Duke-Margolis Center for Health Policy 
• The report describe how local areas can create a ‘care system integrator’ to develop the infrastructure needed to implement care reforms, and highlights regions already implementing care-system-integrator functions, describing the challenges and identifying steps NHS England can take to support its national care integration goals.

Impact of the CQC on provider performance

Impact of the Care Quality Commission on provider performance: room for improvement?
Kings Fund, Alliance Manchester Business School 27 September 2018

• An evaluation of the CQC approach to inspecting and rating health and social care providers across four sectors (acute care, mental health care, general practice and adult social care) that describes eight ways in which regulation can affect provider performance.
• The report examines whether Intelligent Monitoring datasets can predict the outcome of inspections and whether, on the basis of routine data, reported performance in A&E services, maternity services and general practice improves following an inspection.

General Practice

• The analysis found that the quality indicators used in CQC’s Intelligent Monitoring datasets had little or no correlation with the subsequent ratings of general practices. The model did not correctly predict any of the 270 practices that received an ‘outstanding’ rating, or the 172 practices that received an ‘inadequate’ rating. 
• The analysis found that, in the six months after practices were inspected, prescribing behaviour generally improved slightly for practices rated ‘inadequate’ or ‘requires improvement’, but worsened slightly for those rated ‘good’ or ‘outstanding’.

Adult social care at a glance

Adult social care at a glance
National Audit Office July 2018

• This overview provides an up-to-date analysis of key trends in adult social care and highlights pressures on the system and the latest developments.

Achieving the Provision of Integrated Care

Achieving the Provision of Integrated Care
NAPC June 2018

• This report of a two-day symposium attended by health experts from the UK, USA and Ireland, traces the development and impact of integrated care to date, homes in on some examples of successful practice and draws out the key lessons for later-adopters.
• Includes a short case Study of  Dorset Integrated Care System (UK)
• The report identifies factors likely to facilitate success – engaged leadership at all levels, real team working, a clear grasp of local population needs, underpinned by the appropriate tools and technology, continuous learning from experience, and community involvement, with people and patients seen as key partners. It also consider barriers that could impede progress, such as pressure for quick wins, top-down controls impeding bottom-up evolution, outdated competitive behaviours and the eventual need – at least in England – to incorporate the new approach into contracts and legislation.

Developing new care models through NHS vanguards

Developing new care models through NHS vanguards
National Audit Office 29 June 2018

• This report examines whether the NHS is well placed to get value for money from its investment in developing new care models through vanguards. In particular, it focuses on:
• set up and management of the vanguard programme (Part One);
• national support and evaluation (Part Two);
• progress made by the vanguards (Part Three); and
• readiness for the spread of these new care models (Part Four). 
• The report focuses primarily on the two types of vanguards which were designed to test integrated models of health and social care for a local population: integrated primary and acute care systems (PACs) and multispecialty community providers (MCPs)

Report conclusions

Primary Care Strategy across mid and south Essex

Primary Care Strategy
Mid and south Essex CCGs 28 June 2018

• Proposals have been published by the five CCGs in mid and south Essex which build on the development of localities/hubs within each CCG area, and aim to:
• Make it quicker and easier for patients to be seen locally by NHS professionals
• Reduce pressure on GPs so they can focus on patients with the greatest need
• Improve the recruitment of GPs and other healthcare professionals to mid and south Essex

The 15 steps challenge

15 steps challenge
NHS England

• The 15 Steps Challenge, initially developed and published by the NHS Institute for Innovation and Improvement, focuses on seeing care through a patient or carer’s eyes, and exploring their first impressions.
• A series of guides have been published (Maternity, Inpatient Services, Mental Health Inpatient Services' Community Services, Children and Young People’s Inpatient Services, Clinics and Outpatients) as well as a Guidance for Commissioners.
• The 15 steps challenge is an approach to service/quality improvement that focuses on ward (or service) ‘walkarounds’, recording first impressions from the perspective of a service user.

Economic analysis of service and delivery interventions in health care

Economic analysis of service and delivery interventions in health care
Health Serv Deliv Res 2018;6(5)

• Evaluating service innovations in health care and public health requires flexibility, collaboration and pragmatism; this collection identifies robust, innovative and mixed methods to inform such evaluations.
• "Broadly speaking, there are two approaches to producing summative assessments of the costs and benefits of entire programmes. The first involves direct estimation of the summative impacts through exploitation of some experiment or other source of variation in implementation. The second involves modelling the causal chain through component processes to derive an aggregate measure where direct estimation is not feasible." see Recent developments

Abstract

Clinical leadership in service redesign in CCGs

Clinical leadership in service redesign using Clinical Commissioning Groups: a mixed-methods study.
Health Serv Deliv Res 2018;6(2)

• This research examined clinical leadership within 15 CCGs and includes six specific service redesign case studies.

Results

• "Instances of the exercise of clinical leadership utilising the mechanism of the CCGs were strikingly varied. Some CCG teams had made little of the opportunity. However, we found other examples of clinicians stepping forward to bring about meaningful improvements in services. The most notable cases involved the design of integrated care for frail elderly patients and others with long-term conditions. The leadership of these service redesigns required cross-boundary working with primary care, secondary care, community care and social work. The processes enabling such breakthroughs required interlocking processes of leadership across three arenas: (1) strategy-level work at CCG board level, (2) mid-range operational planning and negotiation at programme board level and (3) the arena of practical implementation leadership at the point of delivery. The arena of the CCG board provided the legitimacy for strategic change; the programme boards worked through the competing logics of markets, hierarchy and networks; and the practice arena allowed the exercise of clinical leadership in practical problem-solving, detailed learning and routinisation of new ways of working at a common-sense everyday level."