Module 9 — Evaluation Methods in Practice

How healthcare systems improve confidence in evaluation findings

Module Learning Objective

This module helps explain:

how healthcare systems move beyond simple before-and-after comparisons by using more robust approaches to improve confidence that an intervention genuinely contributed to observed change.

By the end of this module, readers should feel more confident asking:

How strong is the evidence that this intervention worked?

The module focuses on practical healthcare examples, particularly interventions intended to reduce:

  • Emergency Department (ED) attendances
  • emergency admissions
  • avoidable hospital utilisation
  • delayed discharge
  • operational pressure

Rather than assuming healthcare systems can ever prove impact with complete certainty, the module focuses on:

improving confidence in evaluation findings

through more thoughtful and structured approaches.

From Module 8 to Module 9

In Module 8 we explored an important idea:

improvement alone does not prove impact

Healthcare systems frequently observe change after an intervention.

For example:

ED attendances ↓
Emergency admissions ↓

But an important question remains:

Would this improvement have happened anyway?

This idea introduced:

  • counterfactual thinking
  • confidence in evidence
  • contribution versus attribution
  • the limitations of simple before-and-after comparisons

The question now becomes:

How do healthcare systems improve confidence that an intervention genuinely contributed to observed change?

This module introduces practical approaches that are commonly used in healthcare evaluation.

These approaches do not eliminate uncertainty.

Instead, they help decision-makers ask:

How confident should we be?

Why Practical Evaluation Methods Matter

Healthcare systems often need to make decisions quickly.

Programmes are launched:

  • under operational pressure
  • during winter demand
  • amidst workforce shortages
  • alongside other service changes

Decision-makers may need to decide:

  • whether to continue funding a scheme
  • whether to scale it across the system
  • whether to redesign or stop it

Yet waiting for perfect evidence is rarely realistic.

This creates an important challenge:

How do we improve confidence in evaluation findings without waiting years for certainty?

Practical evaluation methods help by:

  • improving comparisons
  • strengthening counterfactual thinking
  • reducing misleading conclusions
  • increasing confidence in observed impact

Importantly:

stronger methods do not guarantee truth.

Instead:

they help reduce the risk of reaching the wrong conclusion.

Randomised Controlled Trials (RCTs)

One of the strongest ways to evaluate whether an intervention caused change is through a:

Randomised Controlled Trial (RCT)

RCTs are often described as the:

gold standard of evaluation

because they help create a stronger counterfactual.

What Is an RCT?

In simple terms:

an RCT compares two groups.

One group receives:

the intervention

The other receives:

usual care (or no intervention)

Participants are assigned randomly.

This is important because randomisation helps reduce bias.

In theory:

the two groups should be broadly similar at the start.

If outcomes later differ:

we can have greater confidence that:

the intervention contributed to the change

rather than:

  • chance
  • selection bias
  • population differences
  • other confounding factors

A Practical Healthcare Example

Imagine an Integrated Care Board introduces a frailty intervention designed to reduce emergency admissions.

Rather than implementing the service everywhere immediately:

several Primary Care Networks (PCNs) are selected to participate.

Patients are randomly allocated.

Group A

Receives:

  • proactive frailty review
  • medication optimisation
  • MDT support
  • community monitoring

Group B

Receives:

usual care

After twelve months:

Measure Intervention Group Usual Care
ED attendances 900 1,100
Emergency admissions 320 420

Question:

Did the intervention contribute to improvement?

Because patients were randomly allocated, we can generally have:

greater confidence that observed differences are more likely to reflect the intervention rather than underlying differences between patients.

Interpreting the Trial Result Fairly

The hypothetical counts above are illustrative, not a complete trial analysis. The groups’ sizes, follow-up time, baseline risks and uncertainty would be needed before drawing conclusions. Report comparable outcomes (such as admissions per 1,000 eligible people or person-years), an estimated effect and its confidence interval rather than interpreting counts alone. Randomisation does not guarantee identical groups in every trial; non-adherence, missing outcomes, contamination between groups and chance can still affect interpretation.

Where possible, analyse people according to their original allocation (intention-to-treat) to preserve the benefits of randomisation. A cluster-randomised trial—for example, allocating PCNs rather than individual patients—must account for similarity among patients within each cluster and requires sufficient clusters, not simply many patients.

Why Randomisation Matters

Without randomisation:

services may unintentionally select patients who are:

  • easier to help
  • more engaged
  • less clinically complex
  • more motivated

This creates a problem known as:

selection bias

For example:

a frailty service might prioritise patients who are already likely to improve.

The intervention may then appear more successful than it truly is.

Randomisation helps reduce this risk.

The key idea is simple:

RCTs attempt to improve fairness in comparison by reducing differences between groups before outcomes are compared.

random allocation improves confidence that differences in outcomes are more likely to reflect the intervention rather than underlying differences between patients.

Why RCTs Are Difficult in Real Healthcare Systems

Despite their strengths:

RCTs are often difficult to implement in real healthcare systems.

RCTs can provide strong evidence.

However:

strong evidence is not always operationally practical

Healthcare environments are messy.

Operational realities include:

  • workforce shortages
  • service redesign during implementation
  • changing pathways
  • political and operational pressure to scale interventions quickly
  • ethical concerns about withholding care

For example:

if early evidence suggests a service is beneficial:

leaders may feel uncomfortable withholding support from some patients.

Similarly:

healthcare systems often need to move quickly.

Waiting several years for a perfect evaluation may not feel realistic when:

  • Emergency Departments are under pressure
  • emergency admissions are rising
  • operational performance is deteriorating

As a result:

many healthcare evaluations rely on:

pragmatic approaches

that improve confidence without requiring a full RCT.

Key Message

RCTs are powerful because they help create stronger comparisons between intervention and non-intervention groups.

However:

strong evidence is not always operationally practical evidence

Healthcare systems often need approaches that balance:

rigour

with

operational reality

Part 1 Reflection

Before moving on, ask:

If a scheme appears successful, how confident are we that the groups being compared were genuinely comparable?

In the next section, we explore another important question:

How much evidence is enough to trust the findings?

This introduces:

sample size and statistical power

Sample Size, Statistical Power and Confidence in Findings

In Module 8, we introduced an important question:

How confident are we that an intervention genuinely contributed to observed change?

Even when groups are fairly compared, another important challenge remains:

Do we have enough evidence to trust what we are seeing?

This is where:

sample size and statistical power

become important.

Fortunately, the core idea is much simpler than the terminology suggests.

Precision, Confidence Intervals and Practical Significance

A point estimate is not the whole finding. For example, an estimated 6% reduction in admissions with a 95% confidence interval ranging from a 2% increase to a 13% reduction is compatible with materially different effects. A confidence interval describes the range produced by a statistical procedure with a specified long-run coverage property; it is not a 95% probability that the true effect lies in this particular interval.

Ask whether the range includes clinically important benefit, little change or harm. A statistically significant result need not be clinically meaningful; a non-significant result does not prove that the intervention has no effect. Power depends on expected effect size, variability, sample size, design, follow-up and—where relevant—clustering and autocorrelation.

Why Small Numbers Can Mislead

Healthcare systems often evaluate schemes using:

  • small pilots
  • short implementation periods
  • highly selected patient groups
  • limited operational data

For example:

A frailty intervention is piloted for:

20 patients

After three months:

Measure Before After
ED attendances 42 28
Emergency admissions 17 10

At first glance:

the scheme appears successful.

But an important question remains:

How confident should we be that this change reflects genuine impact rather than normal fluctuation?

Small numbers naturally fluctuate.

A handful of patients can disproportionately influence results.

For example:

if two highly complex patients experience:

  • hospitalisation
  • bereavement
  • worsening frailty
  • medication changes

outcomes may change substantially.

Similarly:

if several patients improve naturally:

results may look highly positive.

This means:

Small pilots may produce convincing-looking results that are less reliable than they first appear.

What Do We Mean by Sample Size?

Sample size simply refers to:

how many people, events or observations are included in an evaluation

Examples include:

  • number of patients enrolled in a scheme
  • number of Emergency Department attendances analysed
  • number of GP practices or PCNs involved
  • number of months of activity included

In general:

larger samples tend to produce more stable and reliable findings

because random fluctuation becomes less influential.

This does not mean:

larger samples automatically guarantee better evidence

Poor evaluation design can still produce misleading conclusions.

However:

small numbers deserve greater caution

Statistical Power (In Plain English)

Statistical power sounds technical.

But the underlying question is straightforward:

If a real improvement exists, how likely are we to detect it?

Low-powered evaluations may fail to detect genuine effects.

For example:

a frailty scheme may genuinely reduce emergency admissions by:

5%

But if:

  • only a small number of patients are included
  • follow-up time is short
  • variation is high

the evaluation may struggle to detect the change.

Decision-makers may incorrectly conclude:

The intervention didn’t work

when the evaluation simply lacked enough information.

In practice:

small evaluations are more likely to miss real effects — or overreact to random fluctuation.

A Practical Healthcare Example

Imagine two evaluations of the same frailty pathway.

Evaluation A

20 patients
2 months follow-up

Observed result:

Emergency admissions ↓

Question:

Was this meaningful improvement — or random fluctuation?

Confidence remains limited.

Evaluation B

2,000 patients
12 months follow-up
Comparator group included

Observed result:

Emergency admissions ↓

Question:

How confident are we now?

Confidence is greater because:

  • more patients were observed
  • longer time period analysed
  • natural fluctuation matters less
  • comparison is stronger

This does not guarantee truth.

But it increases confidence.

The image below illustrates why small evaluations often produce more unstable and uncertain findings — even when results initially look promising.

Why This Matters for Healthcare Decision-Making

Healthcare systems often pilot schemes quickly.

Leaders may face pressure to decide:

  • continue funding
  • scale across the system
  • redesign
  • stop

before strong evidence exists.

This creates an important tension:

speed of learning

versus

confidence in conclusions

The question is rarely:

Can we be completely certain?

Instead:

How confident should we be before acting?

Regression to the Mean

Even when evaluations include enough observations, another important challenge remains:

could improvement have happened naturally anyway?

This brings us to:

Regression to the Mean

The name sounds complicated.

The idea is not.

It describes a simple tendency:

extreme outcomes often become less extreme over time

even without intervention.

This matters enormously in healthcare.

Because interventions are often targeted at:

  • high-risk patients
  • high-intensity users
  • people with repeated admissions
  • people with severe frailty

In other words:

the very people already experiencing unusually extreme outcomes

A Practical Healthcare Example

Imagine a healthcare system identifies:

20 patients

with very high Emergency Department attendance.

Each attended:

10–15 times

during the previous year.

A targeted intervention is introduced.

For example:

  • care coordination
  • frailty MDT support
  • proactive monitoring
  • social prescribing
  • medication review

One year later:

ED attendance falls.

Question:

Did the intervention work?

Maybe.

But another possibility exists.

Patients selected because they experienced unusually extreme healthcare utilisation may naturally move closer to average levels over time.

Some people improve naturally because:

  • acute illness resolves
  • social circumstances stabilise
  • medication changes occur
  • temporary crises pass

This means:

some improvement may occur even if the intervention had little or no effect

Why Regression to the Mean Matters

Without recognising regression to the mean:

healthcare systems may mistakenly conclude:

The scheme worked

when some improvement might have happened anyway.

This is particularly important when evaluating:

  • high-intensity ED attenders
  • frequent emergency admissions
  • severe frailty cohorts
  • complex multimorbidity pathways

The higher the starting level of risk:

the greater the risk of mistaking natural fluctuation for intervention impact

The image below illustrates an important evaluation risk:

improvement does not always mean intervention impact.

Why This Matters for Healthcare Decision-Making

Healthcare systems frequently target interventions at patients with the highest levels of need.

This makes regression to the mean particularly important.

Without recognising it, decision-makers may:

  • overestimate intervention impact
  • scale ineffective schemes
  • misallocate scarce resources

A useful question becomes:

Would some improvement have happened anyway?

Key Message

Small pilots and extreme patient groups can easily mislead evaluation.

Healthcare systems should therefore ask:

Are we observing genuine intervention impact — or natural fluctuation?

Good evaluation becomes stronger when we ask:

Do we have enough evidence?

and

Could improvement have happened anyway?

Part 2 — Confidence, Small Numbers & Natural Fluctuation

Before moving on, ask:

If outcomes improved, how confident are we that this reflects intervention impact rather than small numbers or natural fluctuation?

In the next section, we explore practical ways healthcare systems strengthen confidence in evaluation findings.

Part 3 — Strengthening Evaluation in Real-World Healthcare

In the preceding sections, we explored:

  • fair comparison through Randomised Controlled Trials (RCTs)
  • why small numbers can mislead
  • statistical power
  • Regression to the Mean

Yet another challenge remains.

Healthcare systems rarely operate in perfect conditions.

Leaders often need to evaluate schemes:

  • quickly
  • under operational pressure
  • without randomisation
  • while multiple changes happen simultaneously

In reality:

RCTs are often difficult, expensive or impractical

This creates an important question:

How can healthcare systems strengthen evaluation when RCTs are unrealistic?

Two practical approaches commonly used in healthcare evaluation are:

Interrupted Time Series (ITS)

and

Difference-in-Differences (DiD)

These approaches do not eliminate uncertainty.

Instead:

they improve confidence in findings by creating stronger comparisons.

Interrupted Time Series (ITS)

One challenge with simple before-and-after evaluation is this:

healthcare systems naturally fluctuate over time.

Emergency admissions rise and fall.

Waiting lists change.

Operational pressure changes.

Winter pressures come and go.

This creates an important risk:

improvement may already have been happening before an intervention started

Interrupted Time Series (ITS) helps address this problem.

What Is Interrupted Time Series?

Rather than asking:

Did outcomes improve?

ITS asks:

Did the pattern of change meaningfully alter after the intervention?

This distinction matters.

Instead of simply comparing:

before
vs
after

ITS asks:

Was the trajectory interrupted?

In simple terms:

ITS looks at:

  • trends before intervention
  • trends after intervention
  • whether the pattern changed meaningfully

For example:

Did emergency admissions:

  • suddenly reduce?
  • begin falling faster?
  • stop increasing?
  • improve in a sustained way?

A Practical Healthcare Example

Imagine an Integrated Care Board introduces a frailty pathway to reduce emergency admissions.

Instead of comparing:

before vs after

we examine:

monthly emergency admissions over time

Suppose the intervention begins in:

Month 13 (after 12 monthly baseline observations, still a short pre-period)

Question:

Did emergency admissions meaningfully change after implementation?

Three possibilities might emerge:

Scenario 1 — No Meaningful Change

Admissions were already falling.

The trend simply continues.

Question:

Did the intervention add anything?

Confidence in intervention impact remains limited.

Scenario 2 — Sudden Change

Admissions drop sharply immediately after implementation.

Question:

Could this represent intervention impact?

This is consistent with an intervention effect, but a concurrent change could also explain it.

Scenario 3 — Sustained Improvement

Admissions begin falling faster after implementation and remain lower over time.

Question:

Does this suggest a meaningful shift in trajectory?

A sustained shift may strengthen the case for an effect, but only if alternative explanations and model assumptions have been examined.

The image below illustrates the key question ITS asks:

Did the trajectory meaningfully change after intervention?

Interrupted Time Series helps distinguish:

normal change over time

from

potential intervention impact

Why ITS Matters for Healthcare Decision-Making

Healthcare systems often mistake:

normal trends

for

intervention impact

For example:

Emergency admissions may naturally improve after winter.

Operational performance may recover after temporary disruption.

Waiting lists may already have been stabilising.

Without considering trends:

decision-makers risk concluding:

The scheme worked

when outcomes may have improved anyway.

A useful question becomes:

Did outcomes improve — or did the trajectory change?

What an ITS Actually Estimates

An ITS generally uses repeated observations before and after a defined change to estimate a shift in level (an immediate difference) and/or slope (a change in trend). The projected pre-intervention trend provides a model-based counterfactual; it is not observed fact. An ITS can be fitted retrospectively when sufficiently reliable historical data exist.

An analyst should consider seasonality, serial correlation (nearby months are not independent), changes in population size or need, shifts in coding and measurement, delayed or phased implementation, and other policies introduced at the same time. A controlled ITS adds a suitable comparison series to help distinguish intervention-associated changes from wider system changes. A short pre-period or only a few observations after implementation makes estimates fragile; there is no universally sufficient number of time points.

WarningA visible change is not proof of causation

A sharp fall in admissions following launch is not necessarily evidence of impact if admission thresholds changed at the same time. Likewise, a sustained change may reflect wider system trends. Examine alternative explanations and uncertainty before describing an ITS estimate as causal.

Using Interrupted Time Series in Practice

Interrupted Time Series is one of the strongest quasi-experimental designs available when randomisation is not possible.

However, choosing an appropriate ITS approach requires careful consideration of both the data and the evaluation question.

Before fitting an ITS model, analysts should ask:

  • Do we have enough observations before and after the intervention?
  • Is there a clearly defined intervention date?
  • Were other important changes introduced at the same time?
  • Is there evidence of seasonality (for example, winter pressures)?
  • Are outcomes measured consistently over time?
  • Has the underlying trend been stable before the intervention?
  • Could changes simply reflect natural variation rather than intervention impact?

These questions help determine whether Interrupted Time Series is likely to provide a meaningful estimate of intervention impact.

TipChoosing the Right Evaluation Design

Good evaluation begins with the evaluation question, not the statistical model.

The aim is not simply to fit an Interrupted Time Series model.

The aim is to select a design that provides the most credible estimate of what would probably have happened had the intervention not occurred.

Evaluating Multiple Organisations

Healthcare evaluations often involve several hospitals, Primary Care Networks or localities.

This introduces additional complexity.

For example:

  • Different organisations may begin with different baseline performance.
  • Improvement may already be occurring at different rates.
  • Interventions may be implemented slightly differently.
  • Local operational pressures may vary considerably.

There is rarely a single correct analytical approach.

Depending on the evaluation question and available data, analysts may choose to:

  • analyse each organisation separately
  • fit a single model that accounts for organisational differences
  • combine findings across organisations using evidence synthesis techniques

The choice depends on the evaluation question, study design and available data.

The important principle is not selecting the most complex model, but choosing the approach that provides the most credible answer to the evaluation question.

NoteLooking Ahead

Real-world healthcare improvement rarely involves a single intervention delivered in a single organisation.

Later in this series, Module 17: Understanding What Made the Difference explores more advanced evaluation challenges, including:

  • multiple interventions introduced at the same time
  • phased implementation across organisations
  • multi-site evaluations
  • mixed-method evaluation
  • attribution versus contribution

These situations require careful evaluation design as well as appropriate statistical methods.

Difference-in-Differences (DiD)

Another practical challenge in healthcare evaluation is this:

Compared with what?

Even when outcomes improve:

another question remains:

Would similar improvement have happened elsewhere anyway?

Difference-in-Differences (DiD) helps strengthen confidence by introducing:

a comparator group

What Is Difference-in-Differences?

Rather than asking:

Did outcomes improve?

DiD asks:

Did outcomes improve more where the intervention happened?

In simple terms:

we compare:

change
vs
change

rather than:

before
vs
after

The idea is simple:

If two similar groups experience similar pressures —

but only one receives the intervention —

then differences in improvement may provide stronger evidence of impact.

A Practical Healthcare Example

Imagine:

PCN A

Introduces a frailty pathway.

PCN B

Does not.

Both experience:

  • winter pressure
  • workforce shortages
  • rising demand

After twelve months:

PCN A

Emergency admissions ↓ 15%

PCN B

Emergency admissions ↓ 5%

Question:

Did the intervention contribute to additional improvement?

The key question becomes:

Was improvement greater where the intervention occurred?

This provides stronger evidence than:

before-and-after comparison alone

The image below illustrates the core idea behind Difference-in-Differences:

good evaluation compares change — not just outcomes

Why DiD Matters for Healthcare Decision-Making

Healthcare systems frequently implement interventions:

  • gradually
  • in some places before others
  • under different operational conditions

This makes DiD especially useful when:

  • RCTs are unrealistic
  • phased roll-out exists
  • comparator organisations or places are available

Without comparators:

decision-makers risk attributing:

system-wide improvement

to

local intervention impact

A useful question becomes:

Would similar improvement have happened elsewhere anyway?

Key Message

Good evaluation becomes stronger when healthcare systems ask:

Did trajectories change?

and

Compared with what?

rather than simply asking:

Did outcomes improve?

Pragmatic evaluation methods such as:

Interrupted Time Series (ITS)

and

Difference-in-Differences (DiD)

help strengthen confidence when Randomised Controlled Trials are not practical.

Part 3 Reflection

Before acting on evaluation findings, ask:

Could this improvement reflect an existing trend — or would similar change have happened elsewhere anyway?

Good evaluation moves beyond:

Did outcomes improve?

to ask:

How confident are we that the intervention genuinely contributed to change?

Choosing an Evaluation When the Scheme Has Already Started

An ICB may commission an evaluation in October 2026 for a pathway launched in April 2024. No evaluation plan was written at launch. This does not automatically mean that causal evaluation is impossible: historical records may allow the eligible population, outcomes, intervention timing and a suitable comparison to be reconstructed.

What can be recovered? Sensible approach Claim it may support
Consistent monthly data before and after implementation Retrospective ITS; controlled ITS if a credible comparator exists Estimated level or trend change, subject to assumptions
A credible untreated or later-treated population with pre- and post-data DiD or suitable phased-rollout analysis Relative change, potentially causal under identifying assumptions
Patient-level eligibility, exposure and outcomes Longitudinal observational design with careful adjustment Association; causal effect only with a credible identification strategy
Only post-implementation data, no defensible comparison Descriptive, process and theory-based evaluation What happened, how delivery worked and plausible contribution—not an attributable impact estimate
Future expansion or pathway redesign is planned Prospective design, possibly controlled or randomised phased rollout Stronger evidence about the next change, if feasible and appropriately designed
ImportantDo not manufacture a counterfactual

A projection from historical data is only a useful counterfactual if its assumptions are defensible. National averages, neighbouring ICBs or model predictions are not automatically valid comparisons. If no credible causal design is available, evaluate implementation, reach, patient and carer experience, outcomes, equity and plausible mechanisms—and report the limits clearly.

An Evaluation Design Checklist

Before selecting a statistical technique, record:

  1. Decision and estimand: What commissioning decision is at stake, and whose outcome under what intervention versus what alternative are we trying to estimate?
  2. Population and eligibility: Who was eligible, including those who did not take up the offer? Are cohorts fixed at baseline or updated over time?
  3. Intervention and timing: What changed, when, for whom and with what implementation intensity? Was introduction gradual?
  4. Outcome and follow-up: Are the outcomes meaningful, measured consistently and assessed over a suitable horizon? Include safety and unintended consequences.
  5. Comparison and assumptions: Is a counterfactual plausible? What selection, confounding, spillover or simultaneous changes threaten it?
  6. Data and precision: Are denominators, missingness, seasonality, clustering and statistical uncertainty adequately addressed?
  7. Equity and complexity: Could results vary by frailty, age, deprivation or place? How will changing need, mortality and competing events be considered?
  8. Triangulation and use: What do patients, carers and staff say? What implementation and economic evidence is needed? What decision would different findings support?

Different Questions Require Different Evidence

Causal impact evaluation asks whether an intervention made a difference compared with an alternative. Process evaluation asks whether it was delivered as intended. Theory-based or realist evaluation examines how, why and under what circumstances outcomes may arise. Equity analysis asks who benefits or is missed. Economic evaluation asks whether benefits justify resources and opportunity costs.

These approaches complement rather than substitute for one another. Interviews and implementation evidence can strengthen an explanation, but do not by themselves establish the number of admissions prevented. Conversely, a credible average impact estimate does not reveal why the pathway worked, whether patients experienced better care or whether it was worth the investment.

For an integrated frailty portfolio, start by assessing impact at the eligible cohort/model-of-care level, where the causal design permits. Pathway analysis may explain mechanisms and variation, but comparing patients who received different components rarely isolates each component’s causal effect because clinical need influences exposure. Module 17 develops this issue in detail.

Sensitivity Analysis: Would the Conclusion Survive Reasonable Alternatives?

A single result should not be presented as certain. Test how findings change when plausible assumptions vary—for example, different start dates, pre-intervention periods, comparison localities, eligible cohorts, seasonal specifications or adjustment for changing population need. Report the range of credible estimates, not only the most favourable one. These checks do not rescue a fundamentally invalid design, and scenario ranges are not the same as statistical confidence intervals.

Match the Claim to the Evidence

Evidence available Proportionate statement
Only post-implementation monitoring “Admissions were lower during the reporting period; causation cannot be established.”
Pre/post comparison only “Admissions fell following implementation, but other explanations remain.”
ITS with defensible assumptions “The model estimates a change in admissions beyond the projected prior trend, subject to model assumptions.”
Credible controlled evaluation “The estimated impact relative to the comparison was X, with uncertainty and assumptions reported.”
Theory-based mixed-method evidence “Findings support a plausible contribution to improved coordination, but do not quantify attributable admissions avoided.”

From Evaluation Question to Evaluation Protocol

A useful evaluation begins before choosing a statistical model. A short, written protocol makes the analysis transparent and reduces the temptation to change outcomes, time periods or comparison groups after seeing favourable results. If the scheme is already running, register the retrospective protocol before analysing outcomes where feasible, and disclose any exploratory decisions made after inspecting the data.

Seven Steps for a Proportionate NHS Evaluation

Step What to document Example: community frailty pathway
1. Commissioning decision Continue, redesign, expand or stop? By when? Decide whether to extend the pathway to other PCNs
2. Target effect (estimand) Eligible population, intervention, comparator, outcome and follow-up Difference in 12-month emergency admission rate for eligible older adults if offered the pathway versus usual care
3. Theory of change What should change first, and why? Earlier assessment → quicker community response → fewer avoidable crises
4. Data and feasibility Eligibility, dates, exposure, outcomes, denominators, missingness, linkage and approvals Monthly admission data, GP population registers and service referral logs
5. Design and threats Comparison strategy, assumptions, competing changes, contamination Controlled ITS; check whether comparator PCNs received similar services
6. Analysis plan Primary outcome, subgroups, uncertainty, sensitivity analyses and stopping rules Pre-specify 12-month rate and adjustment for seasonality
7. Reporting and use Limitations, implementation, patient experience, equity, costs and decision thresholds Report both impact estimate and whether the pathway reached those in greatest need
TipThe estimand: specify what “impact” means

“Did the pathway work?” is too vague for analysis. An estimand is the precise effect the evaluation seeks to estimate: for which population, under which intervention compared with which alternative, over what period, on which outcome?

For example, the effect of offering the pathway to everyone eligible differs from the effect among people who actually received intensive MDT support. These are different questions and may require different designs.

Define the Population, Denominator and Outcome Fairly

Raw activity counts can fall because the population at risk shrank, fewer people were eligible, patients died or records became incomplete. Where appropriate, compare rates using a clearly defined denominator or person-time rather than counts alone.

Measure Potential problem Better question
Total admissions Changing population size or case mix Admissions per 1,000 eligible people or person-years, with suitable adjustment
Admissions among pathway users People selected because they recently had a crisis What happened to all eligible people, including those not referred?
Average admissions per patient Follow-up differs because people enter at different dates How much person-time was each patient observed for?
Admission rate among survivors Death removes high-risk people from observation Were mortality and competing events considered alongside admissions?
Crude comparison between localities Age, deprivation and frailty profiles differ How would findings change after appropriate standardisation or adjustment?

For frailty and end-of-life pathways, interpret activity together with need, complexity, mortality, safety and patient goals. A lower admission rate is not automatically a better outcome. Avoid conditioning the primary analysis only on surviving patients when survival may itself be affected by the intervention.

Bias and Confounding in Observational Evaluation

When random allocation is not possible, comparing those who received a service with those who did not is often misleading. Confounding by indication occurs when the clinical reasons for providing the service also predict outcomes. A frailty MDT may preferentially see people at highest risk of admission; their higher admission rate does not prove the MDT caused harm. Conversely, selective enrolment of easier-to-support patients may exaggerate apparent benefits.

Using a Causal Diagram (DAG) to Think Before Adjusting

A Directed Acyclic Graph (DAG) is a simple map of hypothesised cause-and-effect relationships. It can help analysts identify variables that might confound an exposure–outcome relationship, and variables that should not automatically be adjusted for.

Baseline frailty / illness severity ──→ MDT referral
             │                            │
             └────────────────────────────→ Emergency admission
                                          ↑
                                  Potential MDT effect

Here baseline severity influences both referral and admissions. A fair comparison may need to account for it. But a measure changed by the MDT—such as improved medication management—may be part of the pathway’s effect; adjusting for it could remove some of the effect being studied. Variables affected by both exposure and outcome-related factors can also introduce collider bias when conditioned on. DAGs clarify assumptions; they do not prove those assumptions are true.

Adjustment, Matching and Weighting: What They Can and Cannot Do

Approach Basic idea Important limitation
Regression adjustment Compare outcomes while accounting for measured baseline differences Cannot remove bias from unmeasured or badly measured confounders
Propensity-score matching Match exposed and unexposed people with similar estimated probability of exposure May discard patients and cannot correct unmeasured confounding
Propensity-score weighting Reweight groups to improve balance on measured baseline characteristics Extreme weights and lack of overlap can make estimates unstable
Stratification or standardisation Compare within need/age strata or apply a common population distribution Requires adequate information and comparable strata

Practical checks: Are relevant variables measured before treatment? Do intervention and comparison groups overlap in characteristics (positivity)? Are important baseline differences balanced after matching or weighting? Are results robust to alternative adjustment choices? Never present statistical adjustment alone as proof of causality.

When exposure and confounding change over time

In a longer pathway, clinical deterioration may lead to additional support; that support may then alter later deterioration and admission risk. Ordinary regression adjustment for these time-varying factors can introduce bias. Methods such as marginal structural models with inverse-probability weighting or other longitudinal causal approaches may be appropriate, but require specialist design, reliable longitudinal data and strong assumptions. This is a sign to involve experienced evaluators rather than simply adding more variables to a model.

A Common Retrospective Trap: Immortal Time Bias

Imagine defining the intervention group as patients who received an MDT review within 90 days, while counting outcomes from the initial eligibility date. To qualify as “reviewed”, patients must remain eligible and observable until their review; those admitted or dying before review may be disproportionately assigned elsewhere or excluded. This can make the intervention appear protective even when it is not.

Avoid this by defining a common time zero (the date eligibility and follow-up begin), aligning eligibility and treatment strategies, and handling treatment initiation over time appropriately. In some settings a target trial emulation framework—explicitly specifying the trial we would ideally run—helps prevent these errors. Merely adjusting for age or frailty will not fix incorrect time alignment.

Missing Data, Data Linkage and Repeated Observations

An evaluation can have an elegant design but still produce biased results if data are incomplete or inconsistently captured.

What Is Missing—and Why?

  • Missing completely at random (MCAR): missingness is unrelated to measured or unmeasured values; uncommon in operational data.
  • Missing at random (MAR): after conditioning on observed information, missingness does not depend on the missing value itself. Methods such as multiple imputation may be reasonable if assumptions are credible.
  • Missing not at random (MNAR): missingness still depends on unseen values or other unobserved factors. Sensitivity analysis is particularly important.

For example, frailer patients may be less likely to complete experience questionnaires. Analysing only respondents could overstate satisfaction. Report the amount, pattern and likely causes of missing data; compare characteristics of people with and without records; and test plausible alternative assumptions. Do not automatically replace missing values with zero.

Linkage and Measurement Quality

Before analysis, check whether NHS numbers or other permitted identifiers link reliably, whether patient residence and service boundaries changed, whether coding definitions stayed stable, and whether data flows were disrupted. Use appropriate information governance, secure processing and data minimisation. Poor linkage may disproportionately omit people who move, receive care across boundaries or use multiple services.

Patients, PCNs and Months Are Not Independent

A patient may have repeated attendances; patients within a PCN share staff and pathways; monthly outcomes may be correlated with adjacent months. Standard errors that assume every observation is independent may be too small. Depending on the design, consider cluster-robust uncertainty estimates, mixed-effects models, generalised estimating equations or time-series models that address autocorrelation. With only a few clusters, ordinary cluster-robust methods may be unreliable; seek specialist advice.

The effective amount of information depends on the number of independent units, not just the total number of records. Ten thousand admissions from two PCNs do not provide the same comparative evidence as data from many independently allocated PCNs.

Testing Whether the Result Is Robust

Sensitivity analysis should be planned in advance where possible and distinguish statistical uncertainty from uncertainty about the design and assumptions.

Check Example What it can reveal
Alternative time windows Repeat ITS excluding an unusual pandemic or data-disruption period Dependence on selected history
Alternative implementation dates Treat rollout as gradual rather than a single abrupt change Mis-specified timing
Alternative comparators Compare against several plausible localities Sensitivity to comparator choice
Placebo intervention date Test for an apparent “effect” before the real launch Unmodelled trends or spurious breaks
Negative-control outcome Examine an outcome the pathway should not plausibly affect Shared confounding or coding artefacts, if the control is genuinely suitable
Missing-data scenarios Compare complete-case, imputed and plausible MNAR assumptions Sensitivity to missingness
Exposure and cohort definitions Compare offer-based versus receipt-based definitions where valid Selection and timing effects

A placebo or negative-control result does not validate causality by itself. Conversely, failure of one diagnostic is a reason to investigate and qualify the claim. Report all material analyses, not only those supporting the preferred conclusion.

A Worked Evaluation Protocol: Frailty Pathway Already in Place

Illustrative situation: An ICB introduced proactive frailty care in April 2024 and requests an evaluation in October 2026. The aim is to inform whether to expand the model. This is an example, not a claim about any actual ICB.

Protocol element Proposed specification
Decision Whether to expand, adapt or retain the frailty pathway
Population All people meeting a consistent frailty eligibility definition in relevant PCNs, including those not referred
Intervention Availability of the integrated pathway, with actual rollout dates and coverage documented
Comparator Usual care in credible comparison PCNs, if available; otherwise a carefully assessed historical trend
Primary outcome Emergency admissions per 1,000 eligible person-years, with clear admission coding
Secondary outcomes ED attendances, bed days, mortality, experience, unmet need and costs
Design Prefer controlled ITS where comparable data and assumptions support it; otherwise consider ITS or non-causal evaluation
Bias checks Baseline need, deprivation, secular trends, seasonality, coding, contamination and changing eligibility
Analysis Pre-specify level/trend terms, uncertainty, clustering/autocorrelation and sensitivity checks
Qualitative work Ask patients, carers and staff about access, coordination, burden, implementation and unintended effects
Interpretation Distinguish estimated pathway impact from descriptive change; do not attribute separate effects to overlapping components without an appropriate design

If only post-April-2024 data can be obtained and no credible comparator exists, the team should not calculate “admissions avoided” by treating an arbitrary benchmark as what would otherwise have happened. It can still report reach, implementation, outcomes, equity, patient experience, cost and plausible contribution, and design stronger evaluation around the next pathway change.

NoteConnecting to Module 17

Module 9 provides the methodological toolkit and protocol discipline. Module 17 — Understanding What Made the Difference applies these principles when several pathways overlap, their effects may interact and commissioners need to judge impact, explanation, value and equity at the integrated-model level. The same admissions avoided must not be credited independently to every pathway component.

Matching the Evaluation Method to the Question

One of the most important principles in evaluation is that the evaluation question should determine the evaluation method.

There is rarely a single “best” evaluation method.

Instead, different methods answer different questions.

Underlying them all is one fundamental idea:

What would probably have happened if the intervention had not taken place?

This is known as the counterfactual.

Different evaluation methods estimate the counterfactual in different ways.

Evaluation Question What evidence is needed? Methods Commonly Used
Did it work? Compare outcomes with what would probably have happened without the intervention. Randomised Controlled Trial (RCT), Interrupted Time Series (ITS), Difference-in-Differences (DiD), Controlled ITS
How was it implemented? Examine reach, fidelity, adaptations, resources and experience. Process evaluation, implementation evaluation, qualitative interviews
Why did it work (or not)? Explore mechanisms, contexts and alternative explanations. Qualitative research, Realist Evaluation
Did it work consistently across organisations? Compare patterns of change across multiple organisations while accounting for differences in starting position, trends and implementation. Multi-site evaluation, mixed-effects models, meta-analysis
Was it worth it? Assess incremental outcomes, costs and opportunity costs, using causal estimates where possible. Health economics, Cost-effectiveness analysis, cost-benefit analysis, cost-utility analysis
Should we scale it? Integrate impact, implementation, equity, costs and transferability. Mixed-method evaluation, evidence synthesis, implementation evaluation
TipKey Principle

Good evaluation starts with a clear evaluation question, not a preferred method.

Strong evaluations often combine quantitative and qualitative approaches to understand:

  • whether an intervention worked
  • how it was implemented
  • why it worked (or did not work)
  • whether it represented good value for money
  • whether it should be adopted more widely

Impact designs attempt to identify a counterfactual; process, realist and qualitative methods instead investigate implementation, experience and mechanisms. They answer complementary questions and should not all be presented as methods of estimating a counterfactual.

The credibility of a causal conclusion depends on whether the design and its assumptions fit the question and setting—not simply on the sophistication of the statistical model. No evaluation eliminates uncertainty completely.

Key Takeaways

  • improvement alone does not prove impact
  • fair comparison matters
  • small numbers can mislead
  • some improvement may happen naturally
  • trajectories matter
  • comparison groups strengthen confidence
  • the evaluation question should determine the evaluation method
  • ITS and DiD depend on explicit assumptions, not just visible improvements
  • retrospective causal evaluation may be possible even when no evaluation was planned
  • when no credible counterfactual exists, implementation and contribution evidence can still inform decisions
  • impact, mechanisms, equity and value are different but complementary questions
  • a pre-specified evaluation protocol clarifies the estimand, data, design and intended decision
  • confounding, time alignment, missing data and clustering can undermine apparently convincing results
  • robustness checks help reveal when conclusions depend on fragile assumptions

Good evaluation asks:

How confident are we that the intervention genuinely contributed to change?

Questions Decision-Makers Should Ask

  • Compared with what?
  • Did the trajectory change?
  • Could improvement have happened naturally?
  • Was the evaluation large enough to be informative?
  • How strong is our confidence in attribution?
  • Are the ITS or DiD assumptions plausible and tested?
  • What would change our conclusion in sensitivity analysis?
  • If causal impact cannot be identified, what other evidence would inform the decision?
  • Is the eligible population and denominator consistent over time?
  • Could selection bias, immortal time, missing records or repeated observations distort the result?
  • Was the evaluation question and primary analysis documented before reviewing results?

Choosing the Right Evaluation Method

Linking the Three Modules

  • Module 8 — Thinking About Evidence & Evaluation: why improvement is not automatically impact, and how to scrutinise claims.
  • Module 9 — Evaluation Methods in Practice: how to specify an estimand and protocol, choose a design, address observational bias and data limitations, test assumptions, and evaluate an existing scheme.
  • Module 17 — Understanding What Made the Difference: how to assess the combined effect of overlapping interventions, investigate mechanisms and equity, and inform portfolio commissioning decisions.

Further Reading

Readers who would like to explore evaluation methods in greater depth may find the following resources useful: