Module 8 — Thinking About Evidence & Evaluation

Did it work — or would things have changed anyway?

Module Learning Objective

This module helps explain:

why evaluating healthcare interventions is often more difficult than it first appears — and how decision-makers can think more critically about whether changes in outcomes are genuinely attributable to a scheme or intervention.

By the end of this module, readers should feel more confident asking:

Did this intervention actually work — or would things have changed anyway?

The module focuses on practical healthcare examples, particularly interventions intended to reduce:

Emergency Department (ED) attendances
emergency admissions
avoidable hospital utilisation
delayed discharge
operational pressure

It introduces practical approaches to evaluation without assuming prior statistical knowledge. It also helps readers distinguish observed change, estimated causal impact, and plausible contribution. Detailed evaluation designs follow in Module 9, while Module 17 applies them to integrated care and overlapping interventions.

Why Evaluating Healthcare Interventions Matters

Healthcare systems frequently introduce schemes designed to improve outcomes or reduce demand.

Examples may include:

frailty pathways
virtual wards
Same Day Emergency Care (SDEC) services
care navigation models
community multidisciplinary teams (MDTs)
anticipatory care programmes
admission avoidance services
discharge coordination schemes

These interventions are often introduced with positive intentions:

reduce Emergency Department attendance

avoid unnecessary admissions

improve patient outcomes

reduce pressure on services

After implementation, a common question quickly emerges:

Did it work?

It sounds like a simple question.

Yet in healthcare, it is often one of the hardest questions to answer.

A Key Principle

Healthcare systems frequently struggle to distinguish between genuine intervention effects and changes that may have happened anyway.

Observed improvement does not automatically mean that an intervention caused the improvement.

This is because healthcare systems rarely remain static.

At the same time as an intervention is introduced:

winter pressures may occur
staffing may change
patient need may shift
flu or respiratory illness may rise or fall
other schemes may begin
pathways may change
coding practices may evolve

This means:

Improvement does not automatically imply impact — and observed change does not automatically mean the intervention caused it.

A reduction in Emergency Department attendance does not necessarily mean:

the intervention caused the reduction.

A Practical Healthcare Example

Imagine an Integrated Care Board introduces a new frailty service intended to reduce Emergency Department attendance among older adults.

The service provides:

proactive frailty assessment
community MDT support
medication review
rapid response care coordination

After six months (using comparable six-month periods and consistent population definitions):

Measure	Before Scheme	After Scheme
ED attendances	1,200	980
Emergency admissions	450	390

At first glance:

The scheme appears successful.

ED attendance has fallen.

Emergency admissions have also fallen.

But an important question remains:

Would this reduction have happened anyway?

This question sits at the heart of intervention evaluation.

The Biggest Evaluation Mistake: Before-and-After Comparisons

One of the most common mistakes in healthcare evaluation is relying solely on simple before-and-after comparisons.

For example:

Before intervention:
1,200 ED attendances

After intervention:
980 ED attendances

Conclusion:

“The scheme worked.”

Sometimes that conclusion may be correct.

But sometimes it may not.

Why?

Because healthcare systems are full of variation and competing influences.

For example:

winter demand may have been milder
influenza rates may have fallen
ambulance conveyance patterns may have changed
another community service may have launched
hospital admission thresholds may have changed
random fluctuation may have occurred

A simple before-and-after comparison cannot easily distinguish:

correlation

from

causation

This links directly back to Module 3:

Correlation vs Causation

A change occurring after an intervention does not automatically mean:

the intervention caused the change.

Why Before-and-After Comparisons Feel Convincing

Simple comparisons are attractive because they are:

easy to understand
quick to produce
intuitive to explain

For example:

Admissions reduced by 15%

sounds persuasive.

But healthcare systems are noisy and complex.

Without additional context, before-and-after comparisons can create:

false confidence
overclaiming of impact
poor investment decisions
misleading conclusions

The question should not simply be:

Did things improve?

Instead, decision-makers should ask:

How confident are we that the intervention caused the improvement?

Counterfactual Thinking: What Would Have Happened Anyway?

One of the most important ideas in evaluation is surprisingly simple.

To understand whether an intervention worked, we need to ask:

What would likely have happened if we had done nothing?

This idea is known as:

the counterfactual

In plain English:

What would probably have happened anyway?

This is often the missing question in healthcare evaluation.

A Simple Example

Imagine ED attendance falls following introduction of a community frailty service.

Question:

Did the intervention reduce attendance?

Maybe.

But another possibility exists:

ED attendance might have reduced anyway because:

winter demand was lower
viral illness reduced
population behaviour changed
ambulance conveyance thresholds shifted
other services improved

Without considering the counterfactual:

we risk mistaking coincidence for impact.

Counterfactual Thinking in Practice

A counterfactual is not simply a prediction of the future. It is an estimate of the outcome under a specified alternative, such as continuing usual care instead of introducing the new frailty service. We cannot observe both possibilities for the same people at the same time.

Worked example: observed improvement is not estimated impact

Suppose the frailty service recorded 980 ED attendances in the follow-up period, compared with 1,200 in the equivalent baseline period. That is an observed reduction of 220 attendances (18.3%).

Now imagine a credible evaluation estimated that, without the frailty service, the same eligible population would have experienced 1,080 attendances in the follow-up period.

Quantity	Illustrative ED attendances	Interpretation
Before implementation	1,200	Historical observation
After implementation	980	Observed outcome
Without implementation	1,080	Estimated, unobserved counterfactual
Observed before–after change	−220	Descriptive change; not the causal effect
Estimated intervention effect	−100	Difference between observed and counterfactual outcomes

In this example, 100 fewer attendances is the illustrative impact estimate, if the counterfactual is defensible. The figure of 1,080 must come from an appropriate evaluation design and evidence; it must not be selected merely to produce a favourable result.

The counterfactual estimate would normally carry uncertainty. Even if the observed result is exact, the intervention effect is not known with certainty.

Three Numbers, Three Different Meanings

1,200 → 980 describes what happened. 1,080 represents an estimated alternative outcome. 980 − 1,080 = −100 is a possible causal effect only if the evaluation design supports the comparison.

Evaluation becomes stronger when we compare reality with a plausible alternative.

In simple terms:

Observed outcome
        vs
What might reasonably have happened anyway

The closer we can get to a believable counterfactual:

the greater our confidence that the intervention genuinely contributed to the observed change.

Why Counterfactual Thinking Matters

The image below illustrates a common evaluation mistake:

assuming that improvement after an intervention automatically implies impact.

Good evaluation instead asks:

Compared with what?

The purpose of counterfactual thinking is not to dismiss improvement.

Instead, it encourages us to ask:

How confident are we that the intervention caused the change?

Why Fair Comparisons Are Difficult

A useful comparison should resemble the intervention population in the factors that influence outcomes before the intervention. This is harder than comparing two groups with similar average ages.

Selection bias and confounding

People referred to a frailty service may have higher clinical need, more recent emergency admissions, or better access to community services than those not referred. These characteristics can affect both who receives care and what happens next.

For example, patients receiving urgent community support might have more admissions than non-recipients because they were already more unwell. This does not demonstrate that the service caused admissions.

Confounding occurs when another factor helps explain an apparent relationship between intervention and outcome. Selection bias can arise when the people included in the analysis or intervention groups are systematically different in ways that distort the comparison.

Timing and regression to the mean

Patients may be selected after unusually high hospital use. Some subsequent reduction could occur naturally as an acute episode resolves or activity returns towards a more typical level (regression to the mean).

An intervention also cannot cause an event that occurred before it was delivered. Analysts need to distinguish eligibility, referral, treatment start and outcome dates, and avoid assigning patients to a treated group based on treatment received only later in follow-up.

Changing population need

If the eligible population becomes older or more clinically complex, raw activity may rise even when care improves. Conversely, falling admissions might reflect a changing denominator, admission thresholds or access constraints. Rates, consistent eligibility definitions and appropriate adjustment for baseline population differences can improve interpretation.

Adjustment should be guided by the causal question: variables changed by the intervention may lie on the causal pathway and should not automatically be controlled away.

Questions to Ask Before Comparing Groups

Were people eligible at the same point in time?
Were the groups comparable before implementation?
Could prior hospital use, frailty or deprivation influence both referral and outcomes?
Did the intervention start before the outcomes being attributed to it?
Did the population, recording rules or access to hospital care change?

Multiple Interventions Often Happen at the Same Time

One of the greatest challenges in evaluating healthcare improvement is that organisations rarely introduce a single intervention in isolation.

For example, an Integrated Care Board may simultaneously introduce:

Same Day Emergency Care (SDEC)
Virtual Wards
a frailty pathway
enhanced community services
discharge improvement programmes
Pharmacy First
care navigation

Suppose emergency admissions reduce by 8%.

An obvious question follows:

Which intervention made the biggest difference?

The honest answer is often:

We cannot know with certainty.

These interventions may:

work independently
reinforce one another
depend on one another to be effective
influence different patient groups
produce benefits at different times

Rather than asking:

Which intervention caused the improvement?

Decision-makers may instead need to ask:

How did each intervention contribute, and how might the interventions have reinforced one another?

This illustrates why evaluation often focuses on contribution rather than simple attribution, particularly in complex healthcare systems.

Looking Ahead: Modules 9 and 17

Module 9 introduces practical methods for constructing comparisons, including randomised designs, Interrupted Time Series (ITS) and Difference-in-Differences (DiD). Module 17 examines the harder problem of evaluating an integrated portfolio of care, where multiple interventions overlap and the evaluation may begin after implementation.

At portfolio level, the first causal question is whether the combined model improved outcomes for the eligible population compared with what would otherwise have happened. Explaining the contribution of individual services is a related but different task.

Levels of Evaluation Confidence

Not all evaluation approaches provide the same level of confidence.

Some approaches provide weak evidence.

Others provide stronger evidence.

A useful question for decision-makers is:

How confident should we be that the intervention caused the observed change?

The diagram below illustrates a simplified hierarchy of evaluation approaches and the confidence they provide that an intervention caused observed change.

This hierarchy is not about:

good versus bad methods

Instead, it helps us think about:

how confident we should be in causal conclusions.

Sometimes a simple before-and-after comparison may be all that is practical.

However:

decision-makers should recognise its limitations.

What Makes an Evaluation Claim Credible?

Different designs construct a comparison in different ways. None guarantees that the comparison is valid.

Approach	What makes it useful	Assumption or threat to examine
Before-and-after monitoring	Shows whether outcomes changed	Secular trends, seasonality and changing populations may explain the change
Interrupted Time Series (ITS)	Tests whether outcome level or trend changed around implementation	Other simultaneous changes, seasonal patterns, autocorrelation and the number of observations
Difference-in-Differences (DiD)	Compares changes between intervention and comparison populations	Whether outcomes would have followed parallel trends without the intervention, and whether spillovers or different concurrent changes undermine comparison
Randomised allocation	Helps make intervention and comparison groups comparable at baseline	Adherence, contamination, attrition, implementation and chance remain relevant

A sharp change after implementation is not proof of impact. Nor does choosing a neighbouring locality automatically make a comparison credible. Method choice, assumptions and data quality matter together. Module 9 develops these methods in more detail.

Test how sensitive the conclusion is

Return to the illustrative 980 ED attendances observed after implementation. Different plausible counterfactual estimates produce different conclusions:

Counterfactual assumption	Expected attendances without service	Estimated difference versus observed 980
Lower expected demand	950	30 more than expected
Central estimate	1,080	100 fewer than expected
Higher expected demand	1,150	170 fewer than expected

These are illustrative scenarios, not statistical confidence intervals. They show why analysts should test assumptions and report uncertainty rather than present one number as an unquestionable fact. If a commissioning decision changes under plausible alternative assumptions, that uncertainty is decision-relevant.

A Practical NHS Reality

Healthcare systems rarely operate under ideal evaluation conditions.

In practice:

interventions evolve over time
implementation varies between sites
operational pressures disrupt delivery
schemes overlap
data quality may vary
funding cycles are short

This is particularly important in healthcare, where decisions often need to be made despite incomplete evidence and competing operational pressures.

Healthcare evaluation therefore often involves balancing:

pragmatism

with

confidence in findings

The question is rarely:

Can we be completely certain?

Instead:

How confident should we be?

This means decision-makers often need to act before perfect evidence exists.

Evaluation therefore becomes less about certainty and more about:

making proportionate, evidence-informed judgements.

If the Service Has Already Started

Many NHS evaluations begin only after a scheme is operating. This does not automatically make causal evaluation impossible.

Historical data and a reliable start date exist: a retrospective time-series analysis may be feasible.
A credible comparison population exists: a controlled comparison may be possible, subject to its assumptions.
Neither is available: use descriptive monitoring, implementation and qualitative evidence to understand reach, delivery, experience and plausible contribution, without inventing admissions avoided.
An expansion or redesign is planned: agree eligibility, outcomes, baseline data and a feasible comparison before the next change.

Do Not Manufacture a Counterfactual

When a credible counterfactual cannot be constructed, evaluation still has value. It can explain what changed, for whom, how care was delivered and what may need improvement. But it should not claim a quantified causal impact that the evidence cannot identify.

Module 17 applies these choices to an established integrated frailty model and explores how to use the next commissioning decision to strengthen evaluation.

Match the Language to the Evidence

Claim	What it means
“ED attendances fell by 18.3% after the service began.”	A descriptive before–after finding
“Attendances were lower than projected from historical trends.”	A model-based comparison, dependent on assumptions
“The evaluation estimates 100 fewer attendances attributable to the service.”	A causal claim requiring a credible counterfactual and uncertainty assessment
“Patient and staff accounts suggest improved coordination.”	Evidence about experience or plausible mechanisms, not proof of an admissions effect

Strong evaluation is not always about making stronger claims. Sometimes it is about making more accurate, proportionate claims.

Reflection

Before moving on, ask:

If outcomes changed after an intervention, what evidence would genuinely convince us that the intervention caused the change?

This module has focused on:

thinking critically about evidence

because healthcare systems frequently observe change without always knowing:

whether the intervention caused it.

Good evaluation therefore moves beyond:

Did outcomes improve?

to ask:

How confident are we that the intervention contributed to improvement?

Key Takeaways

improvement alone does not prove intervention impact
healthcare systems change for many reasons at the same time
simple before-and-after comparisons can be misleading
stronger evaluation asks what might have happened anyway
evaluation is about improving confidence, not proving certainty
good decision-making balances pragmatism with evidence
selection, timing, population change and regression to the mean can distort comparisons
ITS and DiD depend on assumptions, not simply the presence of a statistical model
sensitivity analysis can reveal whether conclusions depend on uncertain assumptions
retrospective and theory-based evaluation remain useful when schemes have already started, but they support different claims

Good evaluation moves beyond:

Did outcomes improve?

to ask:

How confident are we that the intervention genuinely contributed to change?

Questions Decision-Makers Should Ask

Before acting on evaluation findings, ask:

What else changed at the same time?
Would improvement have happened anyway?
Are we confusing correlation with causation?
What evidence would increase our confidence?
How certain do we really need to be before acting?
What level of confidence is proportionate for this decision?
What assumptions make our counterfactual credible?
How sensitive is the conclusion to alternative plausible assumptions?
If the scheme is already running, can we reconstruct a comparison—or should we change the evaluation question?

In the next module, we explore practical approaches that healthcare systems use to improve confidence in evaluation findings — from pragmatic NHS evaluation methods through to Randomised Controlled Trials, Interrupted Time Series and Difference-in-Differences.