Public Sector Econometrics and Programme Evaluation

Why this matters in public value

Every year, public bodies renew, expand, or cancel programmes on the strength of an outcomes comparison: participants in the jobs programme found work faster than non-participants; pupils in the tutoring scheme scored higher than pupils who were not tutored. Each comparison is true as a description and dangerous as an argument, because none, on its own, tells you whether the programme caused the difference. People who enrol in a jobs programme are frequently more motivated than people who do not, and pupils selected for tutoring are often chosen because a teacher already expects them to respond well. The gap between 'the treated group did better' and 'the treatment made them do better' is exactly the gap that causal inference exists to close.

This is not an academic nicety. A finance committee asked to renew a programme's funding is implicitly being asked to authorise spending public money on a claim of causation, whether or not anyone in the room uses that word. Topic 2.1 gave you the outcomes framework that states what a programme is meant to change (see Topic 2.1 — The Public Value Scorecard and Outcomes Frameworks); this topic gives you the methods that test whether it actually changed it, and by how much. Cost-benefit and cost-effectiveness analysis (see Topic 2.2 — Social Cost-Benefit and Cost-Effectiveness Analysis) and social return on investment (see Topic 2.3 — Social Return on Investment and Impact Measurement) both need a defensible effect size to work with; this topic is where that effect size comes from, or where you learn that the number you already have cannot bear the weight being put on it.

The stakes are not only financial. A government that scales up an ineffective programme because a flawed evaluation made it look effective has wasted money and, often worse, displaced a genuinely effective alternative that never got the chance; a government that cancels an effective programme because a naive comparison made it look useless has destroyed value a better-designed evaluation would have protected. Getting causal identification right, and knowing when you cannot get it right, is itself a public value competency, not a specialist's side interest.

Core concepts

Causal inference is the branch of statistics concerned with establishing whether, and by how much, one variable's change produces a change in another, as distinct from the two variables merely moving together. It supplies the conceptual foundation — counterfactual reasoning, the language of treatment and control, the catalogue of specific threats to a causal claim — that every method in this topic draws on. Wikipedia: Causal inference.

The counterfactual is the outcome a person or unit would have experienced had they not received the programme, and it is the central, unobservable quantity every causal method tries to approximate: you can watch what happened to a participant, but never what would have happened to that same person unassisted. Every method below is a different strategy for constructing a credible stand-in for that missing observation. The Rubin causal model, developed principally by the statistician Donald Rubin, formalises this as the 'potential outcomes' framework: each unit has a potential outcome under treatment and one under control, and the causal effect is the difference between them, only one of which is ever observed. Wikipedia: Rubin causal model.

Confounding occurs when a third factor influences both whether a unit receives a programme and the outcome the programme is meant to change, making it look as if the programme caused an effect that the confounder actually produced. Motivation, prior ability, local economic conditions, and household wealth are recurring confounders in public programme evaluation, because they routinely affect both who takes up a service and how that person or place fares afterwards. Wikipedia: Confounding.

Selection bias is the specific case of confounding that arises from how participants come to be in the treatment group at all — self-selection, staff referral, or eligibility rules — such that treated and untreated groups differ systematically before the programme ever touches them. A jobs programme that only the most motivated jobseekers sign up for will look effective even if it does nothing, because the comparison group was never a fair stand-in for what participants would have done unassisted. Wikipedia: Selection bias.

Correlation does not imply causation is the general warning this topic operationalises: two variables moving together is consistent with one causing the other, with the reverse, with a shared common cause, or with coincidence, and the pattern alone cannot distinguish between these. Ice-cream sales and drowning deaths both rise every summer without either causing the other, because warm weather drives both; public programme evaluation is full of subtler versions of the same trap. Wikipedia: Correlation does not imply causation.

The randomised controlled trial (RCT), called a randomised evaluation in the policy-evaluation literature, assigns eligible units at random to a treatment group that receives the programme and a control group that does not, so that on average the two groups differ only in whether they were treated. Randomisation is powerful because it deals with confounding wholesale: it balances both the confounders you can measure and the ones you cannot, without needing to name them in advance. Wikipedia: Randomised controlled trial.

Difference-in-differences (DiD) compares the change in outcomes over time for a treated group against the change over the same period for a comparison group that did not receive the programme, assuming that, absent the programme, the two groups would have followed parallel trends. It is the workhorse method when a programme reached one jurisdiction before another and randomisation was never an option, because it can use routinely collected administrative data rather than a purpose-built trial. Wikipedia: Difference in differences.

Regression discontinuity design (RDD) exploits a programme rule that assigns treatment based on a measured score crossing a fixed cut-off — a benefit starting exactly at an income threshold, a remedial class assigned to pupils scoring just below a pass mark — and compares outcomes for units just above and just below, who are otherwise alike. Because which side of the threshold a unit falls on is effectively arbitrary in a narrow band around it, the comparison approximates a local randomised experiment without anyone having randomised anything. Wikipedia: Regression discontinuity design.

Instrumental variables (IV) estimation uses a third variable, an 'instrument', that affects take-up of a programme but has no direct effect on the outcome except through that take-up, to strip out the part of the treatment-outcome relationship that confounding would otherwise contaminate. A classic public-policy instrument is distance to a facility, or a lottery-allocated place, which shifts participation without independently causing the outcome; most of the method's difficulty lies in defending that the instrument affects the outcome only through the programme. Wikipedia: Instrumental variables estimation.

A natural experiment is a situation, arising from policy, geography, or circumstance rather than researcher design, in which some people are exposed to a programme and comparable others are not, as if an experiment had been run without anyone running it. A policy change taking effect at a birthdate cut-off, a boundary putting near-identical households in different jurisdictions, or a lottery rationing an oversubscribed place are natural experiments that RDD, IV, or DiD can then analyse. Wikipedia: Natural experiment.

The Hawthorne effect names the risk that people change their behaviour simply because they know they are being observed, independent of the programme's actual content, which can inflate an estimated effect if the control group is not observed with equal intensity — the act of measuring can itself become part of the treatment. Wikipedia: Hawthorne effect.

Statistical power is the probability that an evaluation will detect a true effect of a given size if one exists, depending chiefly on sample size, the effect size sought, and how much outcomes vary within groups. An underpowered evaluation that finds 'no significant effect' has often not shown the programme does nothing; it has shown the evaluation was too small to tell either way, a distinction leaders frequently miss under pressure to reach a verdict. Wikipedia: Power (statistics).

Best practices

1. Decide the evaluation design before the programme launches, not after. A randomised evaluation, a staggered roll-out suited to difference-in-differences, or an eligibility threshold suited to regression discontinuity design are all far easier to build in from the start than to retrofit once delivery has already begun on an ad hoc basis.
2. State the counterfactual explicitly in every evaluation brief. Write down, in one sentence, what you are comparing participants against — no programme at all, the previous programme, a different intervention — because an unstated counterfactual is where the sharpest evaluation disputes usually turn out to be hiding.
3. Treat a simple before-and-after comparison as a starting hypothesis, never as a finding. Outcomes drift for many reasons unrelated to your programme — the economic cycle, seasonal effects, other policy changes landing at the same time — and a before-and-after design cannot separate your programme's contribution from theirs.
4. Match the method to the rule that actually assigned the programme, not to the method you find most familiar. A programme rationed by lottery calls for a randomised evaluation's logic even if you never call it a trial; a programme assigned by a scored threshold calls for regression discontinuity design; a programme that rolled out to some areas before others calls for difference-in-differences.
5. Pre-register the outcome measures and the analysis plan before looking at the results. Deciding which outcome counts as 'success' after seeing which one moved is a well-documented route to false positives, and pre-registration is the standard defence against it.
6. Power the evaluation for the smallest effect size that would still justify the programme's cost, not for convenience. An evaluation too small to detect a policy-relevant effect will produce an uninformative 'no significant difference' result that gets misread as proof of failure.
7. Report the comparison group's characteristics alongside the treatment group's, not just the headline effect. A credible evaluation lets a sceptical reader check the two groups were alike before treatment; an evaluation that hides this makes the reader take balance on faith.
8. Commission the evaluation from a source independent of the delivery team, or firewall the analysis if you cannot. A team with a stake in a programme's success has a structural incentive, often unconscious, to interpret ambiguous evidence in the programme's favour.
9. Separate statistical significance from practical and fiscal significance in every report. A precisely estimated tiny effect and an imprecisely estimated large effect are different findings that demand different decisions, and collapsing both into 'significant' or 'not significant' erases the distinction a funding decision actually needs.
10. Budget for evaluation as a fixed share of programme cost at the design stage, not as whatever is left over. A programme large enough to matter for public value is usually large enough to fund a credible impact evaluation, and treating evaluation as discretionary is how weak evidence becomes routine.
11. Publish null and negative results with the same visibility as positive ones. A commissioning environment that only publicises successful evaluations creates a selection bias in the published evidence base itself, biasing everyone's later 'what works' judgement toward optimism.
12. Feed every completed evaluation into the wider evidence base your sector maintains, not only your own archive. A single evaluation, however well designed, is one data point; systematic synthesis across many is what turns a programme finding into transferable knowledge (see Topic 2.6 — Evidence Synthesis and What Works).

Questions to discuss with your team

What is the counterfactual we are actually comparing this programme against, and have we said so out loud? Every impact claim implies a comparison, but teams frequently leave it implicit, letting 'this worked' quietly slide between 'better than nothing', 'better than what we did before', and 'better than the best alternative we could have funded'. Ask your evaluation lead to state the counterfactual as a single sentence in plain language, then check whether the evidence you have actually supports a claim about that specific comparison. If the honest counterfactual is 'we don't know what would have happened otherwise', treat the current impact claim as a hypothesis, not a finding, and discuss what it would take to close that gap.

Could the people or places that took up this programme differ systematically from those that did not, in ways that would also affect the outcome? This is the selection-bias question, and it is worth asking even when — especially when — the answer seems obviously 'no'. Motivation, prior circumstances, local capacity, and staff referral patterns are common hidden sources of difference that produce a misleading impact estimate even with a large, careful-looking sample. If the team cannot describe how participants ended up in the programme, they cannot rule this out, and that gap should worry you more than a small sample size would.

Was our evaluation design built into the programme, or bolted on afterwards, and what did that cost us? A randomised evaluation, a staggered roll-out, or a scored eligibility threshold are all far cheaper and more credible when planned before launch than when reconstructed from whatever administrative data happened to survive. Ask what evaluation design options were considered at the programme's design stage, and if none were, treat that as a process gap to close before the next programme, not just a limitation of this one. Discuss what data would need to exist from day one for the next major programme to support a credible causal estimate.

Is our sample large enough to detect an effect worth knowing about, and would we recognise an underpowered 'no effect' finding if we saw one? An evaluation that reports no statistically significant effect is sometimes correctly telling you the programme does not work, and sometimes only telling you the evaluation was too small to see an effect that is genuinely there. Ask your analysts to state the minimum effect size the evaluation was powered to detect, and compare that to the effect size that would actually justify the programme's cost. If those two numbers are far apart, the 'no effect' finding may be uninformative rather than damning, and the team should say so plainly rather than letting silence be read as failure.

Who benefits, professionally or reputationally, if this evaluation comes back positive, and does that change how much independent scrutiny it needs? This is not an accusation of dishonesty; it is a recognition that people who designed or deliver a programme are structurally more likely to interpret ambiguous results charitably, often without noticing they are doing it. Discuss whether the evaluation was commissioned, analysed, or reviewed by anyone independent of the delivery team, and if not, what a proportionate firewall would look like for a programme of this size. The goal is a result your most sceptical funder would also trust, not merely one your own team is comfortable defending.

If this evaluation's finding is robust, what would we actually do differently, and are we prepared to do it? An evaluation commissioned but never intended to change a decision is a sunk cost dressed up as due diligence, and it is worth naming that risk before the results land rather than after. Discuss, in advance, what a strong positive finding would justify (scale-up, and by how much) and what a null or negative finding would trigger (redesign, or cancellation, and on what timeline). If nobody in the room can answer this before seeing the results, the evaluation may be serving a legitimacy function rather than a decision function, and that is worth being honest about (see Topic 2.1 — The Public Value Scorecard and Outcomes Frameworks).

In practice: a public value example

Fictional illustration. The Ministry of Employment Services in a mid-sized, unnamed country ran a one-year pilot of an intensive coaching service for long-term unemployed jobseekers, alongside the existing standard service. At the pilot's end, an internal report compared the group that received intensive coaching against everyone else who left unemployment during the same period: the coached group had a 38% job-placement rate within six months, against 24% for the rest of the caseload. Ministers were told the programme 'worked' and a business case went forward to roll it out nationally.

Before signing off the funding, the finance committee asked a straightforward question: how had jobseekers ended up in the coached group in the first place? The answer turned out to matter enormously. Caseworkers had referred jobseekers they judged 'most likely to benefit' into the coaching pilot, a well-intentioned triage decision that had, in effect, selected a more work-ready group into treatment from the start. The 38% versus 24% comparison was contaminated by exactly the selection bias this topic warns about: it was entirely possible the coached group would have out-performed the rest of the caseload even with no coaching at all.

Rather than approve or reject the roll-out on that comparison, the ministry's evaluation unit proposed a second phase using a lottery. Demand for the coaching service already outstripped the places available, so the unit randomly allocated places among equally eligible applicants and compared job-placement rates between the lottery winners and the lottery losers, an application of randomised-evaluation logic without needing to withhold the service from anyone who would not otherwise have received it. Twelve months later, the lottery comparison showed a smaller but real effect: a 9-percentage-point improvement in six-month placement, not the 14-point gap the original comparison had implied. The ministry used the lottery-based estimate, alongside its own cost data, to build a defensible cost-effectiveness case (see Topic 2.2 — Social Cost-Benefit and Cost-Effectiveness Analysis) for a scaled but resource-constrained roll-out, rather than the larger, undeliverable expansion the flawed initial comparison would have justified.

Four sector lenses

Local government

Local authorities typically run many small and medium-sized programmes rather than a few very large ones, which makes purpose-built randomised evaluations expensive relative to programme size and pushes local evaluators toward quasi-experimental methods that reuse administrative data already being collected — difference-in-differences across service areas that adopted a change at different times, or regression discontinuity design around a local eligibility threshold. Boundary and jurisdiction differences that would be a nuisance elsewhere are often a local evaluator's best natural experiment, since two neighbouring wards or districts can differ in policy while remaining alike in most other respects.

National government

National governments have the scale to fund purpose-built randomised evaluations and the administrative datasets, often linked across tax, benefits, health, and education records, to run credible quasi-experimental designs at low marginal cost. This scale is also where 'what works' evaluation task forces and centralised evaluation guidance, such as a treasury department's standard evaluation handbook, tend to sit, because only a national government can justify the fixed cost of maintaining standing evaluation infrastructure and a common methodological standard across many departments.

Social sector and nonprofit

Nonprofits and non-governmental organisations rarely have the sample sizes or budgets that a well-powered randomised evaluation needs, and their programmes often change adaptively in response to what frontline staff observe, which complicates any design that assumes a stable, well-defined treatment. Partnership with an academic research group or a specialist evaluation intermediary, of the kind that has grown around the randomised-evaluation movement in international development, is frequently the only realistic route to a rigorous causal estimate, and funders increasingly ask for exactly this kind of external partnership before committing multi-year grants.

Multilateral and international

Multilateral bodies fund programmes across many countries with differing data systems, administrative capacity, and ethical review norms, which makes standardised causal methods harder to apply uniformly but also creates natural cross-country variation that can itself become a natural experiment when a policy is adopted in some member states before others. Randomised evaluations in development economics have been particularly influential at this scale, producing evidence that shaped how international development funding is allocated, but they also carry a documented risk of external validity failure: a result that holds in the country and context studied does not automatically transfer to a different context, a caveat multilateral programme design too often glosses over.

Common failure modes

The naive before-and-after. A programme is judged solely by comparing outcomes before it started to outcomes after, with no comparison group at all, so any trend already under way — economic recovery, a demographic shift, a separate policy change — gets misattributed to the programme. The cost is a confident causal story built on a design that cannot actually distinguish the programme's effect from everything else that happened at the same time.

Selection bias mistaken for programme effect. Participants who opted in, or were referred in because they seemed likely to succeed, are compared against everyone else, producing an impact estimate that is really a measure of who the programme attracted rather than what it did to them. The cost is a scaled-up programme that looks less effective, sometimes much less, once it reaches a less self-selected population.

The underpowered evaluation reported as a null finding. A small pilot finds 'no statistically significant effect' and gets read as proof the programme does not work, when the sample was in fact too small to detect an effect of the size that would matter. The cost runs in both directions: a genuinely effective programme can be cancelled on the strength of evidence that was never capable of showing it worked, or resources move on to fund a larger trial that need not have been necessary if the original brief had been powered correctly.

Extrapolating a local estimate past its context. A regression discontinuity design correctly estimates the effect for units near the eligibility cut-off, and an instrumental-variable estimate correctly estimates the effect for units whose treatment status the instrument actually shifted, but both findings then get applied to the whole population as if the estimate held everywhere. The cost is a policy scaled on the strength of a number that was only ever locally true.

Evaluation as theatre. An evaluation is commissioned to create the appearance of evidence-based decision-making, but the decision has effectively already been made, and no plausible finding would have changed it. The cost is a wasted evaluation budget and, more corrosively, a slow erosion of trust in evaluation itself once staff notice that results never seem to change outcomes.

Maturity model

Checklist

• State the counterfactual for each major programme evaluation in one plain-language sentence.
• Confirm how participants came to be in the treatment group, and assess whether that process could itself explain the outcome difference.
• Choose the evaluation method (randomised evaluation, difference-in-differences, regression discontinuity design, instrumental variables, or natural experiment) to match the actual assignment rule, before delivery begins wherever possible.
• Calculate the statistical power needed to detect the smallest effect size that would justify the programme's cost.
• Pre-register outcome measures and the analysis plan before results are seen.
• Report the comparability of treatment and comparison groups alongside the headline effect.
• Commission or review the evaluation through a function independent of the delivery team.
• Separate statistical significance from fiscal and practical significance in every report to decision-makers.
• Budget for evaluation as a fixed share of programme cost at the design stage.
• Publish null and negative findings with the same visibility given to positive ones.
• Check whether a positive local estimate (from regression discontinuity design or instrumental variables) is being extrapolated beyond the population it actually covers.
• Route completed evaluations into the wider evidence base your sector maintains, not only your own archive.

Key sources

The methodological core of this topic draws on the applied-econometrics tradition associated with Joshua D. Angrist and Jörn-Steffen Pischke, and on the potential-outcomes framework formalised by Donald Rubin. The randomised-evaluation movement in public policy and development economics, associated with the Abdul Latif Jameel Poverty Action Lab (J-PAL) and its academic network, supplies much of the applied practice referenced in the sector lenses. Government evaluation standards draw on the UK Treasury's Magenta Book guidance, and evidence-synthesis practice draws on the Campbell Collaboration's standards for systematic review in social policy (see Topic 2.6 — Evidence Synthesis and What Works).

References

1. Causal inference — Wikipedia — https://en.wikipedia.org/wiki/Causal_inference
2. Randomised controlled trial — Wikipedia — https://en.wikipedia.org/wiki/Randomized_controlled_trial
3. Difference in differences — Wikipedia — https://en.wikipedia.org/wiki/Difference_in_differences
4. Regression discontinuity design — Wikipedia — https://en.wikipedia.org/wiki/Regression_discontinuity_design
5. Instrumental variables estimation — Wikipedia — https://en.wikipedia.org/wiki/Instrumental_variables_estimation
6. Natural experiment — Wikipedia — https://en.wikipedia.org/wiki/Natural_experiment
7. Correlation does not imply causation — Wikipedia — https://en.wikipedia.org/wiki/Correlation_does_not_imply_causation
8. Confounding — Wikipedia — https://en.wikipedia.org/wiki/Confounding
9. Selection bias — Wikipedia — https://en.wikipedia.org/wiki/Selection_bias
10. Rubin causal model — Wikipedia — https://en.wikipedia.org/wiki/Rubin_causal_model
11. Hawthorne effect — Wikipedia — https://en.wikipedia.org/wiki/Hawthorne_effect
12. Power (statistics) — Wikipedia — https://en.wikipedia.org/wiki/Power_(statistics)
13. J-PAL (Abdul Latif Jameel Poverty Action Lab) — Massachusetts Institute of Technology — https://www.povertyactionlab.org/
14. Impact Evaluation in Practice — Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch, World Bank — https://www.worldbank.org/en/programs/sief-trust-fund/publication/impact-evaluation-in-practice
15. Mostly Harmless Econometrics: An Empiricist's Companion — Joshua D. Angrist and Jörn-Steffen Pischke, Princeton University Press — https://press.princeton.edu/books/paperback/9780691120355/mostly-harmless-econometrics
16. The Magenta Book — HM Treasury — https://www.gov.uk/government/publications/the-magenta-book
17. Campbell Collaboration — Campbell Collaboration — https://www.campbellcollaboration.org/
18. What Works Network — UK Cabinet Office — https://www.gov.uk/guidance/what-works-network