Public Sector Econometrics and Program Evaluation

Why this matters in public value

Every year, public bodies renew, expand, or cancel programs on the strength of an outcomes comparison: participants in the jobs program found work faster than non-participants; pupils in the tutoring scheme scored higher than pupils who were not tutored. Each comparison is true as a description and dangerous as an argument, because none, on its own, tells you whether the program caused the difference. People who enroll in a jobs program are frequently more motivated than people who do not, and pupils selected for tutoring are often chosen because a teacher already expects them to respond well. The gap between "the treated group did better" and "the treatment made them do better" is exactly the gap that causal inference exists to close.

This is not an academic nicety. A finance committee asked to renew a program's funding is implicitly being asked to authorize spending public money on a claim of causation, whether or not anyone in the room uses that word. Topic 2.1 gave you the outcomes framework that states what a program is meant to change (see Topic 2.1 — The Public Value Scorecard and Outcomes Frameworks); this topic gives you the methods that test whether it actually changed it, and by how much. Cost-benefit and cost-effectiveness analysis (see Topic 2.2 — Social Cost-Benefit and Cost-Effectiveness Analysis) and social return on investment (see Topic 2.3 — Social Return on Investment and Impact Measurement) both need a defensible effect size to work with; this topic is where that effect size comes from, or where you learn that the number you already have cannot bear the weight being put on it.

The stakes are not only financial. A government that scales up an ineffective program because a flawed evaluation made it look effective has wasted money and, often worse, displaced a genuinely effective alternative that never got the chance; a government that cancels an effective program because a naive comparison made it look useless has destroyed value a better-designed evaluation would have protected. Getting causal identification right, and knowing when you cannot get it right, is itself a public value competency, not a specialist's side interest.

Core concepts

Causal inference is the branch of statistics concerned with establishing whether, and by how much, one variable's change produces a change in another, as distinct from the two variables merely moving together. It supplies the conceptual foundation — counterfactual reasoning, the language of treatment and control, the catalog of specific threats to a causal claim — that every method in this topic draws on. Wikipedia: Causal inference.

The counterfactual is the outcome a person or unit would have experienced had they not received the program, and it is the central, unobservable quantity every causal method tries to approximate: you can watch what happened to a participant, but never what would have happened to that same person unassisted. Every method below is a different strategy for constructing a credible stand-in for that missing observation. The Rubin causal model, developed principally by the statistician Donald Rubin, formalizes this as the "potential outcomes" framework: each unit has a potential outcome under treatment and one under control, and the causal effect is the difference between them, only one of which is ever observed. Wikipedia: Rubin causal model.

Confounding occurs when a third factor influences both whether a unit receives a program and the outcome the program is meant to change, making it look as if the program caused an effect that the confounder actually produced. Motivation, prior ability, local economic conditions, and household wealth are recurring confounders in public program evaluation, because they routinely affect both who takes up a service and how that person or place fares afterwards. Wikipedia: Confounding.

Selection bias is the specific case of confounding that arises from how participants come to be in the treatment group at all — self-selection, staff referral, or eligibility rules — such that treated and untreated groups differ systematically before the program ever touches them. A jobs program that only the most motivated jobseekers sign up for will look effective even if it does nothing, because the comparison group was never a fair stand-in for what participants would have done unassisted. Wikipedia: Selection bias.

Correlation does not imply causation is the general warning this topic operationalizes: two variables moving together is consistent with one causing the other, with the reverse, with a shared common cause, or with coincidence, and the pattern alone cannot distinguish between these. Ice-cream sales and drowning deaths both rise every summer without either causing the other, because warm weather drives both; public program evaluation is full of subtler versions of the same trap. Wikipedia: Correlation does not imply causation.

The randomized controlled trial (RCT), called a randomized evaluation in the policy-evaluation literature, assigns eligible units at random to a treatment group that receives the program and a control group that does not, so that on average the two groups differ only in whether they were treated. Randomization is powerful because it deals with confounding wholesale: it balances both the confounders you can measure and the ones you cannot, without needing to name them in advance. Wikipedia: Randomized controlled trial.

Difference-in-differences (DiD) compares the change in outcomes over time for a treated group against the change over the same period for a comparison group that did not receive the program, assuming that, absent the program, the two groups would have followed parallel trends. It is the workhorse method when a program reached one jurisdiction before another and randomization was never an option, because it can use routinely collected administrative data rather than a purpose-built trial. Wikipedia: Difference in differences.

Regression discontinuity design (RDD) exploits a program rule that assigns treatment based on a measured score crossing a fixed cut-off — a benefit starting exactly at an income threshold, a remedial class assigned to pupils scoring just below a pass mark — and compares outcomes for units just above and just below, who are otherwise alike. Because which side of the threshold a unit falls on is effectively arbitrary in a narrow band around it, the comparison approximates a local randomized experiment without anyone having randomized anything. Wikipedia: Regression discontinuity design.

Instrumental variables (IV) estimation uses a third variable, an "instrument", that affects take-up of a program but has no direct effect on the outcome except through that take-up, to strip out the part of the treatment-outcome relationship that confounding would otherwise contaminate. A classic public-policy instrument is distance to a facility, or a lottery-allocated place, which shifts participation without independently causing the outcome; most of the method's difficulty lies in defending that the instrument affects the outcome only through the program. Wikipedia: Instrumental variables estimation.

A natural experiment is a situation, arising from policy, geography, or circumstance rather than researcher design, in which some people are exposed to a program and comparable others are not, as if an experiment had been run without anyone running it. A policy change taking effect at a birthdate cut-off, a boundary putting near-identical households in different jurisdictions, or a lottery rationing an oversubscribed place are natural experiments that RDD, IV, or DiD can then analyze. Wikipedia: Natural experiment.

The Hawthorne effect names the risk that people change their behavior simply because they know they are being observed, independent of the program's actual content, which can inflate an estimated effect if the control group is not observed with equal intensity — the act of measuring can itself become part of the treatment. Wikipedia: Hawthorne effect.

Statistical power is the probability that an evaluation will detect a true effect of a given size if one exists, depending chiefly on sample size, the effect size sought, and how much outcomes vary within groups. An underpowered evaluation that finds "no significant effect" has often not shown the program does nothing; it has shown the evaluation was too small to tell either way, a distinction leaders frequently miss under pressure to reach a verdict. Wikipedia: Power (statistics).

Best practices

1. Decide the evaluation design before the program launches, not after. A randomized evaluation, a staggered roll-out suited to difference-in-differences, or an eligibility threshold suited to regression discontinuity design are all far easier to build in from the start than to retrofit once delivery has already begun on an ad hoc basis.
2. State the counterfactual explicitly in every evaluation brief. Write down, in one sentence, what you are comparing participants against — no program at all, the previous program, a different intervention — because an unstated counterfactual is where the sharpest evaluation disputes usually turn out to be hiding.
3. Treat a simple before-and-after comparison as a starting hypothesis, never as a finding. Outcomes drift for many reasons unrelated to your program — the economic cycle, seasonal effects, other policy changes landing at the same time — and a before-and-after design cannot separate your program's contribution from theirs.
4. Match the method to the rule that actually assigned the program, not to the method you find most familiar. A program rationed by lottery calls for a randomized evaluation's logic even if you never call it a trial; a program assigned by a scored threshold calls for regression discontinuity design; a program that rolled out to some areas before others calls for difference-in-differences.
5. Pre-register the outcome measures and the analysis plan before looking at the results. Deciding which outcome counts as "success" after seeing which one moved is a well-documented route to false positives, and pre-registration is the standard defense against it.
6. Power the evaluation for the smallest effect size that would still justify the program's cost, not for convenience. An evaluation too small to detect a policy-relevant effect will produce an uninformative "no significant difference" result that gets misread as proof of failure.
7. Report the comparison group's characteristics alongside the treatment group's, not just the headline effect. A credible evaluation lets a skeptical reader check the two groups were alike before treatment; an evaluation that hides this makes the reader take balance on faith.
8. Commission the evaluation from a source independent of the delivery team, or firewall the analysis if you cannot. A team with a stake in a program's success has a structural incentive, often unconscious, to interpret ambiguous evidence in the program's favor.
9. Separate statistical significance from practical and fiscal significance in every report. A precisely estimated tiny effect and an imprecisely estimated large effect are different findings that demand different decisions, and collapsing both into "significant" or "not significant" erases the distinction a funding decision actually needs.
10. Budget for evaluation as a fixed share of program cost at the design stage, not as whatever is left over. A program large enough to matter for public value is usually large enough to fund a credible impact evaluation, and treating evaluation as discretionary is how weak evidence becomes routine.
11. Publish null and negative results with the same visibility as positive ones. A commissioning environment that only publicizes successful evaluations creates a selection bias in the published evidence base itself, biasing everyone's later "what works" judgment toward optimism.
12. Feed every completed evaluation into the wider evidence base your sector maintains, not only your own archive. A single evaluation, however well designed, is one data point; systematic synthesis across many is what turns a program finding into transferable knowledge (see Topic 2.6 — Evidence Synthesis and What Works).

Questions to discuss with your team

What is the counterfactual we are actually comparing this program against, and have we said so out loud? Every impact claim implies a comparison, but teams frequently leave it implicit, letting "this worked" quietly slide between "better than nothing", "better than what we did before", and "better than the best alternative we could have funded". Ask your evaluation lead to state the counterfactual as a single sentence in plain language, then check whether the evidence you have actually supports a claim about that specific comparison. If the honest counterfactual is "we don't know what would have happened otherwise", treat the current impact claim as a hypothesis, not a finding, and discuss what it would take to close that gap.

Could the people or places that took up this program differ systematically from those that did not, in ways that would also affect the outcome? This is the selection-bias question, and it is worth asking even when — especially when — the answer seems obviously "no". Motivation, prior circumstances, local capacity, and staff referral patterns are common hidden sources of difference that produce a misleading impact estimate even with a large, careful-looking sample. If the team cannot describe how participants ended up in the program, they cannot rule this out, and that gap should worry you more than a small sample size would.

Was our evaluation design built into the program, or bolted on afterwards, and what did that cost us? A randomized evaluation, a staggered roll-out, or a scored eligibility threshold are all far cheaper and more credible when planned before launch than when reconstructed from whatever administrative data happened to survive. Ask what evaluation design options were considered at the program's design stage, and if none were, treat that as a process gap to close before the next program, not just a limitation of this one. Discuss what data would need to exist from day one for the next major program to support a credible causal estimate.

Is our sample large enough to detect an effect worth knowing about, and would we recognize an underpowered "no effect" finding if we saw one? An evaluation that reports no statistically significant effect is sometimes correctly telling you the program does not work, and sometimes only telling you the evaluation was too small to see an effect that is genuinely there. Ask your analysts to state the minimum effect size the evaluation was powered to detect, and compare that to the effect size that would actually justify the program's cost. If those two numbers are far apart, the "no effect" finding may be uninformative rather than damning, and the team should say so plainly rather than letting silence be read as failure.

Who benefits, professionally or reputationally, if this evaluation comes back positive, and does that change how much independent scrutiny it needs? This is not an accusation of dishonesty; it is a recognition that people who designed or deliver a program are structurally more likely to interpret ambiguous results charitably, often without noticing they are doing it. Discuss whether the evaluation was commissioned, analyzed, or reviewed by anyone independent of the delivery team, and if not, what a proportionate firewall would look like for a program of this size. The goal is a result your most skeptical funder would also trust, not merely one your own team is comfortable defending.

If this evaluation's finding is robust, what would we actually do differently, and are we prepared to do it? An evaluation commissioned but never intended to change a decision is a sunk cost dressed up as due diligence, and it is worth naming that risk before the results land rather than after. Discuss, in advance, what a strong positive finding would justify (scale-up, and by how much) and what a null or negative finding would trigger (redesign, or cancellation, and on what timeline). If nobody in the room can answer this before seeing the results, the evaluation may be serving a legitimacy function rather than a decision function, and that is worth being honest about (see Topic 2.1 — The Public Value Scorecard and Outcomes Frameworks).

In practice: a public value example

Fictional illustration. The Ministry of Employment Services in a mid-sized, unnamed country ran a one-year pilot of an intensive coaching service for long-term unemployed jobseekers, alongside the existing standard service. At the pilot's end, an internal report compared the group that received intensive coaching against everyone else who left unemployment during the same period: the coached group had a 38% job-placement rate within six months, against 24% for the rest of the caseload. Ministers were told the program "worked" and a business case went forward to roll it out nationally.

Before signing off the funding, the finance committee asked a straightforward question: how had jobseekers ended up in the coached group in the first place? The answer turned out to matter enormously. Caseworkers had referred jobseekers they judged "most likely to benefit" into the coaching pilot, a well-intentioned triage decision that had, in effect, selected a more work-ready group into treatment from the start. The 38% versus 24% comparison was contaminated by exactly the selection bias this topic warns about: it was entirely possible the coached group would have out-performed the rest of the caseload even with no coaching at all.

Rather than approve or reject the roll-out on that comparison, the ministry's evaluation unit proposed a second phase using a lottery. Demand for the coaching service already outstripped the places available, so the unit randomly allocated places among equally eligible applicants and compared job-placement rates between the lottery winners and the lottery losers, an application of randomized-evaluation logic without needing to withhold the service from anyone who would not otherwise have received it. Twelve months later, the lottery comparison showed a smaller but real effect: a 9-percentage-point improvement in six-month placement, not the 14-point gap the original comparison had implied. The ministry used the lottery-based estimate, alongside its own cost data, to build a defensible cost-effectiveness case (see Topic 2.2 — Social Cost-Benefit and Cost-Effectiveness Analysis) for a scaled but resource-constrained roll-out, rather than the larger, undeliverable expansion the flawed initial comparison would have justified.

Four sector lenses

Local government

Local authorities typically run many small and medium-sized programs rather than a few very large ones, which makes purpose-built randomized evaluations expensive relative to program size and pushes local evaluators toward quasi-experimental methods that reuse administrative data already being collected — difference-in-differences across service areas that adopted a change at different times, or regression discontinuity design around a local eligibility threshold. Boundary and jurisdiction differences that would be a nuisance elsewhere are often a local evaluator's best natural experiment, since two neighbouring wards or districts can differ in policy while remaining alike in most other respects.

National government

National governments have the scale to fund purpose-built randomized evaluations and the administrative datasets, often linked across tax, benefits, health, and education records, to run credible quasi-experimental designs at low marginal cost. This scale is also where "what works" evaluation task forces and centralized evaluation guidance, such as a treasury department's standard evaluation handbook, tend to sit, because only a national government can justify the fixed cost of maintaining standing evaluation infrastructure and a common methodological standard across many departments.

Social sector and nonprofit

Nonprofits and non-governmental organizations rarely have the sample sizes or budgets that a well-powered randomized evaluation needs, and their programs often change adaptively in response to what frontline staff observe, which complicates any design that assumes a stable, well-defined treatment. Partnership with an academic research group or a specialist evaluation intermediary, of the kind that has grown around the randomized-evaluation movement in international development, is frequently the only realistic route to a rigorous causal estimate, and funders increasingly ask for exactly this kind of external partnership before committing multi-year grants.

Multilateral and international

Multilateral bodies fund programs across many countries with differing data systems, administrative capacity, and ethical review norms, which makes standardized causal methods harder to apply uniformly but also creates natural cross-country variation that can itself become a natural experiment when a policy is adopted in some member states before others. Randomized evaluations in development economics have been particularly influential at this scale, producing evidence that shaped how international development funding is allocated, but they also carry a documented risk of external validity failure: a result that holds in the country and context studied does not automatically transfer to a different context, a caveat multilateral program design too often glosses over.

Common failure modes

The naive before-and-after. A program is judged solely by comparing outcomes before it started to outcomes after, with no comparison group at all, so any trend already under way — economic recovery, a demographic shift, a separate policy change — gets misattributed to the program. The cost is a confident causal story built on a design that cannot actually distinguish the program's effect from everything else that happened at the same time.

Selection bias mistaken for program effect. Participants who opted in, or were referred in because they seemed likely to succeed, are compared against everyone else, producing an impact estimate that is really a measure of who the program attracted rather than what it did to them. The cost is a scaled-up program that looks less effective, sometimes much less, once it reaches a less self-selected population.

The underpowered evaluation reported as a null finding. A small pilot finds "no statistically significant effect" and gets read as proof the program does not work, when the sample was in fact too small to detect an effect of the size that would matter. The cost runs in both directions: a genuinely effective program can be canceled on the strength of evidence that was never capable of showing it worked, or resources move on to fund a larger trial that need not have been necessary if the original brief had been powered correctly.

Extrapolating a local estimate past its context. A regression discontinuity design correctly estimates the effect for units near the eligibility cut-off, and an instrumental-variable estimate correctly estimates the effect for units whose treatment status the instrument actually shifted, but both findings then get applied to the whole population as if the estimate held everywhere. The cost is a policy scaled on the strength of a number that was only ever locally true.

Evaluation as theater. An evaluation is commissioned to create the appearance of evidence-based decision-making, but the decision has effectively already been made, and no plausible finding would have changed it. The cost is a wasted evaluation budget and, more corrosively, a slow erosion of trust in evaluation itself once staff notice that results never seem to change outcomes.

Maturity model

Checklist

• State the counterfactual for each major program evaluation in one plain-language sentence.
• Confirm how participants came to be in the treatment group, and assess whether that process could itself explain the outcome difference.
• Choose the evaluation method (randomized evaluation, difference-in-differences, regression discontinuity design, instrumental variables, or natural experiment) to match the actual assignment rule, before delivery begins wherever possible.
• Calculate the statistical power needed to detect the smallest effect size that would justify the program's cost.
• Pre-register outcome measures and the analysis plan before results are seen.
• Report the comparability of treatment and comparison groups alongside the headline effect.
• Commission or review the evaluation through a function independent of the delivery team.
• Separate statistical significance from fiscal and practical significance in every report to decision-makers.
• Budget for evaluation as a fixed share of program cost at the design stage.
• Publish null and negative findings with the same visibility given to positive ones.
• Check whether a positive local estimate (from regression discontinuity design or instrumental variables) is being extrapolated beyond the population it actually covers.
• Route completed evaluations into the wider evidence base your sector maintains, not only your own archive.

Key sources

The methodological core of this topic draws on the applied-econometrics tradition associated with Joshua D. Angrist and Jörn-Steffen Pischke, and on the potential-outcomes framework formalized by Donald Rubin. The randomized-evaluation movement in public policy and development economics, associated with the Abdul Latif Jameel Poverty Action Lab (J-PAL) and its academic network, supplies much of the applied practice referenced in the sector lenses. Government evaluation standards draw on the UK Treasury's Magenta Book guidance, and evidence-synthesis practice draws on the Campbell Collaboration's standards for systematic review in social policy (see Topic 2.6 — Evidence Synthesis and What Works).

References

1. Causal inference — Wikipedia — https://en.wikipedia.org/wiki/Causal_inference
2. Randomized controlled trial — Wikipedia — https://en.wikipedia.org/wiki/Randomized_controlled_trial
3. Difference in differences — Wikipedia — https://en.wikipedia.org/wiki/Difference_in_differences
4. Regression discontinuity design — Wikipedia — https://en.wikipedia.org/wiki/Regression_discontinuity_design
5. Instrumental variables estimation — Wikipedia — https://en.wikipedia.org/wiki/Instrumental_variables_estimation
6. Natural experiment — Wikipedia — https://en.wikipedia.org/wiki/Natural_experiment
7. Correlation does not imply causation — Wikipedia — https://en.wikipedia.org/wiki/Correlation_does_not_imply_causation
8. Confounding — Wikipedia — https://en.wikipedia.org/wiki/Confounding
9. Selection bias — Wikipedia — https://en.wikipedia.org/wiki/Selection_bias
10. Rubin causal model — Wikipedia — https://en.wikipedia.org/wiki/Rubin_causal_model
11. Hawthorne effect — Wikipedia — https://en.wikipedia.org/wiki/Hawthorne_effect
12. Power (statistics) — Wikipedia — https://en.wikipedia.org/wiki/Power_(statistics)
13. J-PAL (Abdul Latif Jameel Poverty Action Lab) — Massachusetts Institute of Technology — https://www.povertyactionlab.org/
14. Impact Evaluation in Practice — Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch, World Bank — https://www.worldbank.org/en/programs/sief-trust-fund/publication/impact-evaluation-in-practice
15. Mostly Harmless Econometrics: An Empiricist's Companion — Joshua D. Angrist and Jörn-Steffen Pischke, Princeton University Press — https://press.princeton.edu/books/paperback/9780691120355/mostly-harmless-econometrics
16. The Magenta Book — HM Treasury — https://www.gov.uk/government/publications/the-magenta-book
17. Campbell Collaboration — Campbell Collaboration — https://www.campbellcollaboration.org/
18. What Works Network — UK Cabinet Office — https://www.gov.uk/guidance/what-works-network