Evidence Synthesis and What Works

Why this matters in public value

A minister, a mayor, or a charity's program director rarely has the luxury of a single, clean study to act on. More often the desk holds three evaluations that disagree, a literature review that cites them selectively, a pressure group waving the one trial that suits its case, and a deadline set by a budget cycle rather than the pace of research. Evidence synthesis exists because individual studies are noisy: sample sizes are small, contexts differ, and any one result could be a fluke, a local fit that will not travel, or a genuine effect. The question a leader needs answered is rarely "what did this one study find?" but "across everything credible that has been tried, what should I expect if I do this here, now?"

Public value theory makes this more than a technical nicety. A program funded with collective resources rests on a value proposition that citizens, in aggregate, would authorize (see Topic 2.1 — The Public Value Scorecard and Outcomes Frameworks for how that proposition is stated and tracked). A proposition built on a single cherry-picked study, or on no evidence review at all, is a weaker claim on legitimacy than one built on a careful synthesis — and a weaker basis for operational capacity, because staff and budgets get committed on the strength of it. The "what works" movement — the network of independent bodies that rate the quality and strength of policy evidence — grew directly out of this problem: too many public decisions were made on the loudest study rather than the best-supported one.

The harder problem this topic addresses is what a leader does when the synthesis itself is inconclusive, contested, or simply not available in time. Evidence hierarchies were built to bring order to a messy evidence base, but used carelessly, "the evidence isn't strong enough yet" becomes a permanent excuse to avoid a decision that reality will not actually wait for. A leader who understands how synthesis works, and where its limits are, can tell genuine uncertainty that warrants caution from a hierarchy being used as cover for inaction.

This topic is the natural sequel to the causal-inference methods a synthesis draws on (see Topic 2.4 — Public Sector Econometrics and Program Evaluation) and sits alongside, rather than replaces, the outcomes frameworks a synthesis ultimately serves (see Topic 2.1 — The Public Value Scorecard and Outcomes Frameworks). A leader who can read a systematic review, understand a what-works rating, and reason clearly about absent versus negative evidence makes better live decisions than one who can only demand "more evidence" or accept the first study handed to them.

Core concepts

A systematic review is a synthesis of the evidence on a defined question that uses an explicit, pre-specified, reproducible method to find, appraise, and combine studies, in contrast to a narrative literature review, which selects sources at the author's discretion (see Systematic review). Its defining features are a registered protocol written before the searching starts, a search strategy that includes unpublished and gray literature, explicit inclusion criteria, and a standard method for appraising each study's risk of bias. This is what separates a systematic review from a well-written essay with footnotes: another team following the same protocol would find broadly the same evidence and reach a broadly similar conclusion.

Meta-analysis is the statistical technique most often paired with a systematic review: it pools the quantitative effect sizes from multiple independent studies into a single, more precise estimate, usually weighting each study by its sample size and quality (see Meta-analysis). A meta-analysis is only as good as the studies feeding it; pooling five weak studies produces a more precise estimate of a possibly biased effect, not a trustworthy one. Analysts therefore report heterogeneity — how much study results vary beyond what chance would predict — because high heterogeneity signals the studies were not really testing the same thing, and a single pooled number may mislead.

Publication bias is the tendency for studies with positive, significant results to be published while null or negative results sit in a researcher's file drawer, making an intervention look more consistently effective in the literature than it really is (see Publication bias). A rigorous review corrects for this by searching trial registries and unpublished evaluations, not just journal articles. A leader reading a synthesis built only on published papers should ask what the reviewers did to find the studies that never made it into print.

The hierarchy of evidence ranks study designs by how well they control for confounding and bias, typically placing meta-analyses of randomized trials at the top, individual randomized controlled trials next, then quasi-experimental and observational designs, and expert opinion at the bottom (see Hierarchy of evidence). A randomized controlled trial allocates participants to an intervention or a comparison group by chance, the single most powerful tool for isolating a causal effect from everything else going on around it (see Randomized controlled trial; see Topic 2.4 — Public Sector Econometrics and Program Evaluation for how such designs are built and analyzed). The hierarchy is a genuinely useful heuristic, but a heuristic, not a law: a well-conducted observational study of a rare or slow-moving policy can outrank a poorly conducted, underpowered trial, and a leader who treats it as an absolute ranking will sometimes discard the more relevant evidence for the more prestigious design.

The GRADE approach — Grading of Recommendations, Assessment, Development and Evaluations — formalizes a second, separate judgment the hierarchy alone does not make: it rates both the certainty of the evidence and, independently, the strength of a recommendation to act on it, because strong evidence does not always justify a strong recommendation, and weak evidence does not always justify inaction against a severe, urgent harm (see GRADE approach). This separation is the single most useful idea in this topic for a time-pressed decision-maker: it lets a leader say, honestly, "the evidence here is moderate, and given the stakes and the cost of waiting, I recommend we proceed" — a defensible, documented judgment, rather than a false choice between waiting for certainty and ignoring evidence altogether.

A distinction that recurs constantly in live policy debate is between absence of evidence and evidence of absence: a lack of studies on an intervention's effect ("no evidence of effect") is not the same claim as a well-powered body of studies that positively found no effect ("evidence of no effect") (see Evidence of absence). A new local program with no evaluations yet simply has not been tested; a well-studied national program repeatedly found to move no outcome has been tested and shown not to work. Conflating the two is one of the most consequential errors a public leader can make: killing an untested but promising pilot as if it had failed, or continuing to fund a repeatedly disproven program as if nobody had looked closely enough yet.

Evidence-based policy is the broader movement and normative claim that public decisions should be grounded in rigorously established evidence rather than ideology, anecdote, or intuition alone (see Evidence-based policy). Its critics, and increasingly its own proponents, prefer "evidence-informed policy", which concedes that evidence is one input among several — values, mandate, cost, and the legitimacy questions this book treats as foundational — rather than a mechanical decision rule evidence alone can settle.

Out of this movement grew the what-works center or clearinghouse pattern: an independent body that systematically reviews the evidence for interventions in a policy domain and publishes accessible, comparable ratings, rather than leaving each local commissioner to run their own review. Cochrane pioneered the model in healthcare from 1993, maintaining a living handbook of systematic review methodology used well beyond medicine (see Cochrane (organisation)). The Campbell Collaboration extended the same methodology to social policy, criminal justice, education, and international development (see Campbell Collaboration). National clearinghouses followed: the United States' What Works Clearinghouse, run by its Institute of Education Sciences since 2002, reviews education research against explicit design standards (see What Works Clearinghouse); England's Education Endowment Foundation publishes a widely used Teaching and Learning Toolkit rating classroom interventions on cost, evidence strength, and typical impact in one comparable format (see Education Endowment Foundation). Equivalent centers exist for crime reduction, early intervention, homelessness, and ageing well — a small international family built on the same method: systematic review, transparent rating, and a practitioner-facing summary that needs no methodologist to read it.

Best practices

1. Ask for the synthesis before the single study. When a striking result lands on your desk, ask first whether a systematic review or a what-works rating already exists for that intervention, not whether the study looks convincing in isolation.
2. Read the underlying review alongside any toolkit rating, not instead of it. A one-line rating or a padlock icon summarizes a judgment; it is not a substitute for understanding what evidence it rests on and how directly it applies to your context.
3. Separate the quality of the evidence from the strength of your recommendation, explicitly. Borrow the GRADE logic even informally: state how certain the evidence is, then state separately how strongly that certainty, combined with stakes and cost, justifies acting now.
4. Name whether you face "no evidence yet" or "evidence of no effect" before you decide. Put this distinction in writing in every evidence brief, because the two situations warrant almost opposite responses — cautious piloting versus stopping or redesigning.
5. Check the match between the review's context and yours before assuming the effect will transfer. A pooled effect size from studies run elsewhere is an average, not a promise; ask what moderated the effect originally and whether those moderators are present here.
6. Commission a rapid evidence assessment when a full systematic review will not fit your timeline. A structured, time-boxed review of the existing literature — days or weeks, not months — is a legitimate middle path between new primary research and no review at all.
7. Publish your own evaluations regardless of the result. Null and negative findings are as valuable to the next synthesis as positive ones; withholding them worsens the publication bias that undermines every future review in your field.
8. Build a standing relationship with a what-works center or clearinghouse in your domain before you need one. Cold-searching a synthesis body's toolkit for the first time during a live crisis is far slower than already knowing what it covers and who to ask about a gap.
9. Set your evidence threshold before the results arrive, not after. Agree in advance what level of evidence would justify scaling, piloting further, or stopping, to reduce the temptation to reinterpret ambiguous findings to fit a decision already made.
10. Timebox the search for a study that will remove all doubt. Waiting for a perfect, unambiguous evidence base is itself a decision, usually to keep doing whatever you already do; set a deadline that respects the cost of delay as well as the cost of acting on weak evidence.
11. Treat synthesis as an input to the value proposition, not a replacement for it. The best-supported intervention is not automatically the right choice once cost, distributional effect, and legitimacy are weighed alongside it (see Topic 2.1 — The Public Value Scorecard and Outcomes Frameworks).
12. Document which evidence you considered, and why you weighted it as you did. A written record of the studies reviewed and the reasoning behind the final judgment protects both the decision's legitimacy and the next reviewer's ability to learn from it.

Questions to discuss with your team

When did we last change a live decision because a systematic review or a what-works rating contradicted what we had already planned to do? If the honest answer is "never", that is worth sitting with, because it suggests evidence review may function as a confirmation exercise rather than a genuine check. Discuss a recent decision where the evidence base was mixed and walk through whether it actually shaped the choice or was gathered mainly to support a conclusion reached beforehand. Consider what would need to change in how evidence reaches decision-makers, and when, for it to have real influence rather than symbolic weight.

Are we anywhere treating "no evidence of effect" as if it were "evidence of no effect", or the reverse? Walk through your current program portfolio and identify any program you have quietly deprioritized because "the evidence isn't there", and check whether that means genuinely untested or genuinely disproven. Do the same in the other direction: any program you keep funding because "nobody's shown it doesn't work", when in fact nobody has looked. Discuss what it would take to correct the two or three most consequential examples, and who should own that correction.

If a decision needed evidence by Friday, do we actually have a rapid evidence assessment process, or would we improvise one under pressure? Time pressure is not hypothetical; it is the normal condition under which most real evidence-based decisions get made. Discuss whether your organization has a standing method — a named team, a template, a short list of what-works centers to check first — or whether every urgent request starts from a blank page. If the latter, discuss what a minimum viable rapid-assessment process would look like and who would own it.

Whose evidence hierarchy are we actually using, and does it fit the stakes and reversibility of this decision? Different domains apply their evidence standards differently, and a hierarchy calibrated for irreversible, high-stakes decisions is not always the right bar for a cheap, reversible, easily monitored pilot. Discuss whether your organization applies one evidence standard uniformly regardless of stakes, and whether that protects you from bad decisions or simply slows down good ones. Agree, as a team, what standard should be required for the next three decisions on your agenda, before you know what the evidence says.

What have we done, concretely, to stop our own evaluations disappearing into a file drawer? Every unpublished null result your organization sits on is a gap some other organization will have to rediscover the hard way. Discuss your track record honestly: how many of your last 10 evaluations were shared externally, and what happened to the ones that were not. Agree a concrete, resourced commitment — a publication policy, a shared registry — rather than a general aspiration to be more transparent.

Are we using the phrase "we need more evidence" honestly, or as a way of avoiding a decision we do not want to make? This is the most uncomfortable question in this topic, and it deserves to be asked directly. Identify a specific decision currently stalled on evidence grounds and discuss, candidly, what would actually change your mind, and whether the evidence you are waiting for is realistically going to arrive in time. If it is not, discuss what an honest decision under genuine uncertainty would look like instead of continued waiting, and who has the authority to make it.

In practice: a public value example

A regional employment agency, part of a national government department responsible for labor-market programs, was under pressure to decide within one budget cycle whether to scale a job-training pilot from three districts to the whole region. The pilot combined subsidized work placements with structured mentoring, and the agency's own randomized trial across the three pilot districts found a modest but statistically significant increase in sustained employment at 12 months. A rival evidence source complicated the picture: a systematic review published the previous year, pooling nine similar programs across several countries, found a near-zero average effect, with wide variation between studies and no clear pattern explaining which programs worked.

The agency's evidence unit had six weeks before the budget submission deadline to reconcile the two findings. A junior analyst's first instinct was to treat the pooled review as more authoritative simply because it combined more studies, and to recommend scrapping the pilot. The unit's lead evidence adviser pushed back: a near-zero pooled average across genuinely different programs was not strong evidence that this specific, well-implemented local program would fail — it was evidence that job-training programs in general are heterogeneous in quality and fit, which was already known. The high heterogeneity reported in the review was itself the more important finding than the pooled average.

The adviser proposed a middle path rather than a binary choice. She recommended a staged scale-up to half the region, with expansion districts randomly selected so the rollout itself generated further evidence rather than merely spending against an assumption (see Topic 2.4 — Public Sector Econometrics and Program Evaluation for how a phased rollout can be designed as a further evaluation). She also recommended publishing the original trial in full, methods and raw data included, so it could be picked up by the next systematic review rather than sitting in an internal report nobody outside the agency would ever see.

The department's leadership approved the staged approach, framed explicitly in the business case (see Topic 2.5 — Business Cases and Value for Money) as a decision made under moderate-certainty evidence with a built-in mechanism to resolve the uncertainty within 18 months, not as a decision made because the evidence was conclusive. After 18 months, the expansion districts showed an effect close to the original pilot's, and the department scaled further with a stronger evidence base than either source had offered alone — and a clear paper trail showing why it had chosen to act rather than wait.

This scenario is illustrative and fictional; it does not describe any real department's program or evaluation.

Four sector lenses

Local government

A local authority rarely has the budget or caseload to commission its own systematic review, so its practical craft is translating national and international synthesis evidence into a specific local context under real time pressure — checking whether a toolkit-rated intervention's evidence base resembles its own population, service system, and funding level before assuming the rated effect will transfer. Local leaders are also closest to the cost of both errors this topic warns about: acting on an intervention a synthesis has already shown not to work, and refusing a genuinely promising but under-evaluated local innovation because no what-works center has rated it yet.

National government

National government is where most what-works centers and formal evidence-hierarchy guidance sit, giving national departments proximity to the synthesis infrastructure but also the greatest exposure to treating an evidence hierarchy as a bureaucratic gate rather than a decision aid. The characteristic national task is commissioning and maintaining that infrastructure — funding a clearinghouse, setting its evidence standards, and ensuring its ratings actually reach officials writing business cases (see Topic 2.5 — Business Cases and Value for Money), rather than existing as a well-regarded website policy teams rarely consult under deadline pressure.

Social sector and nonprofit

Nonprofits and charities typically lack the resources to run rigorous evaluations at scale, making them heavily dependent on external synthesis bodies and increasingly on funders who require alignment with a rated evidence tier as a grant condition. This creates a distinctive risk: organizations under financial pressure can be tempted to overstate how well their program maps onto a favorable rating — a form of evidence-washing — or, conversely, small but genuinely effective local charities can be squeezed out of funding because they have never been large enough to generate the evidence a national clearinghouse rates highly (see Topic 3.8 — Social Sector and Nonprofit Management for the funding structures behind this pressure).

Multilateral and international

Multilateral and international bodies face the generalizability problem in its most acute form: an intervention proven in one country's health system, agricultural sector, or labor market is routinely proposed for a very different economic and institutional context, and the synthesis literature is better at flagging this unsupported transfer after the fact than before it. Research networks built to close this gap, such as the Abdul Latif Jameel Poverty Action Lab, have pushed development policy toward context-specific replication rather than single-country proof of concept, but the tension between globally comparable evidence and context-dependent effects remains a defining feature of synthesis at this scale.

Common failure modes

Evidence hierarchy as an excuse for inaction. A team facing a difficult, contested decision cites "insufficient evidence" or "no randomized trial yet" as a reason to defer indefinitely, when the honest driver of the delay is political discomfort or a wish to avoid accountability. This costs the organization the opportunity itself, and it costs the evidence base a real-world test that could have resolved the uncertainty, leaving the next decision-maker facing exactly the same gap.

Conflating absence of evidence with evidence of absence. A promising but untested local innovation is killed as though it had been tried and failed, while a repeatedly disproven national program keeps being funded because nobody has framed the evidence as a negative finding rather than a gap. Both errors are expensive: one forecloses promising options, the other protects programs a clear-eyed synthesis would have retired years earlier.

Treating a toolkit rating as the whole analysis. A decision-maker reads a single padlock icon or star rating and stops there, without checking whether the underlying studies match their context, population, or scale. Ratings are deliberately compressed for accessibility; used without the underlying review, they can produce confident decisions built on a thinner evidence base than the rating's simplicity suggests.

Ignoring heterogeneity in favor of the headline pooled result. A meta-analysis's single average effect size gets quoted in a briefing while the wide variation between the studies that produced it — often the more informative finding — is dropped. This produces false precision: a decision-maker acts as if every implementation behaves the same way, when implementation quality and context are doing as much work as the intervention itself.

Letting publication bias silently shape the evidence base. An organization reads only published academic literature when assessing an intervention, missing the unpublished negative trials and internal evaluations that never made it into a journal. The resulting picture of "what works" skews systematically positive, and decisions built on it inherit that skew without anyone choosing it deliberately.

Maturity model

Checklist

• Systematic review or what-works rating checked before relying on a single study.
• Underlying evidence review read, not just the summary rating or icon.
• Quality of evidence and strength of recommendation stated as two separate judgments.
• "No evidence of effect" versus "evidence of no effect" explicitly distinguished in the written brief.
• Context match between the reviewed studies and the decision at hand checked (population, scale, service system).
• Rapid evidence assessment process available and used when a full review will not fit the timeline.
• Evidence threshold for the decision agreed before results were reviewed.
• Plan in place to publish this organization's own evaluation, whatever the result.
• Standing relationship with a relevant what-works center or clearinghouse established, not built from scratch under pressure.
• Reasoning behind the final judgment documented, including evidence considered and rejected.
• Decision reviewed against the value proposition and legitimacy considerations, not evidence strength alone.

Key sources

• The Cochrane Handbook for Systematic Reviews of Interventions, maintained by Cochrane, for the foundational methodology that systematic review practice across sectors is built on.
• The Campbell Collaboration's guidance on systematic reviews and evidence synthesis for social, behavioral, and criminological policy, for the direct extension of that methodology into public policy domains beyond health.
• The What Works Clearinghouse's Standards Handbook (Institute of Education Sciences, United States), for a worked example of how a national clearinghouse sets explicit study-design standards and rates evidence against them.
• The Education Endowment Foundation's Teaching and Learning Toolkit, for a practical, practitioner-facing example of translating pooled evidence into comparable cost, impact, and evidence-strength ratings.
• The Organisation for Economic Co-operation and Development (OECD)'s work on building capacity for evidence-informed policymaking, for a cross-country account of how governments organize the demand and supply sides of evidence use.
• The College of Policing's What Works Center for Crime Reduction and its crime reduction toolkit, for a further worked example of the clearinghouse pattern applied outside health and education.

References

1. Systematic review — Wikipedia — https://en.wikipedia.org/wiki/Systematic_review
2. Meta-analysis — Wikipedia — https://en.wikipedia.org/wiki/Meta-analysis
3. Evidence-based policy — Wikipedia — https://en.wikipedia.org/wiki/Evidence-based_policy
4. Cochrane (organisation) — Wikipedia — https://en.wikipedia.org/wiki/Cochrane_(organisation)
5. Campbell Collaboration — Wikipedia — https://en.wikipedia.org/wiki/Campbell_Collaboration
6. What Works Clearinghouse — Wikipedia — https://en.wikipedia.org/wiki/What_Works_Clearinghouse
7. Education Endowment Foundation — Wikipedia — https://en.wikipedia.org/wiki/Education_Endowment_Foundation
8. Publication bias — Wikipedia — https://en.wikipedia.org/wiki/Publication_bias
9. Randomized controlled trial — Wikipedia — https://en.wikipedia.org/wiki/Randomized_controlled_trial
10. Evidence of absence — Wikipedia — https://en.wikipedia.org/wiki/Evidence_of_absence
11. Hierarchy of evidence — Wikipedia — https://en.wikipedia.org/wiki/Hierarchy_of_evidence
12. GRADE approach — Wikipedia — https://en.wikipedia.org/wiki/GRADE_approach
13. Cochrane Handbook for Systematic Reviews of Interventions — Cochrane — https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current
14. Campbell Collaboration — Campbell Collaboration — https://www.campbellcollaboration.org/
15. What Works Clearinghouse Standards Handbook — Institute of Education Sciences, U.S. Department of Education — https://ies.ed.gov/ncee/wwc/Handbooks
16. Teaching and Learning Toolkit — Education Endowment Foundation — https://educationendowmentfoundation.org.uk/education-evidence/teaching-learning-toolkit
17. Building Capacity for Evidence-Informed Policy-Making — OECD — https://www.oecd.org/en/publications/building-capacity-for-evidence-informed-policy-making_86331250-en.html
18. What Works Center for Crime Reduction — College of Policing — https://www.college.police.uk/research/what-works-center-crime-reduction