Topic 1.3 — Market Failure and Government Failure

Why this matters in public value

Every public and social-sector leader eventually faces a version of the same question: should this service be provided by the market, by government, by a nonprofit, or by some mixture of the three? The question arrives dressed as a budget line, a procurement decision, a privatisation proposal, or a campaign pledge to 'take back control' of a utility, but underneath it is always the same choice about who should provide what, and why.

Two rival diagnoses compete for the answer, and both are half right. The market-failure diagnosis says government should step in wherever markets under-produce something citizens value — clean air, vaccination, flood defence — because no private buyer can capture the full benefit. The government-failure diagnosis says collective provision is no automatic cure, because the officials and regulators asked to fix the market carry incentive problems just as real as the market's. A leader who reads only the first half becomes an uncritical expansionist; one who reads only the second becomes an uncritical privatiser, assuming every public failure proves markets superior even where the underlying good is one markets structurally cannot price.

This topic gives you both diagnoses side by side, because holding only one is a professional liability. It also sets up the rest of Part 1: the strategic triangle (see Topic 1.2 — The Strategic Triangle: Legitimacy, Value, and Capacity) is the tool for weighing a provision choice once you know which failures are in play, and the goods taxonomy (see Topic 1.4 — Public Goods, Merit Goods, and Value Pluralism) names exactly what kind of good you are dealing with before you choose who should supply it.

Core concepts

Market failure is the general term for unregulated private exchange producing an outcome that is not efficient by the market's own standards — value left uncreated, or resources misallocated relative to an idealised competitive market. It is not a claim that markets are bad in general, only that specific conditions break the mechanism that normally makes them work.

Public goods are non-excludable (you cannot stop a non-payer consuming them) and non-rivalrous (one person's consumption does not reduce what is left for anyone else) — street lighting and national defence are the textbook cases. Because no private firm can charge non-payers, private markets systematically under-supply these goods, the classic case for tax-funded provision.

Externalities are costs or benefits of an activity that fall on people not party to the transaction causing them: a factory that pollutes a river imposes a cost on downstream users who paid nothing, while a vaccinated individual confers a benefit on everyone whose infection risk falls. Left alone, markets over-produce activities with negative externalities and under-produce those with positive ones.

Information asymmetry occurs when one party to a transaction knows materially more than the other — a used-car seller knows more about the car's history than the buyer, a patient knows less about their diagnosis than their clinician. Left unaddressed, this can trigger adverse selection (only the worst risks stay in an insurance pool once premiums rise) or moral hazard (a party insulated from a risk takes on more of it), either of which can shrink or collapse a market rather than merely misprice it.

Natural monopoly exists where an industry's fixed costs are so large relative to the cost of serving one more customer that a single supplier can meet the whole market more cheaply than competitors could — water mains and electricity transmission are common examples, since duplicating the pipes or wires would waste resources. Market power is the general case of which natural monopoly is the extreme version: a firm's ability to raise price, or restrict output, above the competitive level without losing all its customers.

Government failure is the mirror-image concept: intervention meant to correct a market failure instead leaves resources worse allocated than the flawed market it replaced. It is studied under public choice theory, the branch of economics that applies the same self-interested, rational-actor assumption to voters, politicians, and bureaucrats that conventional economics applies to consumers and firms, abandoning the older idea of officials as disinterested guardians of the public interest.

Rent-seeking is the pursuit of income through manipulating rules or allocation rather than through producing anything of value — lobbying for a protective tariff, licence, or subsidy is rent-seeking where the effort captures existing value rather than creating new value. Regulatory capture is a specific, damaging form of this: a regulator created to police an industry instead comes to serve that industry's interests, typically through expertise-sharing, post-employment prospects, and asymmetric lobbying rather than outright corruption.

X-inefficiency, a term coined by the economist Harvey Leibenstein, names the internal slack — overstaffing, weak cost discipline, low effort — that creeps into any organisation insulated from competitive pressure, whether a monopoly firm or a monopoly public agency. The principal–agent problem describes the gap that opens whenever one party (the agent) acts for another (the principal) but has different interests and better information: citizens delegate to politicians, who delegate to bureaucrats, each link adding information the principal cannot fully monitor. William A. Niskanen modelled agency heads as budget-maximisers whose incentives push toward larger budgets than the public interest would justify, because legislators cannot easily observe the bureaucracy's true costs.

Best practices

Name the specific market failure before you name the remedy. Say whether you face a public good, an externality, an information asymmetry, a natural monopoly, or ordinary market power, because each implies a different fix — subsidy, tax, disclosure rule, price regulation, and competition policy are not interchangeable.
Treat government failure as a diagnosis, not an insult. Rent-seeking, capture, and X-inefficiency are specific, observable patterns with specific causes; find out which, if any, is actually present before assuming reform means simply shrinking the organisation.
Match the instrument to the failure, not to your organisation's default posture. An agency that always regulates, always contracts out, or always brings services in-house is optimising for habit rather than for the failure in front of it.
Where a natural monopoly cannot be avoided, regulate or own it, but keep the input markets around it competitive. You can rarely create competition for the water main itself, but you can for the firms that build, maintain, or bill against that main.
Separate the roles of funder, regulator, and provider wherever you can. Collapsing all three into one body removes the internal checks that would otherwise surface rent-seeking or capture before it becomes entrenched.
Give non-market outputs an explicit measure of value, and use comparators to substitute for missing competitive pressure. Absent a market price, an agency has no automatic discipline against X-inefficiency; a constructed unit cost or outcome measure, benchmarked against comparable providers, recreates some of that discipline.
Publish who a regulator meets, and when, as routinely as you publish its rulings. Capture typically develops gradually and legally, through information advantage and revolving-door staffing, so transparency about contact is cheaper than a corruption inquiry after the fact.
Build a sunset or review clause into any intervention justified by market failure. The condition that justified it may change; a power granted to fix one decade's problem should not silently become a permanent entitlement.
Route any rent that provision genuinely creates back to the public, rather than pretending it does not exist. A licence or franchise that creates scarcity value should have that value captured through auction or fee, not left as a windfall for whoever holds it.
Weigh operational capacity alongside legitimacy and value before deciding who provides, and revisit that boundary on a fixed cycle. A remedy that looks right on the market-failure/government-failure ledger but that your organisation cannot deliver is not a real option, and waiting for scandal to trigger a review guarantees it happens under the worst conditions (see Topic 1.2 — The Strategic Triangle: Legitimacy, Value, and Capacity).

Questions to discuss with your team

Which specific market failure, if any, justifies our involvement in this service? Walk through the taxonomy — public good, externality, information asymmetry, natural monopoly, market power — and be honest if none clearly applies. Vague appeals to 'market failure' in general are not a diagnosis; they are a way of avoiding one. If your organisation provides something a competitive market could supply perfectly well, the justification has to come from elsewhere, such as equity, and you should say so plainly rather than reaching for an argument that does not fit. Write the diagnosis down before the next budget round, so the team argues about the same claim rather than talking past each other.

Where in our own organisation would a budget-maximising official have the most room to operate? Niskanen's model assumes bureaucratic heads push for bigger budgets and headcounts because legislators and boards cannot easily verify the agency's true costs. Identify where oversight is thinnest in your own structure — a technical function your board cannot independently assess, or a programme with no external benchmark. Ask what evidence, if any, would actually surface slack if it had crept in there, rather than assuming good intentions are a sufficient safeguard. If nobody can answer with a specific mechanism, treat that silence itself as the finding.

Who regulates the regulator, and how would we know if capture were setting in? Capture rarely announces itself as corruption; it shows up as a slow drift in whose evidence gets weighted most heavily, whose meetings happen most often, and whose staff move most easily between regulator and regulated. None of those steps looks improper in isolation, which is exactly why the pattern is dangerous. Discuss what early-warning indicators your organisation tracks, if any, and who outside the relationship is positioned to notice a drift before it becomes a scandal. If the honest answer is 'no one', that gap is itself worth fixing.

If we handed this function to a private or nonprofit provider tomorrow, which failure mode would we be trading for which? Every provision choice swaps one risk profile for another rather than eliminating risk altogether. A contract-out might solve an X-inefficiency problem while introducing a new information asymmetry, or trade a capture risk for a natural-monopoly pricing risk if only one credible bidder exists. Use this question to force a like-for-like comparison, naming the specific failure you expect to gain and the one you expect to acquire, rather than letting an unexamined assumption that the alternative would be failure-free go unchallenged.

Are we measuring cost, or are we measuring value? In the absence of a market price, it is easy to default to tracking what is spent rather than what is achieved, precisely the condition under which X-inefficiency goes undetected for years. A stable, unremarkable spending pattern can look identical whether the underlying service is excellent or quietly deteriorating. Discuss whether your current metrics would actually reveal creeping slack, and if the answer is uncertain, treat building that capability as more urgent than any single budget line. Compare your answer against how a genuine market competitor would be judged, on price and quality together, not spending alone.

What would legitimacy and support not survive, even if the efficiency case for a different provider were airtight? Some remedies that look efficient on a market-failure/government-failure analysis would still be rejected by citizens or elected representatives regardless of the numbers — privatising a much-loved local service, for instance, or outsourcing a function citizens expect uniformed public servants to perform. Discuss where that line actually sits for your organisation, based on evidence of what citizens have accepted or rejected before, not leadership's own guess. Ask whether leadership is being honest about that constraint, or quietly dressing it up as a technical argument to avoid naming it as a political one (see Topic 1.2 — The Strategic Triangle: Legitimacy, Value, and Capacity).

In practice: a public value example

Northbridge is a fictional coastal city of around 240,000 people, used here purely to illustrate the topic's concepts. For decades its household waste and recycling collection was run by the city's own public works department, on a textbook market-failure argument: the collection routes amounted to a natural monopoly at neighbourhood scale, uncollected waste created a negative externality (vermin, disease risk, illegal dumping) no household could price correctly, and competing collectors on the same streets would have meant duplicated truck routes for no efficiency gain.

After 20 years, an incoming city council found a service that still worked but had grown quietly expensive. Benchmarking against comparable cities showed Northbridge's cost per tonne collected was around 40% above the median, its truck fleet older than average — the slow accumulation of X-inefficiency any monopoly provider risks once outside comparison stops happening. A finance review also found a sharper problem: the sole recycling-processing contractor, awarded the contract nine years earlier without re-tender, had steadily expanded the 'contamination surcharges' it billed the city, and contract managers drawn from the same small professional network as the contractor's staff had signed off each increase with little scrutiny — a mild but real case of capture involving no wrongdoing that still cost the public money.

Rather than treating this as proof that public provision itself was the mistake, the city separated the diagnosis into its two halves. The natural-monopoly and externality case for keeping collection publicly organised was still sound: nothing about the cost overruns changed the economics of duplicating truck routes, and an unregulated private market would likely recreate the same externality risk with fewer levers to fix it. What had failed was internal discipline, not the provision logic. The response combined instruments: collection stayed in-house, but mandatory five-yearly re-tendering was introduced for the genuinely contestable elements, an external cost-benchmarking panel was created reporting directly to elected councillors, and contractor-meeting logs were published after local journalists asked why nobody had reviewed the surcharges. None of this required abandoning public provision; each fix targeted the specific government-failure pattern that had emerged.

Four sector lenses

Local government

Local authorities are disproportionately the direct owners of natural-monopoly infrastructure — water, waste, local roads, local transit — because the economics of duplicating pipes or depots bite hardest at neighbourhood scale. This closeness to citizens means legitimacy failures surface fast and personally, but the local supplier market is often thin, making contestability harder than the textbook remedy assumes. Capture risk here more often runs through small, closely networked local supplier communities than large national lobbying operations.

National government

National governments set the framework — competition law, sector regulation, tax and subsidy design — meant to correct market failure across the whole economy, which is also why national agencies host the largest bureaucracies and the sharpest Niskanen-style budget dynamics. Sector regulators for energy, telecommunications, or finance are the classic setting for regulatory capture, because the expertise needed to regulate a complex industry is scarce and concentrated in the same people who move between regulator and regulated. National audit institutions exist largely to give the legislature, and behind it the citizen, an independent check on the agent it has delegated discretion to.

Nonprofits and non-governmental organisations frequently exist to fill goods gaps that neither market nor state will fund fully — a market-failure argument extended one step further, to goods valued collectively but that even electoral incentives do not reliably fund. But the sector has its own version of the principal–agent problem: a funder stands in an analogous position to a citizen-principal yet is often more distant from delivery than a voter is from government, and boards can face the same information asymmetry relative to frontline staff that legislators face relative to bureaucracies. Mission drift — an organisation gradually serving funders' priorities over beneficiaries' needs — is this sector's closest analogue to capture.

Multilateral and international

Cross-border externalities — climate change, pandemic disease, financial contagion — are market failures with no single sovereign able to internalise them alone, the standard justification for multilateral institutions. But the government-failure analogue scales up awkwardly: no single electorate acts as ultimate principal over a secretariat, member states can pursue their own rent-seeking within the institution's rules, and the same information asymmetry that lets a national bureaucracy outmanoeuvre its legislature can let an international agency outmanoeuvre a rotating board of representatives. Because exit and voice are weaker at this scale, transparency and independent evaluation carry even more of the accountability burden than they do nationally.

Common failure modes

Diagnosis by ideology, not by taxonomy. A leader who has decided in advance that markets always work, or government always works, reaches for whichever failure story fits the conclusion already held, rather than checking which failure is actually present. The cost is a remedy aimed at nothing real.

Capture dressed as expertise. Because regulating a complex industry requires people who understand it, and those people usually come from or go on to work in it, capture rarely looks like corruption; it looks like reasonable-sounding technical deference. The cost compounds slowly, as rules come to consistently favour incumbents over new entrants or consumers.

The permanent pilot. A programme launched to correct a market failure is never formally reviewed against the sunset condition that justified it, so it persists on inertia long after that condition has changed. The cost is resources locked into a remedy that has quietly stopped matching the problem.

Rent extraction disguised as service delivery. A contractor or protected supplier gradually expands the scope of what it bills for, using the switching costs and information advantage of a long-standing relationship rather than any single dramatic act. The cost falls on whoever ultimately pays, usually invisibly, through creeping fees rather than a headline price increase.

The unaccountable arm's-length body. An agency created to be 'independent of day-to-day politics' can drift into being independent of accountability altogether, with no principal positioned to observe whether its priorities still reflect the public interest it was created to serve. The cost is a principal–agent gap nobody is responsible for closing.

Maturity model

Sub-dimension	Initiate	Develop	Standardise	Manage	Orchestrate
Diagnosing market failure before intervening	No routine check; follows precedent or preference	Some teams cite public-good or externality logic informally	A framework requires a named failure before approval	Framework use is audited; weak diagnoses are challenged	Diagnosis is benchmarked with peers and revised as conditions change
Monitoring government and bureaucratic failure risk	Undiscussed; slack or capture surfaces only via scandal	Benchmarking happens occasionally, usually reactively	Regular cost and outcome benchmarking is standard practice	Results trigger corrective action, reported to oversight bodies	Benchmarking data is shared openly across peer organisations
Separation of funder, regulator, and provider roles	All three roles sit in one body	Separated on paper; same staff perform more than one role	Structurally separated with distinct reporting lines	Separation tested periodically; gaps are closed	Separation extends across organisational boundaries, with external review
Transparency over capture and rent-seeking risk	No routine disclosure of contact or contract variations	Disclosure only on request or after incidents	Contact logs and variation registers published routinely	Disclosed data reviewed by an independent function	Disclosure is benchmarked and coordinated with peer regulators
Review and sunset discipline on interventions	Interventions treated as permanent once introduced	Reviews happen occasionally, driven by budget pressure	A fixed review cycle is written into every authorisation	Reviews are resourced and reported to oversight bodies	Findings feed a shared, cross-organisational evidence base

Checklist

Name the specific market failure (public good, externality, information asymmetry, natural monopoly, market power) behind each service, or acknowledge none applies.
Identify where a Niskanen-style budget-maximising incentive could operate unchecked in your organisation.
Confirm funder, regulator, and provider roles are structurally separated wherever feasible.
Benchmark cost and outcomes for monopoly-provided services against comparable providers at least annually.
Publish regulator-industry contact logs and material contract variations.
Check every market-failure intervention for a sunset or review clause, and confirm the date has not lapsed.
Test whether current metrics would actually reveal creeping X-inefficiency, not just stable spending.
Trace any licence or scarcity value created by provision to whether its rent is captured for the public or left as a windfall.
Ask, for any proposed change of provider, which specific failure risk is being traded for which.
Confirm legitimacy and capacity have been weighed alongside the efficiency case, not overridden by it.

Key sources

The market-failure taxonomy draws on the neoclassical welfare-economics tradition, most directly Ronald Coase's work on externalities and transaction costs. The government-failure and public-choice tradition draws on James M. Buchanan and Gordon Tullock's work on constitutional and collective choice, and on William A. Niskanen's model of budget-maximising bureaucracy. The synthesis treating both failure modes as inputs to a strategic judgement, not a technical tie-breaker, follows Mark H. Moore's foundational public value work, supplemented by the OECD's and World Bank's comparative governance evidence.

References

Market failure — Wikipedia — https://en.wikipedia.org/wiki/Market_failure
Government failure — Wikipedia — https://en.wikipedia.org/wiki/Government_failure
Public good (economics) — Wikipedia — https://en.wikipedia.org/wiki/Public_good_(economics)
Externality — Wikipedia — https://en.wikipedia.org/wiki/Externality
Information asymmetry — Wikipedia — https://en.wikipedia.org/wiki/Information_asymmetry
Natural monopoly — Wikipedia — https://en.wikipedia.org/wiki/Natural_monopoly
Public choice theory — Wikipedia — https://en.wikipedia.org/wiki/Public_choice_theory
Regulatory capture — Wikipedia — https://en.wikipedia.org/wiki/Regulatory_capture
Rent-seeking — Wikipedia — https://en.wikipedia.org/wiki/Rent-seeking
X-inefficiency — Wikipedia — https://en.wikipedia.org/wiki/X-inefficiency
Principal–agent problem — Wikipedia — https://en.wikipedia.org/wiki/Principal%E2%80%93agent_problem
William A. Niskanen — Wikipedia — https://en.wikipedia.org/wiki/William_A._Niskanen
The Problem of Social Cost — R. H. Coase, Journal of Law and Economics, Vol. 3 (1960), pp. 1–44, published by the University of Chicago Press — https://www.jstor.org/stable/724810
The Calculus of Consent: Logical Foundations of Constitutional Democracy — James M. Buchanan and Gordon Tullock, Liberty Fund (Econlib edition) — https://www.econlib.org/library/Buchanan/buchCv3.html
Bureaucracy and Representative Government — William A. Niskanen, Aldine, Atherton (1971) — https://archive.org/details/bureaucracyrepre0000nisk
Creating Public Value: Strategic Management in Government — Mark H. Moore, Harvard University Press (1995) — https://www.hup.harvard.edu/catalog.php?isbn=9780674175587
Worldwide Governance Indicators — World Bank — https://www.worldbank.org/en/publication/worldwide-governance-indicators
Government at a Glance 2025 — OECD — https://www.oecd.org/en/publications/2025/06/government-at-a-glance-2025_70e14c6c.html