AI and Algorithmic Decision-Making in Government

Why this matters in public value

Public bodies have always rationed attention: a caseworker cannot personally interview every applicant, a customs officer cannot inspect every container, a fraud team cannot review every claim. Artificial intelligence (AI) — the broad field of building systems that perform tasks normally requiring human judgment — and the narrower practice of algorithmic and automated decision-making (ADM) promise to ration that attention more consistently and at far greater scale, scoring, flagging, or directly deciding cases that a human team could never fully review. Topic 5.1 covers the digital service-delivery patterns this capability sits inside (see Topic 5.1 — Digital Government and Digital Transformation); this topic is about what happens once a model, rather than a named official, is doing some or all of the deciding.

The promise is real: a well-built model can catch fraud patterns invisible to a caseworker, apply eligibility rules with perfect consistency across a million claims, and free scarce expert time for the cases that genuinely need judgment. The risk is equally real and harder to see from inside the organization that built the system. A model trained on historical data can reproduce and amplify the very discrimination a public body exists to correct; a complex model can be so opaque that neither the citizen nor the official relying on it can say why it decided what it decided; and a decision an algorithm heavily informs can still, procedurally, leave nobody able to give the citizen a reason or hear their case. None of this is hypothetical: regulators, courts, and ombudsman-equivalent bodies across multiple jurisdictions have already found public-sector risk-scoring and fraud-detection systems that disproportionately flagged people by neighborhood, immigration history, or family structure, in each case because the pattern that made the model accurate also happened to track a protected or sensitive characteristic.

The public value stake is therefore not efficiency versus fairness as a trade-off to be balanced once, but a design discipline to be sustained continuously. A citizen denied a benefit by a human official can ask that official why, and can appeal to another human with the standing and power to change the answer — the accountability machinery Topic 3.4 describes (see Topic 3.4 — Accountability, Transparency, and Legitimacy). A citizen denied a benefit by a model needs that same right to an explanation and that same route to a human who can overturn it, or the public body has quietly exported a due-process guarantee it never formally withdrew. Getting this right is what lets a public body capture algorithmic decision-making's genuine gains in consistency and scale without converting "the algorithm decided" into a phrase that ends accountability rather than starting it.

Core concepts

Automated decision-making (ADM) is the use of data, software, and algorithms to make or materially inform decisions with varying degrees of human oversight, spanning a spectrum from a system that merely surfaces information for a human to one that decides and acts with no human involved at all (see automated decision-making). Where a system sits on that spectrum — decision support, decision augmentation, or full automation — determines which accountability and due-process obligations apply, which is why regulators increasingly classify systems by position on it rather than by the underlying technology.

Human-in-the-loop describes an arrangement in which a person reviews or approves an algorithm's output before it takes effect, as distinct from human-on-the-loop (a person monitors and can intervene) or fully automated processing with no review at all (see human-in-the-loop). A genuine human-in-the-loop control requires that the reviewer has the time, information, and standing to actually disagree with the system; a reviewer who approves nearly every algorithmic recommendation because disagreeing takes longer or carries more personal risk than agreeing is not exercising oversight, even though the process chart shows a human step. This pattern, often called automation bias, is one of the most common and least visible failures in this topic's subject.

Algorithmic bias is the systematic, repeatable tendency of a computerized system to produce unfair outcomes — for example, privileging one group over another in ways the system's designers neither intended nor, often, noticed (see algorithmic bias). In public administration the dominant source is training-data bias: a model trained on historical enforcement, lending, or casework records learns the patterns in those records, including any historical over-policing, under-service, or discriminatory practice baked into who was investigated, approved, or flagged in the past. A model can reproduce this outcome even when a sensitive characteristic — race, disability, migration status — is explicitly excluded from the data, because a correlated proxy (postcode, surname, shift pattern, benefit history) carries the same signal by another name; removing the named variable without testing for its proxies is a common way a public body believes it has fixed a bias problem it has not fixed at all.

Fairness in machine learning refers to the family of technical and procedural approaches for measuring and correcting algorithmic bias in automated decisions (see fairness (machine learning)). A genuinely difficult feature of this field, not a solvable technical footnote, is that several intuitively reasonable definitions of fairness — equal approval rates across groups, equal error rates across groups, and accuracy calibrated equally within each group — are mathematically incompatible whenever the underlying groups have different base rates in the data. A public body cannot satisfy every definition simultaneously; it can only choose, explicitly and with reasons it is willing to defend, which definition the decision at hand should optimize for, which is a policy judgment dressed as a technical one.

Explainability and the black box problem. A model is a black box when its internal workings cannot be inspected or understood from its inputs and outputs alone — true of many high-performing machine-learning models, whose accuracy often comes precisely from the complexity that makes them hard to interpret. Explainable artificial intelligence, or XAI, is the field of methods for making a model's reasoning inspectable, ranging from choosing inherently interpretable model types for consequential decisions to generating post-hoc explanations that approximate, rather than reveal, what a complex model actually did. The distinction matters legally as much as technically: an approximate explanation that sounds plausible but does not reflect the model's actual reasoning can satisfy a transparency checklist while failing the citizen who relies on it to contest the real basis for a decision against them.

The right to explanation is the contested legal principle that a person subject to a significant automated decision is entitled to a meaningful explanation of the logic involved, most closely associated with the European Union's General Data Protection Regulation, which restricts decisions based solely on automated processing that produce legal or similarly significant effects and gives the subject rights to obtain human intervention and to contest the decision (see right to explanation and General Data Protection Regulation). Legal scholars dispute how strong this right actually is in practice, and jurisdictions outside the European Union largely lack an equivalent statutory right, which is why a public body should not treat "we comply with the relevant data protection law" as equivalent to "we can explain this decision to the person it affects" — the two claims are related but not the same, and only the second is what due process actually requires.

Due process and natural justice. In administrative law, due process — the principle, most developed in United States constitutional law but echoed in comparable form across many legal systems, that the state must follow fair procedures before depriving a person of a right or entitlement — and natural justice — the Commonwealth administrative-law tradition requiring a fair hearing and freedom from bias in the decision-maker — both rest on an assumption written before computers existed: that a decision has an identifiable decision-maker who can give reasons, who can be shown free of a disqualifying bias, and who can be asked to reconsider. An algorithm strains all three assumptions at once — reasons may be technically unavailable, the "bias" in question is statistical rather than personal, and reconsideration by the same model produces the same answer — which is why due process for algorithmic decisions has to be engineered explicitly rather than inherited from procedures built for human decision-makers.

Risk-based regulation of AI. Rather than regulating all AI uniformly, an emerging regulatory pattern — most fully developed in the European Union's Artificial Intelligence Act — tiers obligations by the stakes of the application, prohibiting a narrow set of uses outright, imposing the heaviest compliance obligations on "high-risk" uses (which explicitly include public-sector eligibility determination, law-enforcement risk assessment, and migration decisions), requiring transparency disclosures for lower-risk uses, and leaving minimal-risk uses largely unregulated (see Artificial Intelligence Act). Whatever the regulatory pattern in a given jurisdiction, the underlying idea — that governance obligations should scale with the stakes of the decision, not the sophistication of the technology — is a useful design principle even without a direct legal obligation to follow the EU's specific regime.

Best practices

1. Classify every system by the stakes of the decision before choosing the model, not after. A system that flags a case for human review carries a different obligation than one that denies a benefit outright; decide which tier a use case sits in, using a framework such as the EU AI Act's risk categories as a reference point, before building rather than after deployment reveals the gap.
2. Tell people, in plain language, when an algorithm is involved in a decision about them. A generic privacy notice buried in terms and conditions does not satisfy this; the notice should say what the system does, what data it uses, and what happens if the person disagrees with its output.
3. Build a human-in-the-loop control that a reviewer can actually exercise, and measure whether they do. Track the override rate, the average time spent per case, and whether reviewers who disagree with the model face any friction that agreeing does not — if the override rate is near zero, the review is very likely decorative.
4. Test for disparate impact using real outcomes, not the developers' intentions. Before deployment and on a fixed recurring cadence afterwards, measure the model's outcomes disaggregated by protected and sensitive characteristics, including through likely proxies such as postcode or benefit history, because a model that never uses a protected variable directly can still produce a starkly unequal outcome.
5. Give every affected person a route to a human who has the power to overturn the algorithm's decision, not merely explain it. An appeal channel that restates the model's reasoning without a genuine possibility of reversal satisfies the letter of a right to explanation while defeating its purpose.
6. Prefer an interpretable model over a marginally more accurate black-box model for legally consequential decisions. Where the accuracy gap between an interpretable model and a complex one is small, the interpretability gain is usually worth far more to the citizen, the appeal officer, and the auditor than the marginal accuracy is worth to the organization.
7. Document each system's purpose, training data, known limitations, and intended use in a standing record, and update it when any of those change. A model deployed for one purpose and quietly repurposed for another — fraud screening extended to general eligibility triage, for instance — has usually left its original validation and bias testing behind it.
8. Keep training and operational data for a high-stakes model within a governed, purpose-limited pipeline. Reusing a dataset collected for one administrative purpose to train a model for an unrelated purpose is a data-governance failure as much as a modeling one (see Topic 5.4 — Public Sector Data, Interoperability, and Open Data).
9. Put algorithmic audit under a function that does not report to the team that built or operates the system. An audit conducted by the system's own developers finds different problems, and fewer of them, than one conducted by a genuinely independent reviewer with the standing to publish an adverse finding (see Topic 3.4 — Accountability, Transparency, and Legitimacy).
10. Run a shadow period before switching a system on for real decisions. Compare the model's recommendations against actual human decisions on the same cases for a defined period before it starts acting, so any systematic divergence surfaces before a citizen is affected by it.
11. Reserve full automation, with no human review at all, for the least consequential and most reversible decisions only. The more severe and less reversible a decision is — termination of a benefit, a fraud referral that triggers a criminal investigation, a denial of entry — the stronger the case for keeping a human genuinely in the loop, whatever the model's measured accuracy.
12. Retrain, revalidate, and re-test for bias on a fixed schedule, not only when a problem is reported. A model's accuracy and fairness properties can drift silently as the population and the world it was trained on change, and a public body that only investigates after a complaint has already let the drift run for however long the complaint took to surface.

Questions to discuss with your team

Where in our organization does an algorithm currently make or heavily inform a decision that affects a person's rights, entitlements, or liberty, and could every person affected actually find out that it happened? Many organizations discover, when they actually inventory their systems, that algorithmic scoring or triage has crept into more decision points than any single team realized, often introduced originally as a low-stakes efficiency tool. Use this question to build that inventory honestly, including systems procured from a vendor rather than built in-house, since procurement can obscure how much a purchased tool is actually deciding. Discuss whether your current notice practices would let an affected citizen discover, without specialist knowledge, that a model was involved in their case.

If a citizen formally challenged one of our automated decisions tomorrow, could we give them a real, specific reason, or only a description of the process? A process description — "the system considered your income, household size, and payment history" — is not the same as an explanation of why those inputs produced this particular outcome for this particular person. Use this question to pick one live system and try, as a team, to write the explanation a real citizen would receive if they appealed. If the exercise is difficult, that difficulty is itself the finding.

Do we know whether our training data reproduces a historical pattern of bias, and have we tested that against actual outcomes rather than against our intentions? Excluding a protected characteristic from a dataset is not the same as excluding its influence, because correlated proxies can carry the same signal under a different name. Use this question to identify which of your systems have been tested for disparate impact using real post-deployment outcomes, disaggregated by relevant characteristics, rather than tested only for the presence of a protected variable in the input data. Discuss what it would take to run that outcome-based test on your highest-stakes system if it has not already been done.

Is our human-in-the-loop review a genuine check, or a rubber stamp we have not measured closely enough to notice? A human reviewer under time pressure, without full context, or facing more friction for disagreeing than agreeing will tend toward approving the algorithm's recommendation regardless of its quality. Use this question to find your actual override rate for at least one system, and discuss what that number tells you about whether review is functioning as a control or as a compliance formality.

What actually happens, procedurally, when someone wants to appeal a decision our algorithm made or heavily informed — and does a human with real power to reverse it ever see the appeal? Many appeal processes route a citizen back through a description of the model's logic without ever reaching a person who can overturn the outcome on the merits. Use this question to trace one appeal, end to end, for a real or realistic case, and identify the exact point, if any, where a human with genuine authority to change the answer is involved.

Who owns the risk if one of our models degrades or drifts after deployment, and how would we actually know before a citizen or a journalist told us? A model's fairness and accuracy properties are not fixed at launch; they can shift as the population, the economy, or adjacent policy changes, often invisibly to the team that built the system. Use this question to identify who is accountable for post-deployment monitoring, on what cadence, and what specifically would trigger a retraining or a pause. If the honest answer is "nobody is specifically watching this," treat that as the most urgent finding this discussion can produce.

In practice: a public value example

A fictional national social-insurance agency built a risk-scoring model to prioritize which unemployment-benefit claims its fraud-investigation team should review first, after years of a backlog that meant genuine fraud sat uninvestigated for months while honest claimants waited on unrelated processing delays. The model was trained on five years of closed investigation files, learning which claim characteristics had historically preceded a confirmed fraud finding, and it performed well in testing: claims it scored as high risk were confirmed fraudulent at a much higher rate than a random sample, and the agency's leadership, watching its backlog fall for the first time in years, treated the rollout as an unambiguous success.

Eighteen months in, an internal ombudsman-equivalent office noticed something the fraud team's own performance dashboard had not surfaced: claimants living in a small number of postal districts, disproportionately lower-income and disproportionately recent migrants, were being flagged for investigation at nearly three times the rate of otherwise similar claimants elsewhere, and a much higher share of their flagged claims were closed with no fraud found at all. The pattern traced back to the training data: the historical investigation files reflected years in which field investigators, working from their own informal heuristics, had concentrated manual spot-checks in exactly those districts, so the model had learned to treat living there as predictive of fraud, when it was actually only predictive of having been checked more often in the past. Postal district had never been an input to the model; it did not need to be, because household composition, employer sector, and payment-method fields correlated closely enough with it to carry the same signal by proxy.

The agency's investigation confirmed the pattern was real, and also confirmed a second, quieter problem: the human reviewers who were meant to approve a case for full investigation before it proceeded had been approving the model's top-scored recommendations in the overwhelming majority of cases, with review times averaging under two minutes per case — nowhere near enough to meaningfully second-guess a statistical score, which meant the human-in-the-loop control leadership had pointed to as a safeguard had not actually been catching the disparity it existed to catch. The agency's response combined several of this topic's best practices at once: it retrained the model with the historical over-policing pattern corrected for, lengthened review time and gave reviewers authority to close a flagged case without escalation when their own judgment disagreed with the score, and built a plain-language notice and a genuine appeal route to a named case officer for anyone whose claim was flagged. Investigation yield per reviewer hour fell somewhat, but the confirmed-fraud rate among flagged claims from the previously over-flagged districts rose to match the rest of the caseload, which the agency's own audit treated as the real measure of success — not the volume of claims flagged, but whether flagging tracked actual fraud rather than a historical pattern the model had learned to reproduce.

Four sector lenses

Local government

Local authorities increasingly deploy algorithmic tools for the decisions with the most immediate and personal stakes for residents — housing-allocation priority, child-welfare risk scoring, and council-tax or benefit fraud triage — often while lacking the in-house data-science capacity to build, validate, or audit these systems themselves, leaving many to rely on vendor-supplied models whose training data and validation the authority cannot fully inspect. This creates a particular procurement discipline: a local authority commissioning an algorithmic system should demand, as a contractual condition, the same explainability, bias-testing, and audit access it would require of an in-house build, because a resident's due-process rights do not diminish because the authority bought the system rather than built it. Local government's proximity to residents also means an algorithmic misfire is more likely to become personally known in the community before it becomes a formal complaint, which argues for lightweight, early local review.

National government

National agencies operate the largest-scale systems in this topic's scope — tax fraud detection, immigration risk assessment, national benefit eligibility — where even a small error rate translates into a very large absolute number of affected people, and where the agency typically has the deepest technical and legal capacity to build genuine safeguards. National governments are also usually the level at which risk-based AI regulation, where it exists, is written and enforced, giving national agencies the clearest formal obligations and the greatest exposure when an audit or a court finds an automated system has produced discriminatory or unreviewable outcomes at scale. The due-process stakes are highest here in another sense too: an immigration or law-enforcement risk score can affect liberty as well as entitlement, which argues for the most conservative automation posture — the least full automation and the most genuine human review — precisely where the technical case for automation is often strongest because of case volume.

Social sector and nonprofit

Nonprofits and non-governmental organizations (NGOs) increasingly use algorithmic tools to triage aid distribution, screen beneficiaries for eligibility, or prioritize casework in exactly the domains this topic covers, often under real resource constraints that make the efficiency case for automation genuinely compelling. The accountability structure is markedly different from government, however: a nonprofit's beneficiaries rarely have the statutory appeal rights a citizen has against a state agency, and the organization's primary accountability relationship often runs to its funders rather than to the people the algorithm decides about, which is precisely the accountability gap Topic 3.8 — Social Sector and Nonprofit Management — describes for the sector generally. A nonprofit adopting algorithmic triage should therefore build the explanation and appeal rights this topic describes as a voluntary standard, not because a regulator requires it, but because the people the organization serves have no other route to one.

Multilateral and international

Multilateral bodies and international humanitarian agencies apply algorithmic targeting and risk-scoring tools across borders and legal systems, frequently in crisis settings where the speed benefit of automation is most acute and the data available is least reliable — incomplete registration data, data collected for one crisis and reused for another, and populations with limited ability to contest a scoring decision that determines whether they receive aid. No single national due-process law or right-to-explanation regime applies uniformly across a multilateral body's operations, so the safeguards this topic describes have to be built into the organization's own operating standards rather than inherited from an enforceable external legal framework, echoing the representational and accountability gap Topic 3.4 identifies for multilateral bodies generally (see Topic 3.4 — Accountability, Transparency, and Legitimacy). The stakes are also severe here, since an algorithmic decision can determine access to food, shelter, or asylum processing for people with the least practical ability of any group in this topic to appeal a wrong outcome.

Common failure modes

Proxy discrimination. A model excludes a protected characteristic directly but learns an equivalent signal from a correlated variable — postcode, surname, employment sector, benefit history — reproducing the same discriminatory pattern the exclusion was meant to prevent. The cost is a false sense of security: the organization can truthfully say the protected variable was never used, while the outcome remains just as unequal as if it had been.

Automation bias in human review. A human reviewer, formally required to check an algorithm's recommendation, approves it in the overwhelming majority of cases because disagreeing takes more time, more justification, or more personal risk than agreeing. The cost is that a control the organization believes is catching errors is, in practice, rubber-stamping them, and this usually only becomes visible once someone measures the override rate directly rather than trusting that the process exists.

Explainability theater. An organization provides an explanation for an automated decision that sounds specific and satisfying but is generated after the fact by a method that approximates, rather than reveals, the model's actual reasoning. The cost surfaces when the explanation is tested under real scrutiny — an appeal, an audit, a legal challenge — and turns out not to correspond to why the system actually produced that particular output.

Historical bias reproduced through feedback loops. A model trained on the outcomes of past decisions — which neighborhoods were investigated, which applications were approved — learns and then perpetuates whatever pattern of over-scrutiny or under-service shaped those past decisions, and each new round of decisions it produces becomes training data for the next version, deepening the pattern rather than correcting it. This has been documented across multiple public-sector domains internationally, including risk-based policing and welfare-fraud screening, and the underlying mechanism is consistent enough across those documented cases to treat as a general risk in any system trained on historical enforcement or casework data, regardless of jurisdiction.

The accountability gap, or "many hands" problem. When a decision runs through a vendor's model, an agency's data, a policy team's threshold setting, and a caseworker's nominal sign-off, responsibility for a wrong outcome can end up genuinely unclear even to the organization itself, with each party able to point to another as the source of the error. The cost is not only that the affected citizen cannot get a clear answer, but that the organization cannot reliably fix the actual cause, because no single accountable owner exists for the decision as a whole (see Topic 3.4 — Accountability, Transparency, and Legitimacy).

Maturity model

Checklist

• Build a complete inventory of every system that makes or materially informs a decision affecting a person's rights or entitlements, including vendor-supplied tools.
• Classify each system in that inventory by the stakes of the decision it informs, using a defined risk tier.
• Confirm every affected person receives a plain-language notice that an algorithm was involved in their case.
• Draft the specific, case-level explanation a real citizen would receive on appeal, and test whether it would actually satisfy them.
• Test each high-stakes system for disparate impact using real post-deployment outcomes, disaggregated by relevant characteristics, not only by checking for excluded input variables.
• Measure the override rate and average review time for every human-in-the-loop control, and treat a near-zero override rate as a finding.
• Confirm a named human, with genuine authority to reverse the decision, reviews every appeal.
• Place algorithmic audit under a function independent of the team that built or operates the system.
• Confirm training and operational data for each system stays within a governed, purpose-limited pipeline.
• Set a fixed retraining and revalidation schedule for every deployed model, rather than waiting for a complaint to trigger review.
• Identify which of your systems currently have no full automation safeguard for the most severe and least reversible decisions, and correct that first.
• Run the six discussion questions above with your team and record where the honest answer differs from the official one.

Key sources

• OECD AI Principles (2019, updated 2024) — the first intergovernmental standard for trustworthy AI, covering inclusive growth, human rights, transparency, robustness, and accountability.
• European Union Artificial Intelligence Act (Regulation (EU) 2024/1689) — the first comprehensive risk-tiered legal framework for AI, explicitly addressing public-sector eligibility, law-enforcement, and migration uses.
• US Government Accountability Office, Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities (GAO-21-519SP, 2021) — a governance, data, performance, and monitoring framework for AI accountability in the public sector.
• Danielle Keats Citron's scholarship on "technological due process," arguing that automated systems require procedural safeguards purpose-built for algorithmic, rather than human, decision-making.
• Cathy O'Neil, Virginia Eubanks, and Frank Pasquale's public-facing scholarship on algorithmic bias, automated welfare administration, and opaque scoring systems, cited throughout this topic's discussion of training-data bias and the black-box problem.

References

1. Algorithmic bias — Wikipedia — https://en.wikipedia.org/wiki/Algorithmic_bias
2. Explainable artificial intelligence — Wikipedia — https://en.wikipedia.org/wiki/Explainable_artificial_intelligence
3. Automated decision-making — Wikipedia — https://en.wikipedia.org/wiki/Automated_decision-making
4. Right to explanation — Wikipedia — https://en.wikipedia.org/wiki/Right_to_explanation
5. Due process — Wikipedia — https://en.wikipedia.org/wiki/Due_process
6. Fairness (machine learning) — Wikipedia — https://en.wikipedia.org/wiki/Fairness_(machine_learning)
7. Human-in-the-loop — Wikipedia — https://en.wikipedia.org/wiki/Human-in-the-loop
8. Black box — Wikipedia — https://en.wikipedia.org/wiki/Black_box
9. General Data Protection Regulation — Wikipedia — https://en.wikipedia.org/wiki/General_Data_Protection_Regulation
10. Artificial Intelligence Act — Wikipedia — https://en.wikipedia.org/wiki/Artificial_Intelligence_Act
11. Predictive policing — Wikipedia — https://en.wikipedia.org/wiki/Predictive_policing
12. Natural justice — Wikipedia — https://en.wikipedia.org/wiki/Natural_justice
13. OECD AI Principles overview — OECD.AI — https://oecd.ai/en/ai-principles
14. AI Act | Shaping Europe's digital future — European Commission — https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
15. EU AI Act: high-level summary — Future of Life Institute (artificialintelligenceact.eu) — https://artificialintelligenceact.eu/high-level-summary/
16. Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities (GAO-21-519SP) — US Government Accountability Office — https://www.gao.gov/products/gao-21-519sp