Public Sector Data, Interoperability, and Open Data

Why this matters in public value

Every public body already holds more data than it uses well. A tax authority, a hospital network, a transport agency, and a local planning office each collect information as a by-product of running a service, and each discovers, usually too late, that the record it needs to answer a citizen's question or settle a dispute is sitting in a system nobody else in the organization can query. The cost is not abstract: a citizen who has already told one agency their address should not have to tell it again to another agency inside the same government.

Two separate ambitions sit inside this topic's title, and public bodies routinely confuse them. The first is interoperability — making data move reliably between systems that different teams and eras of technology built independently, so one part of government can act on what another part already knows. The second is openness — publishing data outside government entirely, so journalists, researchers, and citizens can use it for purposes the publishing body never anticipated. Interoperability is mostly an engineering and governance problem; openness is a public value in its own right, of the same family as the public goods described in Topic 1.4 — Public Goods, Merit Goods, and Value Pluralism, because published data is non-rivalrous and hard to exclude anyone from once released, so a market alone would under-supply it.

Both ambitions collide with a third fact this topic treats as a genuine, unresolved tension rather than a problem with a tidy answer: data detailed enough to be useful is often detailed enough to identify the individual it describes, even after names and obvious identifiers are stripped out. A government that publishes richly and a government that protects the people in its records well are pulling in different directions on the same lever, and the growing appetite for data to train and evaluate algorithmic systems (see Topic 5.2 — AI and Algorithmic Decision-Making in Government) has raised the stakes, because more actors than ever want public data at exactly the granularity that makes it riskiest to release. A leader who treats governance, interoperability, and openness as one problem, with privacy as its permanent constraint, is better placed to capture this topic's value than one who treats each in isolation.

Core concepts

Data governance is the set of decision rights, standards, and accountabilities that determine who may create, change, access, and retire an organization's data, and to what standard of quality it must be held (see data governance). It is distinct from data management, which is the operational work of moving and storing data day to day; governance decides the rules, management executes them. A public body without data governance still has data, but nobody can say with confidence which copy of a citizen's record is the authoritative one, or who is accountable when that record turns out to be wrong.

A data steward is the named individual or role accountable for a specific dataset's quality, definition, and appropriate use, distinct from the technical staff who operate the systems it lives in (see data steward). Stewardship matters because data quality decays by default: a field that nobody is accountable for accumulates inconsistent entries, duplicate records, and silent drift between what a field is supposed to mean and what staff actually put in it. Assigning a steward to a dataset converts data quality from an occasional clean-up exercise into an ongoing, named responsibility.

Interoperability is the capacity of independently built systems to exchange data and use it correctly without special-case, one-off engineering for every pair of systems involved (see interoperability). It operates at three levels that are easy to conflate: technical interoperability (can the systems connect and move bytes at all, typically through an application programming interface, or API, a defined way for one piece of software to request data or action from another — see application programming interface); semantic interoperability (do both systems mean the same thing by a shared field, such as "household" or "active case"); and organizational interoperability (do the two organizations agree on why the data is being shared and who is accountable for what happens next). Most costly interoperability failures happen at the semantic or organizational level even when the technical connection works perfectly.

A common schema, or data standard, is an agreed, published structure for how a category of data is represented — field names, formats, units, and permitted values — adopted by more than one system or organization so data produced by one can be consumed by another without manual translation. Government bodies that adopt a shared schema for something as basic as an address or a case status convert dozens of potential bilateral translation problems into one standard every new system can simply implement. A unique identifier that persists across systems and over time — a case number, a business registration number, a citizen reference — is the specific mechanism that lets records about the same person or entity be linked correctly, and getting identifier design wrong (reusing numbers, changing formats, or lacking one at all) is one of the most common, avoidable causes of interoperability failure.

Metadata is data about data — the description of a dataset's structure, provenance, update frequency, and meaning, published alongside the data itself so a user who did not create it can understand and trust it (see metadata). A dataset published without adequate metadata is barely usable by anyone outside the team that produced it, however open its license; metadata is what turns a file dump into something a third party can actually reuse.

A data-sharing agreement is a formal arrangement between agencies, or between a public body and an external partner, that specifies the purpose data may be used for, the legal basis for sharing it, how long it may be retained, and what happens if the purpose changes. Purpose limitation — restricting use of shared data to the purpose it was collected or shared for — is the discipline that keeps a data-sharing agreement from becoming an unbounded license to repurpose personal information, and its absence is a recurring theme in public trust failures involving cross-agency data.

Open data is data that is made freely available for anyone to access, use, and redistribute, for any purpose, typically machine-readable and released under a license that permits reuse rather than merely viewing (see open data). Its public value case follows the public goods logic set out in Topic 1.4 — Public Goods, Merit Goods, and Value Pluralism: once a dataset is published, one person's use of it does not diminish another's, and government is often the only actor with both the legal standing and the scale to collect and publish it in the first place, so private markets alone chronically under-supply this kind of information. Open data enables accountability (journalists and citizens checking government's own claims against its own records), third-party innovation (businesses and civic technologists building services on top of public data government itself will never build), and better public decisions elsewhere in the economy, all without the publishing body having to anticipate every use in advance.

Re-identification is the process of matching supposedly anonymized data against other available information to work out which real person a record describes, and it is the central technical fact that makes the privacy tension in this topic unavoidable rather than merely inconvenient (see data re-identification). Removing a name, or an obvious identifier such as a national insurance or social security number, does not remove the risk: a combination of quasi-identifiers such as age, postcode, and a rare medical condition can uniquely identify someone even in a dataset that never named them, especially once it is cross-referenced against another, separately public dataset. K-anonymity is one formal way of reasoning about this risk: a dataset satisfies k-anonymity if every combination of quasi-identifying fields it contains is shared by at least a defined number, k, of records, so no single row can be isolated to fewer than k possible people (see k-anonymity). Differential privacy is a more recent, mathematically rigorous approach that adds carefully calibrated statistical noise to a dataset or to the results of queries against it, so that the presence or absence of any single individual's record changes the published output by only a bounded, provable amount, regardless of what an attacker already knows (see differential privacy). Privacy by design is the broader principle that privacy protection should be built into a system's architecture from the outset, as a default setting, rather than bolted on as a redaction step just before publication (see privacy by design); redacting obvious identifiers after the fact is the weakest form of this discipline, not a substitute for it.

Best practices

1. Name an accountable steward for every dataset that matters, not only a central data office. A data office can set standards, but it cannot know whether a specific field in a specific system still means what it was defined to mean; that judgment belongs to someone close to the data's actual use.
2. Publish open by default, and document the specific reason whenever you do not. Reversing the default — requiring a justification to withhold rather than a justification to publish — surfaces genuine sensitivities for individual review instead of letting caution apply to everything indiscriminately.
3. Adopt an existing, shared schema before designing a new one. A bespoke data structure that fits one project perfectly and matches nothing else in government guarantees a future integration cost; check what your sector, jurisdiction, or an international standards body has already published before building your own.
4. Design persistent, well-governed unique identifiers before a system goes live, not after. Retrofitting an identifier scheme onto records that have been accumulating for years under inconsistent keys is dramatically more expensive than getting the design right at the outset.
5. Build data exchange through governed APIs rather than one-off file transfers. An API that enforces a schema, logs access, and can be versioned gives you a controllable, auditable interoperability point; a spreadsheet emailed between departments gives you none of those things and nobody notices when it goes stale.
6. Write data-sharing agreements that state the purpose, the legal basis, and the expiry, not just the technical mechanism. An agreement that specifies how data moves but not why, or for how long, tends to outlive the purpose that justified it and quietly become permission for something nobody explicitly authorized.
7. Treat de-identification as a risk assessment with a residual result, not a one-time redaction task. Ask what other public or purchasable datasets could be linked against the one you are about to release, and size the re-identification risk against that realistic threat model, not against an idealized attacker with no outside information.
8. Apply differential privacy or an equivalent formal technique to any release involving small populations or rare characteristics. Simple suppression or generalization of obvious identifiers is insufficient once a dataset includes anything unusual enough to be identifying on its own, such as a rare condition, a small local population, or a high-value transaction.
9. Maintain a public, current data catalog with real metadata, not just a list of file names. A dataset a potential user cannot understand well enough to trust is functionally unpublished, however permissive its license; catalog entries should say what a field means, how often it updates, and what its known limitations are.
10. Measure data quality against defined standards on a recurring schedule, and report the results to whoever owns the dataset. Quality that is checked only when something breaks decays invisibly for long stretches in between; a routine measurement turns decay into a visible, manageable trend.
11. Review every published dataset periodically for continued public value and continued risk. Populations change, re-identification techniques improve, and a release that was safe five years ago may not be safe today against data that has since become newly available elsewhere.
12. Separate the decision to share or publish data from the technical team that moves it. The people best placed to judge purpose, proportionality, and privacy risk are rarely the same people best placed to build the pipeline, and collapsing the two roles into one leaves the harder judgment under-examined.

Questions to discuss with your team

Do we know which system holds the authoritative record for each of our core data entities, and does everyone actually treat it that way? Many organizations have, in practice, several "true" copies of the same information — a citizen's address in the billing system, the case-management system, and the correspondence log, all slightly different and all being acted on by someone. Use this question to list your organization's core entities (citizen, case, property, business) and identify, honestly, whether a single authoritative source exists for each or whether staff have quietly learned which system to trust for which purpose. Discuss what it would take to designate and enforce one authoritative source per entity, and what would break in the transition. This is usually cheaper to fix early than after a serious decision has been made on the wrong copy.

If a person asked us why they had to give the same information to two parts of our own organization, what would our honest answer be? The honest answer is often "because our systems cannot exchange it," which is an interoperability failure dressed up as a process requirement. Use this question to pick one service journey that spans more than one team or system and trace, step by step, where the same fact is captured more than once. Discuss whether the repeat request is protecting something genuine — a fresh consent, a verification step — or whether it is simply the absence of a data-sharing agreement or a common schema. Consider what it would cost, in citizen trust as well as staff time, to leave it unfixed for another year.

Where would a determined journalist or researcher most plausibly re-identify someone in a dataset we currently call anonymized? Most organizations that publish de-identified data have not stress-tested it against a realistic combination of quasi-identifiers and other public datasets. Use this question to pick your highest-risk published or soon-to-be-published dataset and walk through what other information — a local news report, another agency's open dataset, a public register — could plausibly be linked against it. Discuss whether your current de-identification approach (suppression, generalization, or a formal technique such as k-anonymity or differential privacy) would actually withstand that combination, and who is responsible for re-testing it as new external data becomes available.

Who currently has the authority to say a dataset should not be published, and are we confident that authority is being used for the right reasons? A default-to-open policy still needs a credible, well-reasoned exception process, and the person exercising it should be identifiable and accountable. Use this question to check whether withholding decisions in your organization are made by a named role against documented criteria, or informally, by whoever happens to be asked and erring on the side of caution. Discuss whether your withholding rate looks more like careful case-by-case judgment or a standing institutional reflex, and what a periodic external review of withheld datasets might reveal.

Are our data-sharing agreements between agencies still being used for the purpose they were written for? Purpose limitation is easy to state in an agreement and easy to erode quietly in practice, especially once a dataset proves useful for something its original signatories never anticipated. Use this question to pick two or three live inter-agency data-sharing arrangements and check the actual current use against the written purpose and expiry date. Discuss what governance step is missing if a mismatch turns up, and who should be accountable for catching that kind of drift before an external audit or a journalist does.

If our interoperability or data standards became a barrier to a legitimate new use of our data, would we find out before or after it caused real harm? Standards that were sensible when adopted can ossify into obstacles as needs and technology change, and organizations rarely build in a mechanism to notice this happening to their own rules. Use this question to discuss how your organization currently reviews whether its data standards, identifier schemes, or sharing processes still serve their purpose, versus simply being inherited and unquestioned. Consider whether the people who would notice a standard has become an obstacle — front-line staff, partner organizations, external developers using your open data — have a real channel to raise it, and whether that channel has ever actually changed a standard.

In practice: a public value example

A regional social-welfare agency and a separate regional health authority — fictional, illustrative of a pattern rather than a report on any real body — had spent a decade running incompatible case-management systems, so a family known to both could be recorded under different reference numbers, with no reliable way to tell agency staff that the household appearing in a housing-support caseload was the same household appearing in a paediatric-outreach caseload. A joint program to identify children at risk of falling through the gap between the two services stalled repeatedly, not for lack of political will, but because nobody could produce a single, trustworthy combined view of which families needed both kinds of support.

The two agencies eventually adopted a shared household identifier and a common schema for the fields both systems needed to exchange, built through a governed API rather than the periodic spreadsheet exports each had relied on previously, under a data-sharing agreement that named the specific purpose — coordinating support for at-risk households — and required annual review. Within a year, caseworkers on both sides could see, for the first time, which families were known to the other service, and joint case conferences that had previously relied on manual cross-checking became routine. The agencies then went further, publishing an aggregated, area-level open dataset showing the number of households receiving combined support by district, intended to help local charities target their own outreach and to let elected councillors see whether the joint program was reaching the areas it was designed for.

A local data-rights advocate, working with a university researcher, discovered that the published district-level figures could be cross-referenced against a separately published, equally well-intentioned open dataset from the local transport authority — showing subsidized-fare applications by small area — to narrow some districts down to household counts small enough that specific families could plausibly be identified, particularly in sparsely populated rural districts where both datasets counted single-digit numbers of households. Neither dataset alone was disclosive; combined, in the least populated areas, they were. The agencies' first instinct, to withdraw the open dataset entirely, would have destroyed the accountability and charity-targeting value it had already begun to deliver. Instead, working with the same advocate, they re-published the dataset using a differential-privacy approach that added calibrated noise to small counts, preserving its usefulness for spotting broad regional patterns while making any single low-population district's true count uncertain enough to defeat the cross-referencing attack that had exposed the original release. The episode became the case study the agencies used in their own subsequent staff training, for why de-identification must be assessed against what else is publicly available, not against the dataset in isolation.

Four sector lenses

Local government

A local authority typically runs more distinct, older, and less-integrated systems per resident served than a national body does, because decades of individually procured departmental systems — housing, planning, waste, social care — were rarely built to share data with each other. This makes interoperability a genuinely internal battle before it is ever an external one: a resident interacting with three council departments in one week may find that none of them can see what the others already know. Local open data, meanwhile, tends to have unusually direct civic value — bin collection schedules, planning applications, local spending — because residents and local journalists engage with it at a level of specificity national datasets rarely offer, which raises the stakes on getting metadata and update frequency right even for modest, everyday datasets.

National government

National governments typically operate the fullest range of data governance and interoperability instruments — a national data strategy, cross-government identifier schemes, mandated open-data portals, and dedicated data-protection regulators — but the sheer number of legacy systems accumulated across decades and administrations makes technical interoperability a persistent, expensive background task rather than a one-off project. National open-data programs carry disproportionate re-identification risk precisely because national datasets are large enough to be genuinely valuable and detailed enough, once linked against other national registers, to identify individuals; the national data strategy documents several governments now publish increasingly name privacy-preserving disclosure techniques explicitly, rather than treating de-identification as an implementation detail beneath policy notice.

Social sector and nonprofit

A nonprofit or non-governmental organization (NGO) is rarely subject to an open-data mandate, but it increasingly faces a parallel expectation from funders, regulators, and the public that program data be shared transparently, and it often works with populations — survivors of violence, people in acute poverty, undocumented residents — for whom re-identification risk carries a far higher personal cost than it typically does for the general population in a government dataset. Data-sharing agreements matter disproportionately here because a nonprofit frequently shares client data with government funders or partner organizations under contracts it did not write, and it needs the capacity to negotiate purpose limitation rather than simply accept whatever data-sharing terms are attached to funding. Interoperability standards adopted by larger government or philanthropic funders can be a genuine burden on a small organization's limited technical capacity, which is a real cost worth naming rather than treating standards adoption as costless.

Multilateral and international

Multilateral bodies aggregate data across member states with vastly different governance maturity, statistical capacity, and legal regimes for privacy, which makes semantic interoperability — agreeing what a shared field actually means — a persistent diplomatic as well as technical problem; a term as basic as "household" or "unemployed" is defined differently enough across countries that international statistical bodies maintain entire standards, such as the Statistical Data and Metadata Exchange (SDMX) format, purely to make cross-country data genuinely comparable. Open data at this level functions as an especially clear public good in the sense set out in Topic 1.4 — Public Goods, Merit Goods, and Value Pluralism, because no single member state has an incentive to fund a global comparative dataset on its own, yet every state and researcher benefits once a multilateral body publishes one. Data sovereignty concerns run deeper here than in a single-country context: a member state may resist sharing data it is legally entitled to withhold, or may dispute how a multilateral body has classified or aggregated its own national figures, which makes the governance question of who controls a shared statistic as consequential as the technical question of how it is exchanged.

Common failure modes

Silent data silos. Systems accumulate independently across an organization with no shared identifier or schema, and the cost stays hidden until a specific case — a safeguarding failure, an audit, a service outage — requires a combined view that turns out to be impossible to produce quickly, at which point the silo's cost becomes visible all at once and expensive to fix under pressure.

Anonymization theater. A dataset has names and obvious identifiers removed and is then treated as safe to publish without any assessment of quasi-identifiers or linkage risk against other available data. The cost surfaces, often publicly and embarrassingly, when a researcher, journalist, or advocate demonstrates re-identification is possible, at which point the organization's credibility on privacy is damaged well beyond the specific dataset involved.

Governance without an owner. Data governance policy exists on paper, but no specific person is accountable for any specific dataset's quality or currency, so standards decay quietly and nobody notices until the data is relied on for a decision it cannot actually support. This is a particularly common failure because the policy document itself looks complete, masking the absence of anyone actually accountable underneath it.

Interoperability by mandate, without resourcing. A shared schema or identifier scheme is mandated centrally but never funded for the teams expected to adopt it, so compliance becomes a checkbox exercise — a field technically present but not properly populated — rather than genuine, usable interoperability. The cost is a false sense that a shared standard exists, discovered only when someone actually tries to rely on the data it was supposed to standardize.

Open-washing. An organization publishes dashboards and summary visualizations and calls this open data, while the underlying machine-readable dataset a third party would need to build something new remains unpublished or published in a format nobody can practically reuse. This captures the appearance of transparency and openness described in Topic 3.4 — Accountability, Transparency, and Legitimacy without its substance, and it is often mistaken for a genuine open-data program because the dashboard looks impressive.

Maturity model

Checklist

• Identify the authoritative source system for each of your organization's core data entities.
• Confirm every dataset that materially matters has a named, accountable steward.
• Check whether a shared schema or standard already exists for a new data project before designing a bespoke one.
• Review your unique identifier design for persistence and consistency across the systems that use it.
• Replace at least one recurring manual or spreadsheet-based data exchange with a governed API.
• Confirm every active inter-agency data-sharing agreement states its purpose, legal basis, and expiry, and matches current actual use.
• Run a re-identification risk assessment, including plausible linkage against other public datasets, before publishing any dataset involving personal information.
• Apply a formal privacy-preserving technique, such as k-anonymity or differential privacy, to any release involving small populations or rare characteristics.
• Confirm your open-data catalog includes real, current metadata, not just file names.
• Review previously published datasets against newly available external data for emerging re-identification risk.
• Confirm withholding decisions are made by a named, accountable role against documented criteria.
• Ask your team the six discussion questions above and record where the honest answer differs from the official one.

Key sources

• International Open Data Charter, its six principles, and its guidance on open-by-default publishing.
• OECD Recommendation of the Council on Enhancing Access to and Sharing of Data (2021).
• US National Institute of Standards and Technology, Special Publication 800-226, Guidelines for Evaluating Differential Privacy Guarantees.
• US Census Bureau material on disclosure avoidance and its application of differential privacy to decennial census data.
• Cynthia Dwork's foundational work formalizing differential privacy, and the broader academic literature on statistical disclosure control.
• Mark H. Moore's strategic triangle and the public goods logic set out in Topic 1.4 — Public Goods, Merit Goods, and Value Pluralism, as the framing this topic builds on.

References

1. Data governance — Wikipedia — https://en.wikipedia.org/wiki/Data_governance
2. Data steward — Wikipedia — https://en.wikipedia.org/wiki/Data_steward
3. Interoperability — Wikipedia — https://en.wikipedia.org/wiki/Interoperability
4. Application programming interface — Wikipedia — https://en.wikipedia.org/wiki/Application_programming_interface
5. Metadata — Wikipedia — https://en.wikipedia.org/wiki/Metadata
6. Open data — Wikipedia — https://en.wikipedia.org/wiki/Open_data
7. Data re-identification — Wikipedia — https://en.wikipedia.org/wiki/Data_re-identification
8. K-anonymity — Wikipedia — https://en.wikipedia.org/wiki/K-anonymity
9. Differential privacy — Wikipedia — https://en.wikipedia.org/wiki/Differential_privacy
10. Privacy by design — Wikipedia — https://en.wikipedia.org/wiki/Privacy_by_design
11. Open Data Charter principles — International Open Data Charter — https://opendatacharter.org/principles/
12. Recommendation of the Council on Enhancing Access to and Sharing of Data (2021) — OECD Legal Instruments — https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0463
13. Guidelines for Evaluating Differential Privacy Guarantees, SP 800-226 (Initial Public Draft) — US National Institute of Standards and Technology — https://csrc.nist.gov/pubs/sp/800/226/ipd
14. Disclosure avoidance and data stewardship — US Census Bureau — https://www.census.gov/about/policies/privacy/statistical_safeguards/disclosure-avoidance.html
15. Data.gov — US federal open data portal, General Services Administration — https://www.data.gov/