Code lists: the unglamorous fix for messy public data

A code list is a fixed set of allowed values for a field, each with an agreed code and meaning: countries, languages, marital status, types of licence. It is about the least exciting thing in public sector data. It is also one of the most effective. When two systems use the same list, “Belgium”, “BE” and “Belgique” stop being three answers to one question.

The European Interoperability Framework (EIF) counts agreements on reference data, including code lists, controlled vocabularies and thesauri, among the key prerequisites for semantic interoperability.² This piece looks at why they work and how to use them well. For the bigger picture, see semantic interoperability.

Why code lists do so much

Most mismatches between public systems aren’t caused by bad data entry. They come from each system having its own list of options. One records employment status with six values and another with nine; one splits “widowed” from “surviving civil partner” and another doesn’t. Every exchange between them needs a mapping, and every mapping is a place where meaning leaks.

A shared list removes the mapping. It also makes meaning explicit, because a good list doesn’t just give codes but says what each one covers.

A good list says what it is not

The EU Publications Office maintains authority tables, also called named authority lists, to harmonise codes and labels across systems and make exchanges between institutions easier.¹ One detail shows why careful definitions matter. The table for types of administrative territorial unit warns that its units must not be confused with the statistical regions of the Nomenclature of Territorial Units for Statistics (NUTS), which are defined by population, or with the areas courts cover, even where they coincide in some countries.¹

Three bodies can each mean something different by “region”. A list that says so up front prevents a whole class of errors that no amount of data cleaning would catch.

Reuse before you create

W3C’s Data on the Web Best Practices asks publishers to reuse existing vocabularies, preferably standardised ones, rather than inventing near-duplicates.⁴ For public bodies the order of preference is usually clear:

1. an international or EU list where one exists, such as the EU authority tables;
2. a national list maintained by a recognised body;
3. a list shared with the other organisations you exchange data with;
4. only then, a local list, published so others can see what it means.

Catalogues show the payoff. DCAT-AP, the European standard for describing datasets, makes the EU’s data theme vocabulary the mandatory code list for classifying datasets by theme, which is what lets records from many portals be filtered together.⁵ There is more on that in DCAT-AP and data catalogues.

Publishing a list so machines can use it

A code list in a PDF helps people. A code list on the web, with a stable identifier for each value, helps systems too. W3C’s Simple Knowledge Organization System (SKOS) is the common format for this: it gives thesauri, classification schemes and code lists a shared web representation, with preferred labels and broader and narrower relationships, without forcing them to become formal ontologies.³

That last point matters. SKOS is explicit that the hierarchies in a thesaurus or classification scheme don’t have formal logical semantics, which is exactly what makes them cheap to publish and easy to adopt.³ The guide to ontologies and core vocabularies explains where a list ends and an ontology begins.

Keeping a list alive

Code lists change: countries are renamed, new licence types appear, categories are merged. A list nobody maintains is worse than none, because systems keep trusting it after it has gone stale. A few habits help:

• Name an owner and publish how to request a change.
• Version every release and keep a history of what changed, as W3C guidance asks for published data generally.⁴
• Retire, don’t delete. Mark a withdrawn code as deprecated and keep its identifier working, so old records still make sense.⁴
• Say what changed meaning, not just what changed label. A renamed value and a redefined value need different handling downstream.

Start small

The EIF recommends an information management strategy set at the highest possible level of an organisation, with priority for metadata, master data and reference data.² That can sound like a large programme. In practice it can start with one field: pick the drop-down that causes the most mapping work between two services, agree a shared list, publish it with definitions and an owner, and move on to the next. It is slow, unglamorous work, and it is the kind that lasts.