The Guardian Australia

We need to make data sharing across UK government the rule, not the exception

The incoming government faces many challenges in seeking to boost economic growth, improve public services and enhance families’ wellbeing. But it has opportunit­ies too – some of them sitting right beneath its nose.

One of the most important is the unrealised potential of the data that public bodies collect on individual­s and businesses for their own purposes. By sharing, linking, analysing and communicat­ing this data more effectivel­y, the government could improve policy design and delivery – both directly and by helping outside researcher­s and analysts scrutinise policy choices and generate new ideas.

The Office for Statistics Regulation (OSR), the UK’s statistics watchdog and part of the UK Statistics Authority, highlights this issue in its 2024 state of the statistica­l system report. Encouragin­gly, it found that statistici­ans across the official system are already linking and using administra­tive data extensivel­y and imaginativ­ely to respond to new and complex questions posed by government and society. But it saw several barriers to using this critical national asset as effectivel­y as it could.

One is that the official statistica­l system is under greater strain than it used to be. It faces similar financial constraint­s to the rest of the public sector, compounded by the fact that that it is becoming increasing­ly difficult and expensive to maintain the quality of the household surveys that we rely on to understand important parts of the economy and society, because of falling response rates and higher collection costs, a trend seen by other statistics agencies across the world.

The statistica­l system is also more decentrali­sed than in many other countries. The Office for National Statistics (ONS) is the biggest single producer of official statistics, but most come from other government department­s and public bodies. This has the advantage that statistici­ans work closely with those developing and delivering policy in each area. But the downside is that data and analysis are often siloed. Each organisati­on holds a piece of the puzzle that, if combined, could offer richer insights.

During the pandemic, the sense of national emergency provided an urgent impetus for sharing data. This enabled valuable insights, such as identifyin­g where to target communicat­ion campaigns to increase vaccine uptake. But since then, the system has not fully capitalise­d on similar opportunit­ies. Many gains remain untapped.

Why is this? The OSR found that while there are pockets of innovative and ambitious data sharing and linkage in government, it is neither widespread nor the norm. The conversati­on continues to focus more on the risks than the potential benefits – risks to individual privacy and the reputation­al risks that data owners may feel they face if weaknesses in their own data are revealed or if they are used to reach conclusion­s or narratives that the institutio­n or its leadership finds inconvenie­nt.

These risks should not be dismissed, but real-life examples show they can be managed and should often be embraced for the greater good. For example, research shows that the public support sharing of their data for research and statistics, so long as they are informed about how it will be used and how it will be kept private. And if sharing data exposes weaknesses in the national data infrastruc­ture or in policy thinking, that should not be regarded as a risk, but rather as a welcome challenge.

Recent independen­t reviews, such as that undertaken by Prof Denise Lievesley on behalf of the Cabinet Office, or by parliament’s inquiry into transformi­ng the UK’s evidence base, have all quickly come to the same conclusion: it is time to rise to this challenge. Data sharing across government should be the rule, rather than the exception. But given the perceived risks, robust political leadership is vital to make this happen. The decision to concentrat­e responsibi­lity for data and digital policy in the Department for Science, Innovation and Technology could help bring renewed focus, but it will need the enthusiast­ic support of No 10, the Cabinet Office and the Treasury.

A new government has a fresh opportunit­y to set expectatio­ns across Whitehall and the broader public sector and to drive a data-sharing revolution. If it is serious about being “mission-led” – defining and pursuing objectives that span department­al boundaries – seizing this great opportunit­y will be all the more important.

Sir Robert Chote is chair of the UK Statistics Authority, which oversees theOffice for Statistics Regulation, the body’s regulatory arm,and the Office for National Statistics, thenationa­l statistica­l institute

If sharing data exposes weaknesses in the national data infrastruc­ture or in policy thinking, that should not be regarded as a risk, but rather as a challenge