When enterprise technology teams deploy machine-learning models to automate decisions, they assume the data feeding those models is reliable. But a sprawling predictive policing experiment in Bristol, England, shows what happens when that assumption breaks down — models get quietly abandoned, public trust erodes, and the whole system faces scrutiny.
The Think Family Database: A Half-Million-Person Risk Engine
Launched in 2016 by Bristol City Council and the regional Avon and Somerset Police, the Think Family Database holds records on close to half a million people, according to a joint investigation by WIRED, Liberty Investigates, the Bristol Cable, and Lighthouse Reports. The database stores police intelligence reports, housing status, mental health records, teenage pregnancies, enrollment in parenting courses, and free school meals. On top of this sensitive data, officials built machine-learning models to assign scores to thousands of adults and children, aiming to create what they called a “picture of threat, harm, and risk” in the region.
At an event in early 2022, one police data scientist described the approach bluntly: “I essentially dump all that data in a big bucket and stir it with a data-science spatula, and we come out with a lovely risk score for everybody.”
At Least 23 Models, Two Abandoned
The Think Family risk-scoring was just one part of Avon and Somerset Police’s predictive analytics program. Among at least 23 separate models were algorithms to predict the risk that people would commit burglary, fail to turn up in court, go missing, or become a victim of domestic abuse. A senior officer described creating a “league table” of the area’s most dangerous criminals — an apparent reference to the Offender Management App, designed to hold data on around 300,000 people.
However, previously unreported documents obtained by the investigation show that at least two of these risk-scoring models were quietly abandoned after Bristol City Council staff deemed they could no longer trust them. Government inspectors and independent reviewers highlighted a startling lack of transparency about some elements of the program and warned that the systems could undermine public trust.
| Model Type | Purpose | Status | Performance Note |
|---|---|---|---|
| Burglary risk | Predict likelihood of committing burglary | Active (status not specified) | — |
| Court non-appearance | Predict failure to appear in court | Active | — |
| Missing persons | Predict likelihood of going missing | Active | — |
| Domestic abuse victim | Predict risk of becoming domestic abuse victim | Active | — |
| Two additional models (unnamed) | Not specified in source | Abandoned | Staff could no longer trust them |
Transparency Gaps and Public Awareness
How the police developed and used these predictive tools wasn’t always clear to the public. John Pegram, leader of a local police accountability group in Bristol, said he didn’t hear about the Offender Management App until 2023, years after its creation. When he did learn about it, he began to suspect he might be included: “I think I knew I was on the app,” Pegram said.
In early 2024, Pegram filed a request to find out how the police were using his data. The police refused to say. Months later, after Pegram hired solicitors, police confirmed he was on the app but declined to elaborate. Like others across Bristol, the UK, and increasingly around the world, Pegram didn’t know whether he had been scored by an algorithm, what that score might be, or how it could affect his interactions with authorities.
Performance Data Raises Red Flags
Police data disclosed to WIRED comprising more than 36,000 model performance scores appear in some cases to show “genuinely poor predictive performance,” according to an independent analyst who reviewed the data. The findings come as the UK appears poised to embrace predictive analytics and artificial intelligence across the criminal justice system. A familiar face is helping lead the charge: the former chief constable of Avon and Somerset, Andy Marsh, who now heads the national standard-setting body for forces across England and Wales. As CEO of the College of Policing, Marsh has said that effective AI should be “injected like heroin” to speed up British police work.
The Bristol case offers a cautionary tale for any enterprise deploying AI in high-stakes contexts. When models are built on flawed data, trained without transparency, and evaluated only after deployment, the result is not just wasted investment — it is a breakdown of trust that can take years to repair. For supply chain, logistics, and trade technology leaders, the lesson is clear: algorithmic accountability begins with the data.
Sources:
