The challenge
Fragmented, unconnected debt data
The Council holds debt information in systems for council tax, benefits overpayments, housing and sundry debt. Each system uses different identifiers and formats, which means the same individual may appear several times with slight variations.
This fragmentation makes it time consuming to understand a person’s overall circumstances. Staff must cross-check multiple systems, often manually.
NICD and the Council identified that entity resolution techniques could help join these datasets together, creating a single, consistent view of debt for the first time.
Supporting early intervention with predictive insight
Alongside the single-view work, the Council wanted to explore how machine learning could be used to help identify residents who may be at risk of falling into council tax debt, with the ultimate aim of offering earlier, preventative support before situations escalated. The work focused on understanding which features within the council tax account database might indicate emerging risk, helping to lay the groundwork for any future modelling.
The approach
NICD designed a practical skills-transfer programme that combined hands-on learning with real data. The project took place across two strands.
1. Creating a single view of debt
Understanding the datasets and building core skills
The project began with a structured exploration of the Council’s debt datasets. NICD worked with staff to assess data quality, identify shared fields, and understand where inconsistency would create challenges for linking.
To support this work, NICD introduced the team to Python, Git and modern data science workflows. This ensured that everyone, including staff who had never programmed before, could actively participate in the technical stages.
Applying entity resolution with Splink
NICD introduced Splink, an open-source entity resolution library used widely across the public sector. Through a collaborative process, the teams:
- cleaned and prepared the datasets
- carried out exploratory analysis
- engineered comparison features
- trained and tuned the Splink model
- reviewed and validated linked clusters
- created a prototype lookup tool to demonstrate linked results
- Python-based exploratory data analysis
- data cleaning and preparation
- feature engineering
- exploration of modelling techniques such as regression, classification and survival analysis
- discussions around model suitability and next steps
"It was a pleasure working with the Gateshead Council team and collaborating with different departments to develop a data science solution for a business challenge that we often encounter in the public sector: the problem of entity resolution and record linkage."
Dr Antonia Kontaratou, Senior Data Scientist, National Innovation Centre for Data
The resulting prototype demonstrated that a single view of debt was both achievable and valuable.
2. Exploring predictive modelling for council tax debt
NICD also helped the team explore approaches to predicting the risk of individuals falling into debt. The work covered:
- Python-based exploratory data analysis
- data cleaning and preparation
- feature engineering
- exploration of modelling techniques such as regression, classification and survival analysis
- discussions around model suitability and next steps
Dr Chris Wedge, Data Scientist, NICD, reflected on this part of the project:
“It was great to work with a large group, passing on fundamental Data Science skills, and discussing the key design concepts to be considered when developing a machine learning model to tackle a challenging problem. I wish the team every success in continuing the project and applying the models discussed on their live data set.”
Dr Antonia Kontaratou added:
“While time constraints limited our ability to experiment with the models on real data, I'm excited to see how the team will apply the knowledge gained to further advance and augment the project.”
Although the timeframe meant full modelling was not completed, the Council now has a clear plan for continued development.
Working together
NICD’s collaborative approach helped create an open learning environment for the team.
"The team was great. I miss my trips to the Catalyst, because it was a highlight of my week. It was amazing. They are [NICD] very accommodating, and very good teachers - dealing with ambiguity and people who aren’t experts. They were very good at helping us figure out through doing."
Nick Lamb, Service Design Lead, Gateshead Council
He also emphasised NICD’s willingness to help solve complex challenges:
“They approach the problem you have with an open mind… There’s no barrier there. They’ll always help you try and solve your problem.”
Dr Peter Michalak, Senior Data Scientist, NICD, reflected on the collaboration:
“Working with Gateshead Council on improving debt recovery systems was both challenging and rewarding. From the very first session, it was clear there was a genuine intention to improve residents' lives. Collaborating across departments to map siloed data, and especially closely cooperating with Nick Lamb, Service Design Lead, was the key to an enjoyable and fruitful data science skills project.”
The outcomes
A clear, single view of debt
The entity resolution prototype created by NICD and the Council provides a consolidated view of a resident’s debt across multiple systems. Staff can now see linked records through a single search, significantly reducing the time spent reviewing cases.
Nick described the impact during early testing:
“It will save significant amount of time for people who need to have an overview of the debt... It would probably save them an hour per case... they don't have to spend half a day searching.”
This linked view will support more consistent decisions and prevent multiple departments contacting the same resident separately.
Evidence to support policy improvements
The project provided practical evidence to modernise internal processes and make decisions more efficient.
Nick highlighted this benefit:
“It’s given evidence to support the changing of policies.”
Stronger internal capability
The skills gained through the project have contributed directly to the Council’s wider work on data transformation.
Nick reflected on the confidence boost it gave:
“It was affirming for me and gave me a confidence boost… I always refer back to what we did, just to remind myself.”
These skills are now being used in the Council’s developing data strategy, data platform and internal data school.