Neo4j

Introduction

As organisations continue to adopt generative AI, one of the biggest challenges is ensuring large language models (LLMs) provide accurate, reliable answers based on trusted information rather than producing inaccurate or fabricated responses.

Neo4j, the knowledge layer for enterprise AI, provided a grant for the National Innovation Centre for Data (NICD) to carry out independent research and investigate whether knowledge graphs can improve the performance of AI systems that use Retrieval Augmented Generation (RAG), to access external data and documents.

The project produced independent evidence that graph-based approaches can improve the truthfulness of AI-generated responses to complex questions by 80%, with a more than 2x increase in questions answered, with only a very modest increase in the total token usage. The published preliminary study helped to validate a key hypothesis underpinning Neo4j’s AI strategy.

The challenge

Neo4j had observed positive results when combining graph databases with AI applications, but wanted independent scientific evidence to objectively validate whether knowledge graphs genuinely improve the performance of RAG systems when compared with traditional semantic search methods.

Dr Jim Webber, Chief Scientist at Neo4j, explained:

“We started to form a hypothesis about graphs and AI being a really good pairing. But, we wanted independent scientists to perform research so we could be certain of the outcomes without bias."

The approach: developing a scalable AI foundation

NICD's data science team designed and delivered an experimental framework to compare multiple approaches to AI-powered question answering.

The project began with an extensive review of academic literature to identify suitable datasets and evaluation methodologies. As Neo4j wanted the research to remain entirely independent, NICD selected all datasets and benchmarks without client influence.

The team chose the MoNaCo benchmark dataset, which contains more than 1,300 complex multi-step questions specifically designed to challenge modern AI systems. They also identified Meta's CRAG evaluation metric as particularly well-suited to the study because it does not merely measure whether answers are right or wrong. Instead, it penalises AI systems more heavily when they provide confident but incorrect answers than when they appropriately respond that the answer cannot be determined.

To conduct the research, NICD built a large-scale knowledge graph containing more than five million nodes and developed AI agents capable of querying the graph structure. The team then compared graph-based retrieval against more conventional vector-search approaches across hundreds of benchmark questions.

Dr Chris Wedge, Data Scientist at NICD, said:

“It was exciting in this project to have total free rein to build an experiment working with challenging benchmark datasets rather than building a solution specific to a client's own data.”

Throughout the project, NICD repeated experiments multiple times to ensure the results were reproducible and statistically robust.

The results

The research demonstrated that knowledge graph-based retrieval can improve the truthfulness of AI-powered question answering systems. With 510 complex questions tested, the study saw an:

80% improvement in truthfulness versus standard vector RAG, and
more than 2x increase in questions answered, with
a modest increase in the tokens used

The study also found that graph-enhanced RAG systems achieved a better balance between correct and incorrect answers than conventional semantic search approaches, particularly when models were encouraged to refuse answering when sufficient evidence was unavailable. While the conventional solution ran in loops to find an answer, which often ended up with, “I don’t know”, as the answer, the graph-based system was able to search for the relevant facts in the right places. The latter used about 10% more tokens but they were spent on achieving twice as many correct answers as the former.

One of the most significant findings was that the performance improvement did not require a highly complex knowledge graph. Even a relatively simple graph representing document structure, such as sections, paragraphs, and relationships between content, produced measurable benefits.

Chris explained:

“It was really interesting to see how our work, while framed in graph terminology, in effect simply allows an LLM to look for answers as I would in a book, using the table of contents to find the right chapters and sections to read. In conventional approaches, it is as if the LLM only has the index, so it might find some specific pages of interest, but it isn't allowed to look at pages on either side, so it doesn't get the full picture.”

This discovery presents new opportunities for organisations seeking practical ways to improve AI systems without the complexity of building highly detailed knowledge graph representations.

Business impact

For Neo4j, the project delivered independent validation that graph databases can play a valuable role in improving AI accuracy.

The findings provide evidence to support Neo4j's go-to-market offerings, while also helping the organisation focus future development efforts on making it easier for customers to transform document collections into knowledge graphs.

Jim said:

“Our hypothesis that this pairing of graphs and large language models wrapped in an agent would be powerful has been borne out by the research that the NICD team did.”

The independent nature of the research was particularly valuable, providing credibility and confidence that the findings were based on sound scientific methodology rather than commercial interest.

Reflecting on the collaboration, Jim added:

“Your team [NICD] took the time to understand it. They owned it, and they delivered.”

Looking ahead

The success of the project has already identified further opportunities for collaboration between Neo4j and NICD. Building on these findings, Neo4j is exploring future research into how graph-based approaches could support more advanced AI memory systems and agent-based architectures.

For NICD, the project demonstrates how independent, research-led data science can help organisations validate emerging technologies, reduce uncertainty, and make evidence-based decisions about future innovation.