Something quietly critical is happening inside production AI agent systems right now. As OpenAI's Memory feature matures and frameworks like LangGraph bring agent persistence into mainstream engineering workflows, UK development teams are shipping systems with a fundamental flaw baked in: they are treating vector embeddings as a complete memory solution. They are not. Research comparing retrieval architectures on multi-hop reasoning tasks puts the accuracy of vector-only retrieval at around 32%, against 86% for graph-based memory. That is not a marginal performance gap — it is the difference between an agent that reasons coherently across a conversation and one that confidently hallucinates.
This matters now because the stakes have changed. Agents are no longer demos. They are handling customer queries, synthesising internal knowledge, drafting compliance summaries, and executing multi-step workflows inside real organisations. When memory architecture underperforms, the failure is not a minor UX irritant — it is wrong answers delivered with the confidence of correct ones. Senior technical leads need to understand what is actually happening beneath the surface, and why the fix is architectural rather than cosmetic.
What Multi-Hop Reasoning Actually Demands
Multi-hop reasoning describes the cognitive task of connecting information across several discrete steps to reach a conclusion. A simple example: a user asks an agent why a particular supplier was removed from an approved vendor list. The answer requires linking a procurement policy update, a risk assessment report, a board decision, and the contract termination record — none of which exist in the same document or embedding cluster. The agent must traverse a chain of related facts, not simply retrieve the single most semantically similar passage to the query.
Vector databases excel at one thing: finding content that is semantically close to an input. They surface relevant fragments with impressive efficiency. But semantic proximity is not logical connectivity. When an answer requires assembling a chain of evidence — where fact B only becomes meaningful once you have retrieved fact A, and fact C depends on B — pure embedding retrieval breaks down structurally. It lacks the traversal mechanism the task requires. This is why the accuracy gap is so pronounced specifically on multi-hop questions, rather than simpler single-turn lookups where vector retrieval performs adequately.
The Architecture Most Teams Are Currently Using
The default pattern that has emerged in most agent builds follows a straightforward flow: user input is embedded, a cosine similarity search retrieves the top-k chunks from a vector store, those chunks are injected into the context window, and the language model generates a response. This is a functional and relatively simple architecture to implement, and for many use cases it works well enough. The problem is that teams are extending this pattern into domains where it was never designed to operate — complex enterprise knowledge graphs, multi-document reasoning, and persistent agent memory that needs to accumulate and relate information over time.
The result is systems that appear to work during isolated demonstrations but degrade under real usage conditions. An agent might answer the first two questions in a workflow correctly, then lose coherent thread when the third question requires it to synthesise what it learned in the first two exchanges. OpenAI's Memory feature and LangGraph's persistence layer give developers tools to store information between sessions, but if that stored information is only retrievable via embedding similarity, the underlying retrieval problem remains unsolved. Persistence without relational structure is just a bigger pile of unconnected facts.
Graph Memory and the Case for Hybrid Retrieval
Graph-based memory represents information not as floating vectors but as nodes and edges — entities and the explicit relationships between them. When an agent processes a document or a conversation, a graph memory system extracts entities (people, decisions, dates, policies, products) and maps how they relate to one another. Retrieval then becomes traversal: the agent can follow a chain of relationships to assemble a coherent, multi-step answer. This is why graph-based retrieval scores so dramatically higher on multi-hop tasks. It is architecturally suited to the problem in a way that vector similarity search simply is not.
The practical recommendation is not to abandon vector retrieval — it remains valuable for fast, fuzzy semantic search across large corpora. The recommendation is to treat it as one layer within a hybrid retrieval stack. Graph memory handles relational reasoning and multi-hop chains. Vector search handles broad semantic discovery. A reranking layer, increasingly common in mature retrieval architectures, combines candidate results from both sources and selects the most contextually appropriate content for injection. Tools such as Neo4j, Microsoft's GraphRAG, and LangChain's graph integrations make this pattern increasingly accessible for production teams, though implementation complexity should not be underestimated.
Evaluating Your Current Memory Architecture
Before investing in a refactor, it is worth diagnosing whether your current agent actually has a multi-hop problem. The clearest signal is this: ask your agent questions that require connecting information from at least two or three distinct sources or prior exchanges. If accuracy degrades noticeably compared to single-source queries, you are experiencing the retrieval ceiling that vector-only systems impose. Log the failure cases. Most teams find that a disproportionate share of hallucinations and incoherent responses cluster around exactly these relational query types.
A second diagnostic is examining how your agent handles long-running tasks and conversations. Does it maintain consistent context across a ten-message thread, or does it effectively start fresh each time? If it struggles to reference and build on earlier exchanges, the memory layer is not functioning as true memory — it is functioning as a contextual lookup with significant gaps. Identifying this early, before a system is embedded in a critical workflow, is considerably less costly than remediation after deployment.
The architectural lesson here is straightforward, even if the implementation is not: embeddings are a retrieval tool, not a memory system. Conflating the two produces agents that feel intelligent in simple scenarios and fail in precisely the situations where reliable reasoning matters most. As agent capabilities become central to how UK organisations manage knowledge, automate decisions, and interact with customers, the quality of the memory architecture underneath will increasingly determine whether those systems add genuine value or erode trust through confident inaccuracy.
If your team is currently building or scaling an AI agent and you have not explicitly designed the memory layer to handle relational, multi-hop retrieval, now is the right time to address it. The gap between a vector-only approach and a well-constructed hybrid stack is not a future optimisation problem — the accuracy data suggests it is already a present limitation. iCentric works with technical and product teams to audit existing agent architectures and design memory systems that hold up under real-world reasoning demands. If this is a challenge your organisation is navigating, it is worth having the conversation early.
What is the practical difference between vector memory and graph memory in an AI agent?
Vector memory stores information as numerical embeddings and retrieves content based on semantic similarity to a query. Graph memory stores information as entities and explicit relationships, enabling the agent to traverse logical chains between facts. Vector memory is good at finding relevant content; graph memory is good at reasoning across connected information — these are distinct capabilities that serve different retrieval tasks.
Is the 32% vs 86% accuracy figure specific to a particular benchmark or broadly applicable?
These figures come from evaluations on multi-hop question answering tasks, which are specifically designed to require connecting information across multiple sources or reasoning steps. They are not representative of all retrieval tasks. On single-hop or direct lookup queries, vector retrieval performs significantly better. The gap narrows as task complexity decreases and widens as relational reasoning demands increase.
What is GraphRAG and how does it differ from standard RAG?
GraphRAG, popularised by Microsoft Research, augments the standard Retrieval-Augmented Generation pattern by building a knowledge graph from source documents rather than relying solely on vector embeddings. At query time, it retrieves information by traversing graph relationships rather than just performing similarity search. This makes it substantially more capable on questions that require synthesising information from multiple connected sources.
Can we add graph memory to an existing LangGraph or LangChain agent without rebuilding from scratch?
It depends on how tightly the current memory layer is coupled to the agent logic. LangChain and LangGraph both have integrations with graph databases such as Neo4j, and it is often possible to introduce a graph retriever as an additional tool or memory component without a full rebuild. However, you will also need a process to extract and populate the graph structure from your existing data, which requires careful design.
What types of enterprise use cases most urgently need graph-based memory?
Any use case where an agent needs to reason across interconnected documents, policies, or data records benefits significantly from graph memory. High-priority candidates include regulatory compliance monitoring, legal and contract analysis, enterprise knowledge management, financial due diligence, and any workflow where decisions depend on tracing a chain of prior events, approvals, or policy changes.
How does a reranking layer work within a hybrid retrieval stack?
A reranker takes candidate results retrieved from multiple sources — typically a vector search and a graph traversal — and scores them for relevance to the original query before injecting them into the context window. Cross-encoder reranker models evaluate the query and each candidate together, producing more contextually accurate relevance scores than the initial retrieval step alone. This reduces noise in the context and improves answer quality.
Does OpenAI's native Memory feature use graph memory or vector retrieval?
OpenAI's Memory feature for ChatGPT and the Assistants API primarily stores and retrieves discrete facts as text memories, with retrieval influenced by semantic relevance to the current conversation. It does not expose a graph traversal mechanism. For production applications requiring structured relational reasoning, teams typically need to implement their own memory architecture rather than relying on OpenAI's built-in feature.
What volume of data is needed before graph memory becomes worth the implementation overhead?
Graph memory offers advantages even at modest data volumes if the reasoning task is inherently relational. A knowledge base of a few hundred interconnected documents can still produce multi-hop failures with vector-only retrieval. The decision should be driven by the complexity of the reasoning task rather than data volume alone. If your use case involves tracing relationships between entities, graph memory is likely warranted regardless of scale.
How do we measure whether our current agent has a multi-hop retrieval problem?
Design a test set of questions that require connecting information from at least two distinct sources or prior conversation turns to answer correctly. Run these against your agent and compare accuracy to single-source questions. Clustering your existing hallucination or error logs by query type is also revealing — if relational or cross-document questions are significantly over-represented in failures, you have a retrieval architecture problem rather than a model capability problem.
Are there open-source tools that support hybrid vector and graph retrieval?
Yes. Neo4j offers both vector and graph retrieval capabilities and integrates with LangChain. Weaviate supports hybrid search combining vector and keyword retrieval, with graph-style relationship modelling on its roadmap. Microsoft's GraphRAG implementation is open-source on GitHub. For teams building on LlamaIndex, there are graph index components that can be combined with vector retrievers in a hybrid pipeline.
Get in touch today
Book a call at a time to suit you, or fill out our enquiry form or get in touch using the contact details below