Retrieval, Memory, and Governance Are Three Different Problems

Three questions that reveal whether your AI agents have what they need — or whether you're building on gaps.

We've been building Personize — our memory and governance infrastructure for AI agents. Along the way, I've found that three questions help explain what agents actually need, where the gaps are, and what's critical for each.

I'm sharing this as a talking doc. Something you can bring into your team and ask: Do we care about these three things? If yes, how are we handling them today? And if we're not, what's the risk?

The three questions are:

1. "What's relevant?" — That's retrieval.
2. "What do we know about this entity?" — That's memory.
3. "What are the rules?" — That's governance.

Each of these three questions requires different infrastructure. Mixing them up won't break your system with a catastrophic failure. It will break it with the kind of slow drift that's hard to diagnose until it isn't.

"What's Relevant?" — That's Retrieval

RAG is the most mature of the three. It works. Graph-based retrieval, agentic multi-step pipelines, hybrid search with re-ranking — the ecosystem has gotten really good at this. The global RAG market is projected to surpass $40 billion by 2035 for a reason.

The question RAG answers is: "Given a query, what information in our corpus is most similar?"

You have a knowledge base. Documents, manuals, case studies, product docs. An agent needs information. Retrieval finds the most relevant chunks, ranks them by semantic similarity, and returns them as context.

What retrieval does well:

• Finds relevant information across large document collections
• Reduces hallucination by grounding responses in real sources
• Adapts search depth to query complexity
• Scales to millions of documents

What retrieval was never designed to do:

• Tell the difference between a draft proposal and a board-approved policy
• Make sure the most authoritative content wins (RAG has opinions about similarity, not authority)
• Push constraints into every agent interaction, whether the agent asked or not
• Check whether an agent's output actually follows organizational rules
• Give you an audit trail back to a versioned, accountable source

The key thing: retrieval is query-driven. If the agent doesn't ask the right question, it doesn't get the right context. And it retrieves the most similar information, which isn't always the most authoritative.

"What Do We Know About This Entity?" — That's Memory

Memory answers a different question: "What do we know about this specific entity, and how does that knowledge evolve over time?"

An entity is a person, a company, an account, a customer. Over time, you learn things about them. Their preferences. Their history. Their constraints. Their relationships. Memory is about keeping that knowledge alive across interactions — not searching a corpus for what's similar to the current query.

Two examples.

You talk to a customer on Monday. They mention their account is up for renewal in Q3 and they prefer email over phone. On Wednesday, a different agent talks to the same customer. The second agent should know about the Q3 renewal and the email preference — not because it retrieved them, but because they were recorded as entity knowledge. That's memory.

A sales team learns that Prospect XYZ prefers technical conversations with their VP of Engineering, has had three failed implementations with competitors, and is looking specifically for Salesforce integration. That institutional knowledge about that entity should be accessible to every sales agent, forever — not surfaced by a lucky query, but maintained as persistent state.

What memory does well:

• Keeps entity-specific knowledge across sessions and across agents
• Builds understanding of a person or organization over time
• Gives context that's specific to a relationship, not just a query
• Makes personalization possible without re-retrieving past conversations every time

What memory was never designed to do:

• Find relevant information about arbitrary topics (that's retrieval)
• Enforce organizational rules and constraints (that's governance)
• Handle information that should apply to everyone, not just one entity

Memory is entity-specific and persistent. It answers "what do we know about this person?" not "what's relevant to this query?" When you try to store organizational rules inside entity memory — mixing "what we know about this customer" with "what the company's pricing policy is" — you get something that's impossible to update at scale. When the pricing policy changes, you're updating thousands of entity records instead of one place.

What we learned building memory

In Personize, we introduced schema-enforced extraction. You define the properties you care about (job title, pain points, budget, timeline) and the system extracts them from content. This ensures you never miss insights about customers as soon as they appear in the data, and it makes querying and routing significantly more effective.

But it has limits. A prospect mentions three competing vendors on a sales call, but your schema doesn't have a "competitive landscape" property. That insight just vanishes.

To cover that gap, we added open-set extraction alongside it — where the system pulls out any fact it finds regardless of whether the schema asked for it. On its own, open-set gives you everything but nothing structured to query against.

Running both simultaneously on every piece of content is what actually works. Structured properties and open-set contextual facts, extracted from the same source. Across thousands of records in production, the open-set facts are frequently the ones that matter most for personalization — the things no one thought to put in the schema.

The other thing we learned the hard way: extraction without quality gates just creates noise. Before any fact reaches the memory store, it has to pass validation for completeness, self-containment, coreference resolution, temporal anchoring, and deduplication. In our first batch across 2,500 records, 12% of candidate facts were near-duplicates — usually a meeting transcript and its follow-up email describing the same discussion. Without dedup at write time, those duplicates compound and quietly degrade recall quality over months.

"What Are the Rules?" — That's Governance

Governance answers: "What is this agent allowed to do, and how should it operate within organizational constraints?"

Governance isn't about information. It's about rules and enforcement. Policies, boundaries, constraints, validation gates, audit trails. It's the thing that makes sure agent behavior stays within organizational rules — whether the agent asks about those rules or not.

Governance concerns include:

• Policies: Pricing rules, brand voice guidelines, compliance requirements, approval workflows
• Constraints: Never offer more than 15% discount without VP approval. Never mention competitor names in writing. Always get authorization before charging.
• Validation: Before an output reaches the customer, check that the pricing is accurate, the tone matches brand guidelines, and the commitment is something the company actually honors.
• Audit trails: Every agent decision is traceable back to the specific policy that governed it, with version history.
• Ownership: Marketing owns brand guidelines. Legal owns compliance rules. Finance owns pricing. Domain teams maintain their own rules, and the system ensures every agent consumes the current version.

What governance does well:

• Enforces rules at system boundaries, regardless of what the agent asked
• Gives you audit trails connecting outputs to specific, versioned policies
• Keeps things consistent across every agent and platform
• Catches problems before they reach customers, not after

RAG doesn't have opinions about authority. It has opinions about similarity. Those are not the same thing.

Governance is universal and proactive. Rules apply whether the agent asks for them or not. The policy finds the agent, not the other way around. Singapore's Model AI Governance Framework for Agentic AI — launched at Davos in January 2026 as the first dedicated framework for agentic systems — says it directly: organizations must implement "technical controls and processes" that bound agent behavior proactively, not just provide information on request.

What we learned building governance

We almost used RAG to deliver governance context. It seemed natural. You have policies, you have a vector store, just retrieve the relevant ones.

Here's what actually happened. The query "write a cold outbound email" returned chunks about email server configuration because of semantic similarity. A pricing policy competed for relevance with a product FAQ that happened to mention the word "pricing." The system was retrieving policies, but it had no idea which ones were authoritative for the task at hand.

That failure pushed us to build something different: a tiered routing engine that matches policies, compliance rules, and playbooks to the right agent for the right task based on explicit criteria — not semantic similarity. It reduced candidate governance variables from 15–60 down to 1–3 critical selections per task. In early production, that translated to up to 70% token savings, because agents get only the governance context they actually need instead of everything vaguely related.

The lesson that stuck with me: governance routing and retrieval look similar on the surface. Both deliver context to an agent. But they work on completely different principles. Retrieval optimizes for relevance. Governance routing optimizes for authority and applicability.

The Three Mistakes

Mistake 1: "We have RAG, so we have governance."

You put pricing policies in a vector store. The agent retrieves them. That's retrieval, not governance. The policy might be stale. It might compete for relevance with a draft someone uploaded six months ago. If the agent violates it, there's no validation gate. You have a searchable filing cabinet, not a governance system.

Mistake 2: "Our memory system is our knowledge base."

You build a system that remembers entity facts — a customer's preferences, an account's constraints. Then you try to store organizational rules that should apply to everyone in the same place. You mix "what we know about this customer" with "what the company's pricing policy is." When the pricing policy changes, you update it in thousands of entity records instead of one.

Mistake 3: "Our retrieval pipeline handles governance."

You build an agentic RAG pipeline that proactively retrieves governance policies alongside domain knowledge before every generation. The agent gets the policies as context, interprets them (or doesn't), and generates an output that may or may not follow them. Retrieval gave the agent information. It didn't enforce rules. There's no validation gate.

The numbers back this up: 40–60% of RAG implementations fail to reach production due to retrieval quality issues and governance gaps. If you're stopping at retrieval and assuming governance is covered, you're accumulating that risk.

How They Work Together

You need all three, layered correctly.

Retrieval finds domain knowledge. Agent needs to know how to integrate with Salesforce, or what your API rate limits are? Retrieval finds it.

Memory keeps entity-specific knowledge. Agent talks to the same customer three times? Memory knows they prefer email, have budget authority for deals under $50k, and expressed interest in a specific feature last month.

Governance enforces the rules. Before the agent's output reaches the customer, governance checks: Is the pricing accurate? Does the tone match brand guidelines? Is there a compliance concern that needs human review?

Here's a concrete example. An agent helping a customer renew their contract:

1. Retrieval surfaces the relevant contract renewal process documentation and pricing templates.
2. Memory provides entity knowledge: "This customer has been with us for 3 years, renewed twice before, prefers email, and has authority to commit up to $100k without escalation."
3. Governance validates before delivery: Are the renewal terms consistent with their account tier? Does the discount follow pricing policy? Is the tone on-brand? Flag anything unusual for human review before it sends.

If any layer is missing:

• Without retrieval, the agent has no baseline knowledge of renewal processes
• Without memory, the agent re-establishes context the customer already provided, wastes their time, and misses history that should inform the offer
• Without governance, the agent might offer a discount the company wouldn't approve, use the wrong tone, or make a commitment that contradicts the contract terms

The confusion happens because people call all three "context" or "knowledge." But they operate at different levels. Mix them up and you get a retrieval system that can't enforce policies, a memory system bloated with organizational rules, and a governance layer that's actually just a RAG query.

How Do You Know Your Memory Actually Works?

This is the question that keeps coming up. Not "does retrieval return results," but "does the system actually remember accurately across long conversations, multiple sessions, and evolving facts?"

We benchmark Personize against LoCoMo, an independent benchmark for long-term conversational memory. It tests the scenarios that break naive approaches: multi-session reasoning, temporal updates, open-ended inference from accumulated context.

Personize scores 74.8% overall accuracy on LoCoMo — the highest among the enterprise memory systems we've seen evaluated. On open-ended inference, the hardest category where the system has to synthesize facts that were never explicitly stated, we now exceed human-level performance under benchmark conditions.

That last part surprised us too. But it makes sense when you think about it. Open-ended inference is exactly where retrieval stops and memory starts. Retrieval can find the chunks. Memory can synthesize them into understanding. The benchmark made that gap measurable for us, not just something we felt architecturally.

The Test That Reveals the Gap

Ask your team three questions:

1. Retrieval test: If an agent asks "how should I write a cold email?", can it find your email best practices in your knowledge base? If yes, you have retrieval.

2. Memory test: If the same agent talks to the same customer three times, does it remember the customer's preferences, history, and constraints from previous conversations — without the customer mentioning them again? If yes, you have memory.

3. Governance test: If an agent tries to offer a discount that violates company policy, does something stop it before the customer sees it? If yes, you have governance.

If you're missing any of these, you have a gap. It won't hit you with a catastrophic failure. It'll hit you with drift — agents making small decisions that violate policy, knowledge going stale, entity context lost between conversations, retrieval surfacing old guidance that contradicts current rules.

Gartner says over 40% of agentic AI projects are at risk of cancellation by 2027 if governance, observability, and oversight aren't established early. The teams that will avoid that aren't the ones with the best retrieval. They're the ones that understood these as three separate problems and addressed all three.

The industry has spent two years getting retrieval right. The next two years will be about memory and governance. That's what we're building at Personize — and if you want to see what all three layers look like working together, explore the platform.

References

• Squirro — "RAG in 2026: Bridging Knowledge and Generative AI"
• Singapore IMDA — "New Model AI Governance Framework for Agentic AI" (Jan 2026)
• MIT Technology Review — "From Guardrails to Governance: A CEO's Guide for Securing Agentic Systems" (Feb 2026)
• Gartner — "Over 40% of Agentic AI Projects Will Be Canceled by End of 2027" (Jun 2025)
• BusinessWire / ResearchAndMarkets — "RAG Industry Report 2025–2035: Global RAG Market to Surpass $40 Billion by 2035" (Oct 2025)
• AWS — "Amazon Bedrock AgentCore Adds Quality Evaluations and Policy Controls" (Dec 2025)
• Anthropic — "Building Effective Agents" (2025)