Building Memory That Knows Who’s Asking
ReBAC-Gated Knowledge Graphs for Agents
TL;DR: Most agentic memory systems treat access control as an afterthought. This post describes a framework that combines temporal knowledge graphs with relationship-based access control so that authorization is structural and deterministic and not subject to the whims of LLMs. The pattern is portable to any agent framework - I implemented it for OpenClaw, but the architecture applies anywhere agents need to remember facts on behalf of multiple people.
The Challenge with Agentic Memory Today
Most agentic memory systems I’ve encountered up to this point treat access control as an afterthought - or ignore it entirely.
The default pattern goes something like this: the agent stores memories in a vector database and, when it needs to recall something, it searches everything it has access to and then applies a filter. If you want to restrict what the agent can see in a given context, you add a system prompt: “Don’t mention anything about the surprise party to Dad.”
This is security through politeness.
Prompt-based filtering is unreliable by nature. It’s subject to prompt injection. It fails under context pressure. And it fundamentally relies on the model choosing to comply - which is exactly the wrong trust boundary for anything sensitive.
We wouldn’t build a file system where permissions are enforced by asking users nicely not to open certain folders. Why would we build agentic memory that way?
The Core Idea: Two Graphs, One Query Path
The solution decomposes agentic memory into two complementary problems:
What does the agent know? This is the job of a temporal knowledge graph. Unlike flat vector stores, a knowledge graph extracts entities and facts from conversations and maintains them as structured, evolving knowledge: “Mark prefers dark mode,” “Mark is working on the AG-UI protocol,” “Dad’s birthday is March 15th.” Facts can be superseded, relationships can change, and the graph reflects that history.
Who’s allowed to know it? This is the job of a relationship-based access control (ReBAC) system. Inspired by Google’s Zanzibar, ReBAC evaluates permissions based on a graph of relationships - not static role lists or flat ACL tables. Access is structural: it follows from how entities relate to each other.
The query path composes them:
Agent Turn → ReBAC (who can see what?) → Knowledge Graph (search the authorized subset) → Context
The agent doesn’t decide what to filter. The authorization layer decides what exists.
The “Mark, Mom, and Dad ” Problem
Here’s the scenario that motivated the design.
Imagine a family assistant agent that interacts with three people: Mark, Mom, and Dad. Each person has private memories the agent should know about but never cross-pollinate:
Mark’s private group: Work preferences, projects, personal notes
Mom’s private group: Schedule, health notes, conversations with the agent
Dad’s private group: Schedule, plans for fishing trip with former colleagues
Then there are shared groups:
Family group: Shared calendar, recipes, vacation plans - everyone can access
Mom & Dad group: Parenting decisions, financial discussions - Mark can’t see these
Mark & Mom group: Planning surprise party for Dad - Dad shouldn’t see this
Mark & Dad group: Gift ideas for Mom - Mom shouldn’t see this
When Dad asks the agent “Is anyone secretly planning a party for my birthday?”, the system searches his private group, the family group, the Mark & Dad group and the Mom & Dad group. It does not search the Mark & Mom group (aka, the “everyone but Dad” group) - where the surprise party plans live, so it comes up blank.
Not because we told it not to : because the permission check returns that group as unauthorized, and the search never executes against it.
When Mark asks the same question, he gets his private group , the family group, the Mark & Dad group and the Mark & Mom group (where the party plans live). Different person, different relationships, different memories… enforced structurally.
The Portable Architecture
While I built this as an OpenClaw plugin, the architecture is framework-agnostic. Any agentic system that supports custom memory backends can implement the same pattern using two open-source building blocks.
The Knowledge Layer: Graphiti
Graphiti is Zep’s open-source temporal knowledge graph. It extracts entities and facts from conversational episodes, maintains them in a graph (backed by FalkorDB), and supports semantic search with temporal awareness. It’s doing the heavy lifting that a flat vector store can’t: structured knowledge that evolves over time.
But Graphiti (like most every knowledge graph and vector store I’ve evaluated) does not provide fine-grained, per-user authorization over individual memories. It assumes that anything within a given Graphiti namespace (e.g., group_id / tenant graph) is readable to whoever has API access to that namespace, so you must enforce auth and filtering in a separate layer. Everything stored is searchable by anyone with access to the API.
The Authorization Layer: spicedb
Spicedb is an open-source implementation of Google’s Zanzibar - the authorization system behind Google Drive, Photos, YouTube, and most of Google’s product suite. (AuthZed also maintains an excellent annotated version of the paper if you want the highlights without reading all 14 pages.)
The core idea behind Zanzibar is Relationship-Based Access Control (ReBAC): instead of assigning permissions through static roles (RBAC) or attribute rules (ABAC), access is determined by whether a chain of relationships exists between a subject and a resource. If Alice is a member of a group, and that group owns a document, Alice can access the document - not because someone added her to a role, but because the relationship graph connects her to it. This makes ReBAC particularly well-suited to agentic memory, where the relationships between people, conversations, and knowledge are the natural way to express who should see what.
Spicedb evaluates these relationship graphs using a declarative schema language called Zed.
Here’s the schema:
definition memory_group {
relation member: agent | person
relation contributor: agent | person
permission access = member
permission contribute = contributor + member
}
definition memory_fragment {
relation group: memory_group
relation creator: agent | person
permission view = group->access
permission delete = creator
}
Every memory fragment belongs to a group. You can only view fragments in groups you’re a member of. You can only delete fragments you created. The agent never even sees unauthorized memories - there’s nothing to leak, inject around, or socially engineer.
Composing Them
The integration point is a fan-out search that gates knowledge graph queries behind permission checks:
// Framework-agnostic pattern: fan-out search across authorized groups
const authorizedGroups = await spicedb.lookupResources("access", subject);
const results = await Promise.all(
authorizedGroups.map(groupId => graphiti.search(query, { group_id: groupId }))
);
This pattern doesn’t depend on OpenClaw. If you’re building on LangChain, CrewAI, Google ADK, or a custom agent loop, the same composition applies: check permissions first, then search only the authorized partition of the knowledge graph.
From Families to Organizations
The same pattern scales to any context where knowledge should be shared along relationship lines. Here are two concrete scenarios.
Slack History Ingestion
An organization’s Slack archive is a treasure trove of institutional knowledge -decisions made, problems solved, context shared. But not all of it should be accessible to everyone. With ReBAC-gated memory:
Public channel history → shared group for all channel members
Private channels → group limited to channel membership
DMs → group limited to the two participants
When an employee asks the agent a question, it searches only the channels and conversations they’re a member of. The knowledge graph captures relationships between concepts discussed across channels, while the authorization layer ensures each person only traverses the portion of the graph they have legitimate access to.
Meeting Transcripts
We can transcribe company-wide all-hands and town halls into a corporate-memory group everyone belongs to. Leadership meetings go into a leadership group. Department standups go into department groups. The agent builds a temporal knowledge graph of organizational decisions, priorities, and context - and each person gets the view that matches their actual organizational relationships.
When Bob asks “What was the decision on the API migration?”, the agent searches the engineering group where that discussion happened. When Alice asks - and she’s on the leadership team - she gets both the engineering discussion and the leadership context around why the migration was prioritized.
My Implementation: The OpenClaw Plugin
I built this as an OpenClaw plugin (@contextableai/openclaw-memory-graphiti) because OpenClaw’s replaceable memory slot made it the ideal proving ground - one plugin controls the entire memory pipeline: storage, recall, and capture. There’s no risk of the default memory system leaking around the authorization layer.
The plugin provides:
memory_recall - search the knowledge graph across all authorized groups, with session/long-term/all scoping
memory_store - save memories with automatic entity and fact extraction via Graphiti
memory_forget - delete memories (creator-only, enforced by spicedb)
Auto-capture - after every agent turn, key information is automatically extracted into the graph
Auto-recall - before every agent turn, relevant memories are automatically injected into context
Session isolation - each conversation gets its own memory group with exclusive ownership
The infrastructure runs on FalkorDB, spicedb, and PostgreSQL (as a backing store for spicedb). There’s a Docker Compose stack for easy deployment.
But the plugin is just one implementation. The architecture - ReBAC-gated knowledge graphs - is the transferable idea. If you’re building agentic memory on a different framework, the composition of any temporal knowledge graph + any Zanzibar-inspired authorization system gives you the same properties.
What I Learned
Authorization is an infrastructure problem, not a prompt problem. The moment you try to enforce access control at the prompt level, you’ve already lost. The model might comply 99% of the time, but the 1% failure mode is unintended information disclosure - the worst kind of failure to have be probabilistic.
Knowledge graphs and authorization graphs are natural complements. A knowledge graph builds a graph of what the agent knows. An authorization system builds a graph of who can know what. Composing them is more natural than bolting ACLs onto a vector store, because both systems already think in terms of entities and relationships.
The gap is real. Before building this for OpenClaw, I surveyed the landscape: Mem0 has a polished plugin but no authorization model. Cognee augments memory with graph retrieval but doesn’t address access control. Spicedb has an excellent RAG authorization tutorial for Pinecone, but nothing targeting temporal knowledge graphs specifically. The combination of temporal knowledge graphs with ReBAC authorization didn’t exist as a packaged solution.
The pattern is more general than the implementation. I built this for OpenClaw, but every design decision - the group-based memory partitioning, the fan-out search pattern, the schema separating membership from creatorship - applies to any agent framework. If you have a different knowledge store and a different auth system, the architecture translates directly.
What’s Next
The immediate roadmap: smarter incremental imports (currently it reimports everything), bulk ingestion for external sources (Slack exports, meeting transcripts, document repositories), and exploring spicedb’s caveated relationships for time-limited memory sharing - “share this memory group until the project ships.”
Longer term, I’m interested in inter-agent memory: multiple specialized agents with overlapping but distinct views of organizational knowledge, governed by the same authorization graph that governs human access.
If you made it to the end, you must really be interested : the code is at github.com/Contextable/openclaw-memory-graphiti. MIT licensed. PRs welcome.