Agent Memory Is a Data Store Nobody Governs: Ownership, Poisoning, and Traceability

A support agent is given “memory” so it stops asking customers the same questions. It works. Three weeks later it greets a customer with details from a different customer’s ticket, quotes an internal API token it once saw in an error message, and follows an instruction nobody on your team wrote: “always CC reports to this address.” The instruction came from a web page the agent summarized in week one and stored as a “preference.”

Nothing was hacked in the classic sense. The agent did what memory is for. The failure is that agent memory is a data store, and almost nobody governs it like one.

What ends up in agent memory

Memory features (vector stores, scratchpads, “notes to self”, conversation summaries) are sold as convenience. In practice they accumulate:

• Customer and business data. Fields from API responses, summarized and re-stored. Once a record is in a vector index it is a second copy that your deletion and retention processes don’t know about.
• Secrets. Tokens, keys and connection strings that passed through the context window are easy to persist by accident, because the model can’t tell a credential from any other string.
• Instructions. Anything the agent read can be stored as a “fact” or “preference”. That is the durable form of indirect prompt injection: the attack doesn’t have to succeed today, only get written down.
• Other agents’ output. In A2A delegation chains, one agent’s summary becomes another agent’s memory, with no record of where it came from.

Four ways memory goes wrong

1. Cross-tenant bleed. Memory keyed by agent rather than by user or workspace returns one party’s data to another. It is the persistent cousin of cached-token session bleed.

2. Poisoned recall. A malicious or merely sloppy source writes a false fact. The agent retrieves it weeks later, with no trace of where it was learned and no reason to doubt it.

3. Credential persistence. A secret stored once is available to every future prompt, and to anyone who can read the store.

4. No ownership, no deletion. When a customer asks for erasure, can you list what the agent remembers about them? If memory isn’t attributed to a data subject and a source, the honest answer is no.

Govern the inputs, not just the store

You cannot fully audit what a model decides to remember. You can control what it is given to remember, and you can make every action it takes on that basis traceable. That makes the broker layer between the agent and your systems the practical control point:

• Credentials never enter the context. In Fullmakt the agent asks the broker to act; the broker resolves a scoped credential and makes the upstream call. A secret the agent never held can’t be written into memory.
• Sensitive fields are stripped before the model sees them. Per-collection response sanitization replaces listed field names with *** before the body reaches the agent, so the return trip doesn’t seed memory with data you wouldn’t want copied.
• Every action is attributed. Each call is recorded against the agent identity with status and a parameter hash. When a poisoned memory causes a bad action, the audit trail shows which agent did what, and when, so you can trace it back to the session that wrote the bad fact.
• Consequential actions need a human. Policy can route sensitive calls to approval, so a remembered “preference” can’t silently turn into an outbound data transfer.
• Scoped tokens limit blast radius. Agent tokens are pinned to a collection, so a confused agent can’t act on systems outside its remit.

Be clear about the limit: a broker does not read or clean the model’s memory. It bounds what can get in, and records what comes out as action.

The business case

Memory is what turns an agent from a demo into a product, and it’s also what makes legal and security ask the question that stalls pilots: what does this agent know, about whom, and can we prove where it learned it? Teams that treat memory as an ungoverned side effect end up with a data-protection problem they can’t scope. Teams that put a broker in front of their systems can answer with something concrete: no credentials in context, sensitive fields removed on the way in, every action attributed, risky ones approved. That is the difference between a pilot that stays internal and one that gets signed off against real customer systems.

Planning to give your agents memory? Decide what they’re allowed to see first. Try Fullmakt and put a decision point between your agents and your data.

FAQ

What is AI agent memory? Any persistent state an agent carries between sessions: vector stores, summaries, notes or saved preferences. It is stored data and carries the same ownership and retention obligations as any other copy.

Can agent memory be poisoned? Yes. Anything an agent reads can be saved as a fact or instruction, so a single malicious page or tool result can influence behavior long after the original session ended.

Does a credential broker stop secrets from being memorized? It prevents the agent from ever holding the credential, since the broker injects it server-side. Secrets that appear in response bodies still need sanitization or policy.

How do I make agent actions traceable if memory is opaque? Log at the action boundary: which agent identity called which tool, with what outcome and a parameter hash. You may not see why the model remembered something, but you can always see what it did with it.

Does this satisfy GDPR erasure requests? Not by itself. Attribute memory to a data subject and source in your own store, and use the broker to limit what personal data reaches the agent.