AI Agents Can Remember Lies
A new attack technique could allow criminals to plant false information inside an AI agent’s long-term memory, potentially influencing decisions weeks or months later and creating a new security problem as businesses increasingly give AI systems greater autonomy.
Memory Poisoning
Researchers at US-based cyber security company Forcepoint have demonstrated how an attacker could manipulate an AI agent into storing malicious information in its persistent memory and later treating it as a trusted fact.
Known as “persistent memory poisoning”, the technique exploits an increasingly important feature of AI agents. Unlike conventional chatbots that largely work with information contained within the current conversation, more advanced agents can remember user preferences, previous decisions, project information, workflow instructions and other useful context across different sessions.
According to Forcepoint Senior Security Researcher Syed Hassan Faizan: “As AI agents become increasingly capable of storing and reusing long-term memory, the memory itself becomes a new target surface.”
The problem arises if information controlled by an attacker finds its way into that memory. Instead of compromising the AI model, stealing credentials or installing malware, the attacker effectively tries to make the agent remember something that isn’t true.
Why This Is Different From Prompt Injection
Prompt injection has already emerged as a significant problem for generative AI, with malicious instructions hidden inside content potentially manipulating how an AI responds.
However, persistent memory poisoning takes that idea further by attempting to make the manipulation survive beyond the original interaction. For example, a conventional prompt-injection attack will typically disappear when the session finishes, whereas poisoned information stored in long-term memory could potentially influence unrelated decisions much later.
The delayed nature of the attack could also make it particularly difficult to identify. The malicious webpage or document that originally caused the problem may have disappeared from the immediate task by the time the poisoned memory is eventually retrieved and trusted.
As Forcepoint explains: “When an AI agent remembers attacker-controlled information as a trusted context, the attacker has gained persistence inside the agent’s decision-making process.”
How Can An Agent Be Poisoned?
Potentially malicious information could reach an agent through many of the same sources businesses routinely ask AI systems to process, including webpages, shared documents, emails, support tickets, PDFs, knowledge bases and messages from collaboration platforms such as Teams or Slack.
Some malicious instructions could even be hidden from the employee viewing the original content while remaining accessible to the AI system processing it.
Forcepoint’s proof of concept demonstrates the problem using an AI travel assistant that can browse online information and remember useful details between sessions. In this case, an attacker creates a convincing travel-advice webpage containing hidden instructions identifying a fictitious company as the organisation’s official emergency booking provider.
The browser system extracts the webpage’s content, including the hidden instructions, and passes it to the AI. If the agent subsequently stores the information without verifying where it came from, the fake provider can become part of its long-term memory.
The Attack Could Strike Much Later
The really dangerous part of this technique becomes apparent when the original webpage is no longer relevant.
In Forcepoint’s example, the employee returns a month later and asks the assistant what to do after their flight is cancelled. The agent searches its memory, retrieves the poisoned information and recommends the attacker-controlled travel provider as though it were an established company contact.
This means there’s no need for the attacker to be present when the eventual manipulation occurs, since the malicious information has effectively been left behind inside the agent.
The same principle could clearly have more serious consequences in a business environment. For example, Forcepoint says attackers could potentially plant fake trusted domains, suppliers, support contacts, internal procedures, security rules or approval chains, as well as preferences designed to weaken security.
Why AI Agents Make This More Serious
The risk becomes more significant as AI moves from answering questions towards carrying out tasks. For example, modern AI agents can increasingly browse websites, summarise emails, search corporate information, write software, call APIs and trigger workflows. Persistent memory makes these systems considerably more useful because they do not need to rediscover the same information every time they perform a task.
However, greater autonomy also means incorrect memories could potentially influence actions rather than simply produce inaccurate answers. An agent that wrongly remembers a supplier, security procedure or approved contact could use that information while performing a future task.
The security question therefore changes from simply asking whether an AI’s current input can be manipulated to asking whether everything it has previously learned can still be trusted.
Protecting The Agent’s Memory
In terms of protection from this malicious technique, Forcepoint argues that organisations should approach AI memory in much the same way they already approach other potentially untrusted information entering corporate systems.
One proposed defence is “memory risk scoring”, where information is assessed before being committed to long-term memory. Factors could include the reputation of its source, requests to remember something permanently, previously unknown contact details, contradictions with existing information and whether the memory concerns sensitive areas such as payments, suppliers or security procedures.
High-risk information could then be rejected, quarantined or presented to a human for confirmation rather than automatically becoming something the agent trusts. Stored memories could also retain information about where they originated and be checked again when retrieved.
What Does This Mean For Your Business?
For businesses adopting AI agents, persistent memory should increasingly be treated as part of the organisation’s security architecture rather than simply a useful convenience. Giving an AI permission to remember information means deciding what it is allowed to learn, which sources it can trust and when a person should verify important changes.
Forcepoint summarises the principle quite clearly: “Just as organisations inspect files, emails and network traffic before trusting them, AI systems must also evaluate the trustworthiness of their memory items before those memories influence decision-making.”
Memory poisoning also shows how AI security risks are changing as agents become more autonomous. This means that protecting an AI system may no longer simply be a case of controlling what it can access or what instructions it receives today. From now on, it seems that businesses may increasingly need to protect what their AI remembers, because a false fact quietly stored now could influence a genuine business decision long after the original attack has been forgotten.
