Loading...
New
See what manual alert triage really costs your security team.
New
See what manual alert triage really costs your security team.
See what manual alert triage really costs your security team.
See what manual alert triage really costs your security team.
Loading...
Data poisoning is an attack that corrupts the training data, retrieval store, or memory an AI system learns from, so it makes wrong decisions later. Yes, a self-improving AI SOC can be poisoned through fine-tuning on analyst verdicts, stored context, or the person approving its fixes. In PoisonedRAG (2025), five planted texts per question hit 90% attack success.
Brutal truth about a SOC that learns: if you can teach it, so can the attacker.
One of the most valuable properties of any AI Agent in security is self-improvement. We've said that loudly. We still believe it. From an attacker's chair, it's also the best thing about us, because a system that learns takes instruction from whoever can reach its memory. And the poison doesn't have to be false.
Data poisoning is an attack on what an AI system learns from. The attacker corrupts its training data, retrieval store, or memory, and the system carries the bad lesson into later decisions, often with no visible change in how it behaves the rest of the time.
AI data poisoning can hit the training set, the documents a model retrieves at runtime (RAG poisoning), or what an AI Agent keeps between tasks (memory poisoning). The last two should worry a SOC most.
Targeted poisoning makes one attacker's tooling read as safe, backdoor poisoning sits quiet until a hidden trigger shows up, and untargeted or availability poisoning drags quality down for everybody. Everyone expects label flipping, a stream of wrong verdicts, since analysts produce labels all day.
The nastier family needs no wrong labels at all.
Machine learning calls it "clean-label" poisoning, a term from the 2018 Poison Frogs paper on image classifiers, whose attacks "don't require the attacker to have any control over the labeling of training data." A SOC isn't an image classifier, so that's an analogy. It's a good one.
A hypothetical example of data poisoning in a SOC is an attacker who copies a legitimate job for weeks until analysts teach the AI SOC Agent that the activity is expected. No label in that story has to be wrong.
Say someone steals the credentials for svc-backup, a service account whose nightly job runs encoded PowerShell on BUILD hosts and trips the odd alert. For five weeks, they copy that job from the same account, at the same hour, with harmless commands. Your SOC's context never recorded that the real job launches from CI, so a non-CI parent process doesn't stand out to anyone.
Your AI SOC Agent looks at the commands, sees nothing harmful, and closes the alert as benign. Five weeks of that, and somebody on the team is sick of the noise, so they teach it that svc-backup running encoded PowerShell is expected. A reviewer looks at all those clean closes and says yes.
In week six, the attacker uses the same pattern to pull down Mimikatz, and that alert reads as expected too.
Every verdict on those copies was right about the command and wrong about who ran it.
Alankrit Chona, our CTO, described this for behavior baselines: "if an attacker establishes persistence during the learning window, you're not detecting anomalies - you're learning to ignore the intrusion." And five weeks is a short wait. CISA's 2024 advisory has Volt Typhoon holding footholds in some victim IT environments "for at least five years."
The best data poisoning target in a learning SOC is a stored lesson that looks like good engineering, because nobody argues with good engineering. We know because our own AI SOC Agent wrote one for itself this year, and a person approved it.
At one large enterprise customer, the EDR kept classifying a signed Admin By Request component as process hollowing. Every alert sent the Agent off hunting across the whole host. Every time, it came back legitimate. So it proposed a skill: validate the signed component, confirm the managed-endpoint workflow, check for malicious follow-on activity, and stop.
Investigations on that customer's matching alerts now stay on the process chain instead of roaming the whole host, and verdict quality held. Input tokens on those alerts fell 42%. The Agent's own citations show the skill running every time.
Read as an attacker, that skill is a learned reason to look at less of the host.
Every alert in that comparison was legitimate, so the results can't show how the skill handles an attack, and a DLL side-loaded into the signed component could pass the first check. The follow-on check carries the weight. Every self-improving SOC makes that trade, ours included. Ask to see it written down.
Our Agent won't propose a fix without recurring evidence across investigations, or one severe and well-supported failure, plus high confidence. A patient attacker can manufacture recurrence, which is why nothing applies until a person approves.
Real data poisoning examples start with security systems that learn from feedback or shared data, including Gmail and VirusTotal, and a 2009 study showed a slow version against network detection. The trick is older than LLMs, and it rewards the patience a SOC doesn't have.
Google's Elie Bursztein wrote in 2018 that advanced spammer groups tried to throw Gmail's filter "off-track by reporting massive amounts of spam emails as not spam." Between late November 2017 and early 2018, he counted "at least four malicious large-scale attempts to skew our classifier." Swap "not spam" for "benign," and that's an attack on your analysts' close button. His warning from that post belongs on the wall of anyone building a learning SOC: "Attackers actively attempt to shift the learned boundary between abusive and legitimate use in their favor."
VirusTotal got the shared-data version. Someone uploaded machine-mutated, mostly non-working copies of one ransomware family, and several vendors' engines labeled them all as that family (MITRE ATLAS case AML.CS0002, 2020). If your tools trust shared verdicts, you inherit whatever got labeled there.
The svc-backup story has a 2009 ancestor, the "Boiling Frog" attack from the ANTIDOTE study. Researchers raised junk traffic a little each week (on simulated traffic) while a network anomaly detector retrained on it. At 15% weekly growth, the miss rate climbed from 3.67% to over 70% in three weeks. Each retraining absorbed a little more of the attack. The authors named why detection gets harder on real traffic: "the learner's inability to distinguish between benign data drift, and adversarial poisoning." Seventeen years on, that still describes a SOC that learns.
Data poisoning enters a self-improving AI SOC in three places: a model fine-tuned on your verdicts, the stored context the AI SOC Agent reads at runtime, and the person who approves what it learns. Each place needs a different control.
If a vendor fine-tunes a model on your alerts and verdicts, those five weeks of benign closes go into the weights, and good luck taking them back. Anthropic's 2024 "Sleeper Agents" research found planted backdoors could be made to survive standard safety training, and adversarial training could teach a model to recognize its trigger better, hiding the unsafe behavior. So "do you train on our data?" comes first on any vendor call.
RAG poisoning plants misleading content in the store an AI system retrieves from at runtime, and the model then treats it as trusted context. AI Agent memory poisoning does the same thing to whatever an Agent carries between tasks. In a SOC, that's where you keep what each service account normally does, so one line saying svc-backup's PowerShell is expected changes how every matching alert reads until somebody audits it.
Microsoft's Zero Trust guidance has the right instinct in six words: "Treat memory writes as privileged operations." MITRE ATLAS tracks the attack as AI Agent Context Poisoning (AML.T0080).
You don't need an attacker for this. We've seen the accidental version in the field: a team adds a red-team exercise as expected activity and sets no end date. The exercise ends, the context stays, and investigations keep discounting real alerts on that asset.
Governed learning ends at a person who says yes, and that person can fail you. In the hypothetical, they read the evidence and it looked real. With a stolen analyst login, nobody needs convincing. Same goes for a reviewer who quit reading months ago. We found no study measuring either path in a SOC, so call this a threat model. If you can approve context, you can poison it, and so can anyone holding your credentials.
Data poisoning attacks can work with almost no data: 250 training documents in a 2025 Anthropic backdoor study, and under 0.1% of an AI Agent's memory in AgentPoison. Those are lab numbers from controlled tests, and anyone quoting them as SOC results is stretching.
| Target | Study | Dose | Result |
|---|---|---|---|
| Training data | Anthropic, UK AISI, Alan Turing Institute, Oct 2025 | 250 documents | Backdoored models from 600M to 13B parameters, regardless of model size |
| Retrieval store | PoisonedRAG, USENIX Security 2025 | 5 texts per target question | 90% attack success in a knowledge base of millions |
| Agent memory | AgentPoison, NeurIPS 2024 | Under 0.1% of memory or knowledge base, plus an optimized trigger in the query | Over 80% average attack success on three test Agents, under 1% effect on normal behavior |
| Agent memory | MINJA, NeurIPS 2025 | Queries only, no write access | An ordinary user planted malicious memory records by querying the Agent and reading its replies |
| Security logs | LogInject, USENIX Security 2026 | Payloads in log fields, 2,569 adversarial entries tested | 83.4% average attack success across three production LLMs; layered defenses cut attacks 90.4%, with 8.4% residual vulnerability |
Anthropic's study used a narrow backdoor (gibberish output) that the authors call unlikely to pose significant risk in frontier models, and none of these studies touched a production SOC. LogInject comes closest.
The table settles one thing, though. Doses this small won't show up on your dashboards. You catch poison at the write, or you don't catch it.
Prompt injection becomes data poisoning when an injected instruction changes what an AI system stores. Injection steers one decision. Poisoning changes what the system believes from then on. The two meet in your logs, because attackers control fields your AI reads, such as User-Agent strings, usernames, script blocks, and file names.
Sygnia showed the first half of this in 2025 inside its own MDR service. It hid a prompt in a PowerShell script block, and the AI summary called a Mimikatz download a "Scheduled WMI maintenance task." A human analyst there flagged the mismatch right away.
Take that analyst away.
Had the summary closed the ticket and the verdict been saved, a one-off trick would have become a standing blind spot, with no five-week wait. We're inferring that last step, and nobody has shown it in a SOC yet, as far as we can find. Our guide to AI Agent guardrails for the SOC covers how log-borne injection works and how to test for it.
Simbian limits data poisoning with three controls: no training on customer data, a tracked diff with provenance for every context update, and human approval before it changes itself. None of them beats a patient attacker on its own. Run the hypothetical, and you can see where each one runs out.
The easy answer is to switch learning off. We think that's worse. Attackers pick up new techniques, your network changes, your log schema changes, identity drifts, and an Agent stuck with the knowledge it shipped with falls behind all of it without anybody poisoning a thing.
So keep the learning, and make every lesson answer to someone.
Simbian learns in context, reading your knowledge at runtime and improving the skills and context around the model, so those five weeks of verdicts never reach a model's weights. TrustedLLM™, the architecture around the model, is private and hardened against prompt injection and data poisoning in the war lab.
Defensive training data does not exist in nature, so we manufacture it. We build a synthetic company with its own people, machines and daily traffic, then have one AI attack it while another defends. Because we authored both sides we know the attacker's goal, every path it tried, and which defensive action stopped it. The product ships into 300+ enterprise environments, and wherever it falls short of what your team would have done, that shortfall tells us where the lab was thin. We fix the lab and ship again, the way a vendor fixes a bug from a bug report. Your telemetry never enters the lab.
Every stronger frontier model upgrades our attacker for free. What learns there is the skills and system around the model, never its weights. We never train on your data. A bug report is not your data, and neither is this.
Follow the svc-backup lesson into the Context Lake™. It shows up as a tracked request with a before-and-after diff, like anything else an analyst or Simbian's AI SOC Agent submits, and no investigation can use it until it's applied. If the submitter lacks write access, it waits for a reviewer. If they have it, the lesson skips that queue. Hand out that permission the way you'd hand out domain admin.
A conflicted request is never applied. A person has to decide which version wins, even when the Agent recommends one. The svc-backup lesson contradicts nothing, so that check stays silent and the lesson's wording decides the case.
| Lesson as taught | The attacker's command in week six |
|---|---|
| "Ignore PowerShell alerts." | Discounted, along with every other PowerShell alert |
| "svc-backup running encoded PowerShell is expected." | Discounted as expected |
| "svc-backup on BUILD hosts runs encoded PowerShell nightly at 02:00 UTC. Expected CI behavior." | Right hour, but CI didn't launch it, so it's investigated in full |
An attacker in CI launches from inside that exception. Put an expiry on anything temporary. When context does shape a verdict, the investigation links back to the request behind it, so after a breach you at least know who taught the lesson and when.
Our Agent's own fixes join the same queue and always wait for a person. Each of its requests names the share of investigations it will affect and links the evidence, so an approver can see when that evidence is one account's noise. Allowlists and rules that automate a verdict are outside what it can propose. The shortcut in our own story is neither, and it still narrows what gets looked at, so that limit won't save you. Self-improving, not self-driving.
The approver stays the weak point, in Simbian or anywhere else. Keep requests small, one fact each, because a small request is easier to review. If your tool lets you, add a second approver for anything that changes how a high-impact alert type gets handled, and keep an eye out for approvals nobody's reading anymore. A rubber stamp waves a poisoned lesson straight through, and our diffs will only tell you who did it.
To test an AI SOC vendor for data poisoning, plant a lesson in a proof of concept and watch what the product does with it. A sound product holds a low-privilege lesson for review, keeps it inside its scope, and traces it back to its author.
Ask two questions first. Do you train or fine-tune any model on our alerts, verdicts, or telemetry? Can your Agent ever propose an allowlist or apply a change to itself without a human? Then run five steps on a test account.
Plant a broad lesson from a low-privilege analyst account, saying the test account's encoded PowerShell is expected, and see whether it goes live or waits for review, then repeat from an account with write access and through a crafted log field.
Submit a contradiction against existing context and check that nothing overwrites silently.
Approve a scoped lesson, then fire around its edges: the same command at another hour or from another parent process, and a malicious payload at the same hour under the same parent.
Trace the closed alert back to who taught the lesson and when.
Retire the lesson, re-fire the activity, and ask which past verdicts it touched.
Step five matters because, as Alankrit puts it, closing an alert is "a provisional judgment contingent on a future that hasn't happened yet." A vendor that can't get through steps one and four is asking you to trust every person and every log line that ever touches its memory.
The poisoned lesson that gets through will be new, plausible, one line long, and approved late on a Friday by someone clearing a queue. Ask every vendor what stops that one, including us. Lots of work to do on defense.
Come talk to us if you'd like to see a planted "this is normal" request sitting in review with its diff and author attached. Book a demo.
Q: What is an example of data poisoning? A documented example of data poisoning is the series of spam attacks on Gmail's filter. Between late November 2017 and early 2018, Google counted at least four large-scale attempts to skew its classifier by reporting spam as not spam. In a SOC, the same move means teaching an AI Agent that attacker activity is benign.
Q: What is data poisoning in an LLM? LLM data poisoning corrupts what a large language model is trained on, retrieves, or remembers, so it learns a false pattern or a hidden backdoor. It can happen in training, in a RAG store, or in an AI Agent's memory. OWASP lists the risk as LLM04:2025, Data and Model Poisoning.
Q: What are the main types of data poisoning attacks? The common types of data poisoning attacks are label flipping, clean-label poisoning, backdoor poisoning, targeted poisoning, and availability poisoning. Some aim at one input, some plant a hidden trigger, and some drag down the whole model. NIST's AI 100-2 report classifies them by attacker objective and capability.
Q: What is the difference between data poisoning and model poisoning? Data poisoning corrupts the data a model learns from or retrieves, while model poisoning tampers with the model itself, for example by shipping a backdoored checkpoint through the supply chain. OWASP groups both under LLM04:2025, so check the model's source as well as its data.
Q: How does data corruption or poisoning defeat an AI algorithm? Data poisoning defeats an AI algorithm by changing what it treats as normal. Feed it corrupted or crafted data, and the patterns it learns or retrieves shift, so the inputs an attacker picked pass as safe while everything else looks the same. In AgentPoison, the effect on normal behavior stayed under 1%, so routine testing may miss it.
Q: How do you detect data poisoning? Detect data poisoning with provenance and replay testing, because a targeted or backdoored system acts normally until the attacker's input arrives and can pass for data drift. Provenance records who made each learned change and when. A golden set of curated attacks and normal content, checked before each change ships, adapts a defense Google's Elie Bursztein recommended in 2018.
Q: Can you remove data poisoning from a model? Data poisoning is hard to remove from a trained model. For a detected backdoor, NIST lists pruning, fine-tuning, and retraining, and retraining assumes you have clean data. Anthropic's Sleeper Agents work showed backdoors can survive standard safety training. A poisoned memory or retrieval store is easier. Find the bad entry, remove it, and review what it touched.
Q: Can an AI SOC be poisoned or hijacked? An AI SOC can be both poisoned and hijacked. Poisoning plants false "normal" facts that it keeps and applies to later alerts, and MITRE ATLAS catalogs that as AI Agent Context Poisoning (AML.T0080). Hijacking uses prompt injection hidden in logs, tickets, or MCP tool descriptions to redirect one decision at a time.
Q: How do you protect an AI SOC from data poisoning? To protect an AI SOC from data poisoning, control what it's allowed to learn. Refuse vendor training on your verdicts, route every write to stored context through a tracked diff with a named author, scope and expire facts, and keep allowlists and self-changes behind human approval. OWASP's Top 10 for Agentic Applications lists this as ASI06, Memory and Context Poisoning.
Q: Does Simbian train models on customer data? Simbian does not train or fine-tune models on customer data. It learns in context, reading your knowledge at runtime. Changes to what it knows about your environment go through tracked, reviewable requests, and the Agent's own proposals never apply without human approval.
