News

Why Anomaly Detection Breaks In The SOC, And What Data Scientists Can Fix

Posted tyrant

Build an anomaly detection model on a public intrusion dataset and you can reach a 0.98 AUC in an afternoon. Put that model in front of a security operations centre (SOC) and the analysts will probably mute it within a month.

That gap is old news in security research. Robin Sommer and Vern Paxson wrote about it in 2010, in a paper titled Outside the Closed World, and noted that machine learning was rarely used in real intrusion detection despite years of academic work. Sixteen years later, with AI agents now triaging alerts in some SOCs, the same failure modes still decide whether a model helps or hurts.

This piece is for data scientists who get pulled into security work. Most of the problems are statistical, so they sit squarely in your field.

The Base Rate Problem Eats Your Precision

Security data is lopsided in a way most business datasets are not. A mid-sized company can log tens of millions of events a day, and almost none of them are attacks.

Run the numbers on a model that sounds excellent. Say it scores 10 million events a day with 99.9% specificity and catches every one of 10 real attack events. It also flags 10,000 benign events. Precision lands at about 0.1%, so an analyst opens roughly a thousand alerts to find one real attack.

Daniel Arp and colleagues call this the base rate fallacy in their USENIX Security 2022 paper on machine learning pitfalls. They reviewed 30 security papers and found every single one affected by at least three common pitfalls. The base rate fallacy was among the ones the original authors themselves agreed shows up often.

The field data looks the same. The SANS 2025 Detection and Response Survey found that 73% of teams name false positives as their top detection challenge, up from 64% the year before. Microsoft and Omdia’s State of the SOC 2026 research estimates that 46% of alerts are false positives and 42% are never investigated at all.

Those two figures do not measure the same thing, and neither tells you how many real attacks slipped through. They do tell you where the pain sits. A model that adds alerts without adding context makes the queue worse, however good its ROC curve looks.

Your Labels Are Thin, Late And Often Wrong

Supervised learning needs ground truth. In a SOC, ground truth arrives slowly and in poor shape.

An analyst closing an alert as “benign” might mean it was checked properly. It might also mean the queue was long and the alert looked like yesterday’s. Researchers at Oxford named their USENIX 2022 study of SOC analysts after a line from one participant: “We know 99% of the alarms we generate are false positives.” It was a small study of 21 practitioners across seven SOCs, so treat the number as a feeling, not a measurement. Training on closure labels from people who feel that way teaches the model to copy tired humans.

Confirmed attacks are rarer still, and they often surface weeks after the activity that would have revealed them. By then the logs around the event may have aged out of retention.

Two habits help. Treat analyst verdicts as noisy labels and model them that way, with confidence weights or label cleaning. And keep a small, carefully reviewed evaluation set that nobody trains on, refreshed from incidents your team has actually confirmed.

Normal Keeps Moving

Unsupervised methods learn what normal looks like and flag deviations. The trouble is that normal changes constantly.

A company rolls out a new SaaS tool and login patterns shift overnight. Month-end finance work looks like data exfiltration if you squint. A merger doubles the number of user accounts in a week. Sommer and Paxson flagged this diversity of benign traffic as one of the core reasons anomaly detection struggles in practice.

Log data adds its own mess. Fields change names when a vendor updates an agent, timestamps arrive in mixed time zones, and one firewall can send a thousand near-identical events for a single blocked scan. Before tuning any model, profile the inputs. Many “anomalies” turn out to be parsing bugs.

Retrain on a schedule you can defend, track feature distributions over time, and alert on drift in the model itself. A detector that quietly goes stale is worse than one that fails loudly.

What Actually Works In Production

The models that survive in a SOC tend to share a design choice: they rank and explain, and they leave the final call to the analyst.

Start by scoring entities instead of firing alerts. Attach a risk score to users, hosts and service accounts, and let it build up across related events. One odd login is noise. An odd login followed by a new mailbox rule and a large download is a story worth an analyst’s time.

Then bring the context with the alert. Analysts lose most of their time gathering background: who owns this machine, whether this user has done this before, whether the destination appears on a threat feed. A model output that answers those questions saves more time than a small gain in recall.

Finally, close the loop. Feed analyst decisions back into the model, but weight them by how carefully they were made. A verdict from a full investigation is worth far more than a one-click close at 3 a.m.

This is also where AI agents have entered the picture. In an AI SOC, agents pick up an alert, pull the surrounding logs and identity data, check it against past cases and hand the analyst a summary with a recommended verdict. For data scientists the shift matters, because the anomaly model stops being the last step. Its job becomes producing signals an agent can reason over, with clear features and calibrated scores.

Measure It The Way The SOC Feels It

AUC and F1 on a balanced test set will flatter almost any detector. Arp and colleagues list inappropriate performance measures alongside the base rate fallacy for a reason.

Report precision at the alert volume your SOC can actually handle. If the team can investigate 200 alerts a day, the useful question is how many real incidents sit in your top 200. Report the false positive count in absolute numbers per day, since a rate of 0.1% means little until someone multiplies it by the event volume.

Track time-based measures too. How long after the first malicious event does the model raise its score? How often does a high score lead to a confirmed incident within a week? Those are the numbers a security lead will ask for, and they make for a far better conversation than a confusion matrix.

Where This Leaves Data Scientists

Security teams do not need another model that finds unusual activity. They already have more unusual activity than they can read. What they need is help deciding what to read first, with enough context to decide quickly.

That makes it a ranking problem, a labelling problem and an evaluation problem. Data scientists who treat it that way, and who sit with an analyst for a full shift before writing any code, tend to build tools that stay switched on.

Sources

Share this