Anomaly detection

Anomaly detection identifies unusual patterns and behaviour on systems and networks.

It looks for activity that departs significantly from the established norm, and alerts on it as a possible threat.

Its value is that it does not depend on recognising a technique somebody has already described, which is exactly where rule-based detection runs out.

What characterises it

Continuous monitoring. It watches activity as it happens, using statistical models to establish what normal looks like and to identify deviations from it.

Machine learning. Many implementations use learned models so that the baseline adapts as the environment changes, which matters because an environment that never changes does not exist.

Early warning. The point is to raise something while there is still time to act, which is what reduces the exposure rather than documenting it afterwards.

What makes it hard

False positives. Legitimate variation looks like anomaly. A quarter end, a new deployment, a person returning from leave. Every one of those generates deviation that means nothing.

Adversaries adapt. Attackers deliberately keep their activity inside the shape of normal traffic, which is the entire idea behind living off the land.

Interpretation needs context. Reading a result correctly requires knowing what the business actually does. An analyst who does not know that this team migrates data every Friday will investigate the same event every week.

A worked example

An organisation deploys anomaly detection on its internal network.

The system continuously observes traffic and learns the normal communication patterns between devices and users.

One day it raises an alert because a user is accessing an unusually large number of files in a short period.

The security team reviews it and finds that the account is downloading confidential data in bulk from a server it does not normally touch at that scale.

Because the deviation was visible, the team can act, block the activity and prevent a data leak. Note what the alert was based on: not a signature, not a known technique, just the fact that this account had never behaved this way before.

Anomaly detection and rule-based detection

The glossary has two families of detection and this entry is one of them. They do not compete: they cover each other’s gaps.

Rule-based detection describes what is bad: somebody writes the logic that recognises a specific technique. That is detection engineering, and its limit is that it only finds what somebody thought to write, in an environment where somebody tested it.

Anomaly detection describes what is normal and alerts on what departs from it. Its limit is the opposite one: it finds the unusual without knowing whether it is bad, which is why it lives with false positives and needs somebody who understands the business to interpret it.

Threat hunting is neither of the two: it is the deliberate search, starting from a hypothesis, for what the rules did not raise. When it works, its output is a new rule, so it feeds the first family.

XDR is where part of this runs: correlating endpoint, identity, email and cloud so that a sequence which says nothing in isolation is recognised as an intrusion.

And why the baseline matters more than it looks: against an attacker who signs in with valid credentials, at a normal hour, from a plausible location, no rule fires, because every field in the event is correct. The only thing that distinguishes it is knowing what normal is for that account.

Detecting the anomaly is the beginning. What comes next is incident detection and the response to it.

Where to read more

Rapid7, anomalous data transfer detection: a practical description of how deviation from a baseline is used to spot data leaving.