AI threat detection is limited less by how many attacks a model can recognize than by how often it raises a false alarm, because real attacks are rare in the traffic being watched. A 2000 analysis in ACM Transactions on Information and System Security showed that the false alarm rate becomes the limiting factor for intrusion detection, and a 2010 paper from Berkeley researchers explained why machine learning that works well elsewhere struggles with attack detection. Both findings still describe the problem behind security products that advertise AI.
Key takeaways
- When attacks are a tiny share of events, even a detector with a small false alarm rate produces mostly false alerts (an illustrative calculation appears below).
- Stefan Axelsson's 2000 article named this the base-rate fallacy problem for intrusion detection.
- Sommer and Paxson argued in 2010 that finding attacks differs from tasks such as spam filtering, mainly because errors are costly and normal traffic varies widely.
- NIST's March 2025 taxonomy of adversarial machine learning adds that attackers can craft inputs designed to evade a model.
- These are published research findings and voluntary guidance, not legal requirements.
Why rare attacks swamp a good detector
Detection accuracy is usually quoted as two numbers: the share of attacks caught and the share of harmless events wrongly flagged. A third quantity decides whether the alerts are useful, namely how many flagged events are real attacks. That depends on the base rate, the fraction of all events that are malicious. If almost everything is harmless, then even a small error rate on harmless events produces a large pile of false alerts compared with the few real ones.
Axelsson's article, published in August 2000, worked through this with Bayes' theorem. Its abstract states that, for a reasonable set of assumptions, the false alarm rate is the limiting factor for the performance of an intrusion detection system. To reach a substantial probability that an alarm signals a real intrusion, the false alarm rate has to be very low, in some cases perhaps unattainably so. The article reviewed reported detector performance of its time and concluded that some types of intrusion detection had far to go.
An illustrative calculation
The figures below are invented to show the arithmetic and do not describe any real product. Assume a network produces 1,000,000 logged events a day, of which 10 belong to an attack, and a detector catches 90 percent of attack events (nine of 10). The table varies only the false alarm rate.
| False alarm rate on harmless events | False alerts per day | Total alerts per day | Share of alerts that are real |
|---|---|---|---|
| 1 percent | about 10,000 | about 10,009 | 0.09 percent |
| 0.1 percent | about 1,000 | about 1,009 | 0.89 percent |
| 0.01 percent | about 100 | about 109 | 8.3 percent |
| 0.001 percent | about 10 | about 19 | 47 percent |
A 1 percent false alarm rate sounds strict, yet it leaves analysts sorting roughly 10,000 alerts to find nine real ones. Only at one false alarm per 100,000 harmless events does about half of the alert queue become genuine. This is the same logic as a medical screening test for a rare condition.
What the 2010 critique added
Robin Sommer and Vern Paxson presented "Outside the Closed World" at the IEEE Symposium on Security and Privacy in May 2010. Their abstract notes that machine learning was rarely used in operational network intrusion detection despite extensive research, and that finding attacks is fundamentally different from areas where machine learning succeeds. The paper lists several reasons.
- Closed-world training: anomaly detectors learn what normal traffic looks like and flag deviations, so they are trained on the opposite of what they are meant to find.
- Cost of errors: a false positive consumes analyst time, while a missed attack can compromise an organization. The authors contrast this with product recommendations, where a poor guess rarely does harm.
- Semantic gap: a flagged anomaly means only that something has not been seen before, and an operator still needs to know what it means.
- Diversity of traffic: bandwidth, connection length, and application mix vary enormously even within one network, which makes a stable idea of normal hard to find.
The authors also observed that anomaly detection is likely better suited to finding variations of known attacks than previously unknown ones, since a system can then be trained on examples of both attacks and normal traffic. The paper dates from the era before large language models, and detection tools have changed. The structural points about base rates, costs, and context have not been overturned by anything in the sources reviewed.
Attackers can respond to the model
A detector is also a target. NIST's report "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations" (NIST AI 100-2 E2025), finalized in March 2025, organizes attacks on machine learning systems by attacker goals, capabilities, and knowledge. Its keywords include data poisoning, evasion, privacy breach, and abuse. For a security team, evasion means an adversary may shape activity to look normal to a model, and poisoning means tampering with the data a model learns from. NIST presents the document as shared vocabulary for other standards and practice guides, not as a rulebook.
Questions to ask about any AI detection claim
- What share of alerts turned out to be real incidents in live conditions, not only in a test data set?
- What base rate of attacks was assumed in the evaluation?
- Does each alert explain what happened and why it matters to this organization's policies?
- How does performance hold up on traffic that differs from the training data?
- Has the vendor or team tested evasion and data poisoning, as described in the NIST taxonomy?
This article is general information about published research and guidance. It is not legal or security consulting advice, and none of the sources cited here creates a legal duty on its own. Organizations with specific obligations should check the laws and regulators that apply to them.
FAQ
What is the base-rate fallacy in intrusion detection?
It is the mistake of judging a detector by its hit rate and false alarm rate alone while ignoring how rare attacks are. When malicious events are a very small share of traffic, most alerts can be false even from a detector that looks accurate.
Does adding AI fix the false alarm problem?
Not by itself. The arithmetic applies to any detector, whether it uses rules, statistics, or machine learning. A model can lower the false alarm rate, and the table above shows why even small improvements matter, but the claim needs evidence from realistic conditions.
Are anomaly detectors the same as attack detectors?
No. Sommer and Paxson point out that an anomaly detector reports what is unusual, whether benign or malicious. Turning an unusual event into a meaningful alert is the semantic gap they describe.
Is any of this required by law?
The sources cited here are a journal article, a conference paper, and a NIST technical report. None is a statute or regulation. Legal duties around security depend on the sector and jurisdiction.
Related reading
- The contest for cyber security advantage
- How companies use AI for security measures
- What AI does well on complex problems
- Limited memory AI and stored data
More in AI Policy and Ethics.
Last verified: October 2026
Sources
- The base-rate fallacy and the difficulty of intrusion detection, Stefan Axelsson, ACM Transactions on Information and System Security, volume 3, issue 3, 2000
- Outside the Closed World: On Using Machine Learning for Network Intrusion Detection, Robin Sommer and Vern Paxson, IEEE Symposium on Security and Privacy, 2010
- Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2 E2025, National Institute of Standards and Technology, 2025