How much security coverage does your SIEM really have?

How much security coverage does your SIEM really have? Quantitative optimization of Detection Coverage

SIEMs and other security tools are crucial for defending today's digital infrastructure. Ever wonder how much of the bases are covered? Based on my review (see references) I found lots of helpful information on "detections as code", using threat intelligence to understand attacks and lots on validating detection rules. But no list that identified which detections were the most important — in order of importance and absolutely no indication of how much relative or absolute coverage was provided. One of the dangers of an engineering education.

There are also an enormous number of detections available. One of the websites with which I am familiar (detectionexplorer.io) is at 11,977 detections. (The count will be higher now.) This is substantially more than most SIEMs can handle. Judicious choices will have to be made.

If you read one of the reports on major breaches — think Verizon DBIR report, the risk of a breach (read SIEM failure) is altogether too real. So if you're running a Security Operations Center things look pretty scary — a common occurrence will likely lead to some fairly intense questions about the choice of detection rules in addition to validation type issues.

There is, in fact, some information on this — which is the CISA RVA Vulnerability data. (https://www.cisa.gov/news-events/alerts/2024/09/13/cisa-releases-analysis-fy23-risk-and-vulnerability-assessments). The fine print says:

"The RVA is intended to assess the entity's network capabilities and network defenses against known threats. In Fiscal Year 2023 (FY2023), CISA and the USCG conducted a total of 143 RVAs across multiple CI sectors. Each RVA maps the results to MITRE ATT&CK framework, which includes 14 tactics, techniques and procedures (TTPs) that cyber threat actors use to obtain and maintain unauthorized access to a network or system …"

Restated, a pen testing program carried out by the U.S. Government based on their research and using their pen testing teams. This is actually pretty handy — the government has enormous resources and, by virtue of being a a pen test, both perspectives (attacking and defending) are known.

Things also get better in that there is data for 2019, 2020, 2021, 2022 and 2023. In total there is coverage of ~451 breaches over 5 years.

There is one hitch, the data is tricky to use. There are 11 pie plots with tabular data for 11 different Mitre Att&ck Tactics that each sum to 100 percent and there is no easy way to determine what the actual count on a global basis is for most years — which would allow determining the overall percentages. Fortunately, there is data from 2019 that indicates the prevalence of the different Mitre Tactics:

The same Mitre technique can show up in multiple Mitre Tactics. This means either spreadsheets must be made and/or a little magic done with some python scripting to analyze the data. Furthermore, the count in the different years is not the same.

A natural question to ask is whether the ordering of detections changes from one year to the next? The answer to this is yes. However, the changes are not major and one way to handle this is to weight the more recent years more heavily and weight the older data less. The weighting for data comes from the only data available — which is the 2019 data shown above.

The results across the years are very similar. The year weighted data over 2020 to 2023 resolves as shown below (2020 weighted at 4.285%, 2021 at 8.512%, 2022 at 17.203% and 2023 weighted at 70.000%):

The listing results are shown partially below:

Note that there is an "Other" category (item 6 on the list).

What's important is that the top 99 Mitre techniques of 577 cover 97.1 % of Mitre techniques used in the CISA RVA data set. It also means (577 — 99) or 478 Mitre techniques are much less likely to produce preventative results.

Note that Mitre techniques do not correspond directly to techniques — most techniques will require more than one detection. A good starting place for detections would be the publicly available libraries from SigmaHQ, Splunk and Elastic Search. Note there are other libraries. The list generated above has 18 Mitre techniques for which there are no detections in the libraries listed.

Realistic Limitations

The SIEM detections available from the listed public libraries do not cover everything and complete coverage is impossible. Experience has shown that other steps should be taken such as EDR (IPS and SASE too). User endpoint behaviour is also important — EUBA cannot be ignored. Some of the comments I have received point out that high level threat actors research the library rules with the intent to bypass them. This is absolutely true. So there is still more work to both validate rules and keep them up to date. That said covering 99 Mitre techniques in detail is way easier than keeping up to date on 577 Mitre techniques. It has been pointed out that the CISA RVA data is somewhat dated — related to the time it takes the CISA to summarize the data. This is also true. The need for threat intel has not be removed — prudence requires this still be monitored and new APT techniques will need the appropriate additional detections in addition to the above. Threat intel has been pivotal in dissecting how attacks are made and is critical to the development of detection rules. That said, in my experience there is little data on relative occurrence counts. The CISA RVA data is an exception. If, in the future, broader breach reporting becomes mandated, this might allow more quantitative analysis of frequency. The threat actors will respond ….

If there is sufficient interest can outline the nuts and bolts of how the data can be summarized and interesting intermediate results

References

Threats and Risk

1. Shostak, Adam: Threat Modelling, designing for security; Wiley 2014.
2. Shostak, Adam: THREATS, What Every Engineer Should Learn From Star Wars, Wiley 2024
3. Leirvik, Ryan: Understand, Manage and Measure Cyber Risk, Practical Solutions for Creating a Sustainable Cyber Program, Second Edition; Apress 2023.
4. Coburn, Andrew; Leverett, Eireann and Woo, Gordon: Solving Cyber Risk, Protecting Your Company and Society; Wiley 2019.
5. Hubbard, Douglas W. and Seiersen, Richard: How to Measure Anything in Cybersecurity Risk (First Edition); Wiley, 2016.
6. Hubbard, Douglas W. and Seiersen, Richard: How to Measure Anything in Cybersecurity Risk, Second Edition; Wiley, 2023.
7. Seiersen, Richard: The Metrics Manifesto: Confronting Security with Data; Wiley, 2022.

Active Defender and Testing

1. Ullman, Catherine J.: The Active Defender, Immersion in the Offensive Security Mindset; Wiley 2023.
2. Gregg, Michael: The Network Security Test Lab, A Step-by-Step Guide, Wiley, 2015.

SOC Setup and Operation

1. Muniz, Joseph, McIntyre and Al Fardan, Nadhem: Security Operations Center, Building, Operating and Maintaining Your SOC; Cisco Press 2016.
2. Kan, Nouman, Ahmed: Next Gen Security Operations Center: From Concept to Reality, The Comprehensive Guide to Building, Managing and Transforming your Organization's SOC, First Edition; IPSpecialist, 2023.
3. Muniz, Joseph: The Modern Security Operations Center, The people, process and technology for operating SOC services; Addison Wesley 2021.
4. Murphy, James: Security Information and Event Management (SIEM) Implementation; Self Published / Amazon ~ 2020.
5. Nathans, David: Designing and Building A Security Operations Center; Syngress, 2015.
6. Thomas, Arun E.: Security Operations Center, SIEM Use Cases and Cyber Threat Intelligence, Self-published, ~ 2020

Detection Engineering

1. Hettama, Hinne: Agile Security Operations, Engineering for agility in cyber defense, detection and response; Packt 2022.
2. Roddie, Megan; Deyalsingh, Jason and Katz, Gary J.: Practical Threat Detection Engineering, A hands-on guide to planning, developing and validating detection capabilities.
3. Peiris, Chris; Pillai, Binil and Kudrati, Abbas: Threat Hunting in the Cloud, Defending AWS, Azure and Other Cloud Platforms Against Cyberattacks; Wiley 2022.
4. Mohanta, Abhijit and Saldanha, Anoop: Malware Analysis and Detection Engineering; Apress, 2020.
5. Palacin, Valentina: Practical Threat Intelligence and Data-Drive Threat Hunting, A hands-on guide to threat hunting with the ATT&CK Framework and open source tools; Packt 2021.
6. Pease, Andrew: Threat Hunting with Elastic Stack, Solve complex security challenges with integrated prevention, detection and response; Packt 2021.

Blogs

1. VanVleet: The Threat Detection Balancing Act: Coverage vs. Cost, Medium 2024–01–22.
2. Van Os, Rob et al.: MaGMa — a framework and tool for use case management, online https://www.betaalvereniging.nl/wp-content/uploads/FI-ISAC-use-case-framework-verkorte-versie.pdf.
3. Van Os, Rob et al.: ThHiTI — integrates threat hunting intelligence in a process for investigations; https://www.betaalvereniging.nl/wp-content/uploads/TaHiTI-Threat-Hunting-Methodology-whitepaper.pdf
4. French, David: From soup to nuts: Building a Detection-as-Code pipeline, Medium, 2023–07–27.
5. Pescatore, John: Detection Engineering Best Practice for Implementing a Threat-Informed Defense, SANS Institute, 2023.
6. Bastides, Leo: On the Road to Detection Engineering, trustedsec.com, 2023–04–11.
7. Palantir: Alerting and Detection Strategy Framwork, Medium, 2017–12–19.
8. Prager, Joshua: Prioritization of the Detection Engineering Backlog, 2022–10–05.
9. Atkinson, Jared: Introducing the Funnel of Fidelity, Medium, 2019–11–20.
10. Bailey, Kyle: Detection Engineering Maturity Matrix, https[:]detectionengineer[.]io,2021–04–25.
11. Atkinson, Jared: Detection Spectrum, Medium, 2020–02–21
12. McGeehan, Ryan: Lessons Learned in Detection Engineering, What I've Learned from "good" intrusion detection programs, 2017–03–27.
13. Roth, Florian: About Detection Engineering, Medium, 2022–09–11.
14. Cross, David B.: How to Improve Security Monitoring With Detection Engineering Program, https[:]blogs[.]oracle[.]com, 2021–05–05.
15. Sohan, G.: Establishing a Detection Engineering Program from the ground-up; Medium, 2023–02–07.