DISC InfoSec blogMITRE ATTACK MATRIX Archives

Sep 27 2023

It’s Time to Do Better as an Industry with MITRE Evaluations

Category: Attack Matrix — disc7 @ 12:37 pm

CrowdStrike achieved the highest coverage across the last two consecutive MITRE Engenuity ATT&CK® Evaluations. We achieved 100% protection, 100% visibility and 100% analytic detection coverage in the Enterprise Round 5 evaluation — which equates to 100% prevention and stopping the breach. We also achieved the highest detection coverage in the Managed Security Services Providers testing.

However, interpreting the results of the Round 5 test can quickly become very confusing, with endless representations of test results from every provider. Unlike other third-party analysts, MITRE doesn’t place vendors on a quadrant or graph, or provide a comparative score. It leaves interpretation up to each vendor and customers themselves — meaning you’ll be flooded with claims of “winning” the evaluation.

In MITRE, there are no winners or leaders, only raw data on a vendor’s coverage against either a known or unknown adversary. Without better guidelines and enforcement from MITRE, the results will continue to confuse customers, given the wildly different solutions being tested and approaches to the evaluation.

Evaluations like MITRE can help clarify your choice. We use the evaluations to further sharpen the capabilities of the CrowdStrike Falcon® platform, as well as ensure our customers understand our point of view on cybersecurity: Stopping the breach requires complete visibility, detection and protection that you can actually use in a real-world scenario.

How Should You Interpret the Results?

First, it’s important to understand the nuances of the two types of evaluations run by MITRE: open-book and closed-book tests.

Open-book testing for known attackers: The MITRE ATT&CK Enterprise Evaluations, such as the recent Round 5, give vendors months of advance notice on the adversary being emulated and their tactics, techniques and procedures (TTPs), and then measure for coverage in a noiseless lab environment.

Figure 1. CrowdStrike detects 143 (100%) steps during the MITRE Engenuity ATT&CK Evaluation: Enterprise Round 5 with high-quality analytics (Tactic and Technique)

Not all results are equal, which is hard to see in a comparative chart like this, as vendors have the opportunity to tune their systems in advance and apply configuration changes on-the-fly with teams of experts who may be working behind the scenes 24/7 during the testing period. For instance, we’ve seen vendors make updates to operating systems for the test, while others manually fix verdicts or add new context and detections.

Round 5 emulated Turla, which CrowdStrike classifies as VENOMOUS BEAR, a sophisticated Russia-based adversary. Given their advanced tactics, few vendors were able to identify all of their tradecraft, with the average visibility being 83%. High-quality analytic detection of Tactic and Technique were even less, with the average dropping to 66% — with CrowdStrike achieving full 100% coverage with analytic detections.

High-quality analytics are extremely important, as they provide insight into what an adversary is attempting to achieve and how they are attempting to achieve it. High-quality analytic detection provides the context that analysts need, letting them spend less time trying to determine if the alert is a true or false positive, and also provides insight into what an adversary is trying to do. With tactic and technique detections, security analysts can spend more doing what matters: stopping breaches.

In a comparative chart like the one above, it isn’t possible to see if the capability provided is noisy annotated telemetry or important context added to a high-fidelity alert.

Closed-book testing for unknown attackers: MITRE’s Managed Security Services Providers test is a truer measure of how vendors will protect a customer in the real world, with no do-overs or chances to hunt for additional evidence. The only notification vendors receive in advance is a start date, with no visibility into the adversary being emulated or their TTPs. MITRE runs the test, and you get a coverage score.

Figure 2. CrowdStrike detected 99% of adversary techniques during MITRE ATT&CK Evaluations for Managed Security Services Providers.

To find the cybersecurity partner for you, it’s worth reviewing and correlating performance across many different tests that use different TTPs and force products to behave differently to find the true outcome of the platform. Ensure you look at the results of both open-book and closed-book tests, including those that measure false positives and performance, and know exactly what vendors did to achieve their results. Most importantly, make sure you can achieve those same outcomes in your enterprise. Sophisticated adversaries don’t provide the luxury of a heads-up, and customers won’t have potentially dozens of people working behind the scenes on their deployment in the real world.

Stopping Breaches Matters

Next, it’s critical to evaluate how effectively a vendor can stop adversaries without manual intervention. In the open-book Round 5 test, the average blocking rate was 86%, compared to CrowdStrike’s 100% protection. Even more important than the coverage is understanding how the scores were achieved.

When digesting the MITRE results, ask vendors these three questions, and ask them to prove it:

Did they use easily bypassed signatures or custom detections requiring prior knowledge?
Are the analytic detections and protections high-fidelity and suitable at enterprise scale?
How can I reproduce this result in my own environment?

For comparison, the CrowdStrike Falcon platform stopped 13 of the 13 scenarios with no prior knowledge, using advanced AI and behavior detection. Our AI-powered prevention will be just as effective in your environment as in MITRE’s testing, against both known and unknown adversaries in the real world.

How Do You Bring It All Together?

At the end of the day, how a platform achieved its results matters as much as coverage itself. With open-book tests like the Enterprise Evaluation Round 5, you could hire enough experts to manually add custom tags, detections and context to achieve perfect coverage. That’s why you’ll see vendors shouting their coverage from the rooftops — as at face value many did well.

All comparative charts, including the ones we’ve shown above, only tell part of the story. What’s important is looking at the details: how you do it matters as much as what you do. If you can’t actually achieve the results in your environment, it’s simply a number on a comparative chart. It can’t stop adversaries and it can’t stop breaches.

Ask your provider, including us, how they achieved their scores — and ensure it wasn’t a herculean manual effort that could never work in the real world. It’s also important to understand exactly what the full bill-of-materials looks like to reproduce the results. Some vendors require a complex point product deployment, others an expensive combination of software and network security hardware, and others a significant headcount investment to operate.

The factor to consider most carefully are vendors that use custom test configurations that are impossible to reproduce in a real-world production environment. With CrowdStrike, our platform will always be delivered via our single lightweight agent that’s easy to deploy, easy to manage and never requires a reboot. We consolidate cybersecurity, with better outcomes, at a much better ROI.

We stand behind our platform and the way we delivered our superior coverage across both MITRE’s open-book and closed-book testing for known and unknown adversaries — providing true breach prevention for the real world.

We encourage everyone in the industry to follow MITRE’s intention: Its testing yields valuable raw data that needs to be applied in your environment — with the context around how a vendor achieved its results — to be meaningful. And to our friends at MITRE, the time is now to shut down the endless noise and ensure customers understand your purpose: to make the world safer with better-informed decisions.

If you want to learn more about using MITRE for your organization, register for our virtual event: MITRE Engenuity ATT&CK Evaluations: Inside the Enterprise Round 5 Results.

Source: https://www.crowdstrike.com/blog/understanding-mitre-enterprise-round-5-evaluations/

DISC InfoSec previous posts on MITRE ATT&CK

Aligning Security Operations with the MITRE ATT&CK Framework: Level up your security operations center for better security

InfoSec tools | InfoSec services | InfoSec books | Follow our blog | DISC llc is listed on The vCISO Directory

Tags: CrowdStrike Falcon, MITRE ATTACK MATRIX, MITRE Engenuity ATT&CK Evaluations

Comments (0)

Feb 02 2021

Attempted Attack Matrix

Category: Attack Matrix — DISC @ 3:42 pm

Use ATT&CK to map defenses and understand gaps

The natural inclination of most security teams when looking at MITRE ATT&CK is to try and develop some kind of detection or prevention control for each technique in the enterprise matrix. While this isn’t a terrible idea, the nuances of ATT&CK make this approach a bit dangerous if certain caveats aren’t kept in mind. Techniques in the ATT&CK matrices can often be performed in a variety of ways. So blocking or detecting a single way to perform them doesn’t necessarily mean that there is coverage for every possible way to perform that technique. This can lead to a false sense of security thinking that because a tool blocks one form of employing a technique that the technique is properly covered for the organization. Yet attackers can still successfully employ other ways to employ that technique without any detection or prevention in place.

The way to address this is the following:

Always assume there is more than one way to perform an ATT&CK technique
Research and test known ways to perform specific techniques and measure the effectiveness of the tools and visibility in place
Carefully log the results of the tests to show where gaps exist for that technique and which ways of employing that technique can be prevented or detected
Note which tools prove to be effective at specific detections and note gaps where there is no coverage at all
Keep up with new ways to perform techniques and make sure to test them against the environment to measure coverage

For example, if antivirus detects the presence of Mimikatz, that doesn’t mean that Pass the Hash (T1075) and Pass the Ticket (T1097) are covered as there are still several other ways to perform these techniques that don’t involve the use of Mimikatz. Keep this in mind if trying to use ATT&CK to show defensive coverage in an organization.

Source: Use ATT&CK to map defenses and understand gaps