CrowdStrike has received the highest score in the last two consecutive MITRE Engenuity ATT&CK® Evaluations . The company achieved 100% protection, 100% visibility and 100% detection analysis in the Enterprise Round 5 assessment - which equates to 100% breach prevention and stopping. CrowdStrike also achieved the highest detection rate in testing for Managed Security Service Providers.

However, interpreting Round 5 test results can quickly become very confusing due to the different representations of test results from each vendor. Unlike other third-party analytics companies, MITRE does not place vendors in a quadrant or on a graph, nor does it provide a comparative score. It leaves the interpretation up to each vendor and the clients themselves, which means you'll be inundated with news of "winning" scores.

In MITRE, there are no winners or leaders, only raw data on a vendor’s coverage against either a known or unknown adversary. Without better guidelines and enforcement from MITRE, the results will continue to confuse customers, given the wildly different solutions being tested and approaches to the evaluation.

Evaluations like MITRE can help clarify your choice. We use the evaluations to further sharpen the capabilities of the CrowdStrike Falconplatform, as well as ensure our customers understand our point of view on cybersecurity: Stopping the breach requires complete visibility, detection and protection that you can actually use in a real-world scenario.

How Should You Interpret the Results?

First, it’s important to understand the nuances of the two types of evaluations run by MITRE: open-book and closed-book tests.

Open-book testing for known attackers: The MITRE ATT&CK Enterprise Evaluations, such as the recent Round 5, give vendors months of advance notice on the adversary being emulated and their tactics, techniques and procedures (TTPs), and then measure for coverage in a noiseless lab environment.

Figure 1. CrowdStrike detects 143 (100%) steps during the MITRE Engenuity ATT&CK Evaluation: Enterprise Round 5 with high-quality analytics (Tactic and Technique)

Not all results are equal, which is hard to see in a comparative chart like this, as vendors have the opportunity to tune their systems in advance and apply configuration changes on-the-fly with teams of experts who may be working behind the scenes 24/7 during the testing period. For instance, we’ve seen vendors make updates to operating systems for the test, while others manually fix verdicts or add new context and detections.

Round 5 emulated Turla, which CrowdStrike classifies as VENOMOUS BEAR, a sophisticated Russia-based adversary. Given their advanced tactics, few vendors were able to identify all of their tradecraft, with the average visibility being 83%. High-quality analytic detection of Tactic and Technique were even less, with the average dropping to 66% — with CrowdStrike achieving full 100% coverage with analytic detections.

High-quality analytics are extremely important, as they provide insight into what an adversary is attempting to achieve and how they are attempting to achieve it. High-quality analytic detection provides the context that analysts need, letting them spend less time trying to determine if the alert is a true or false positive, and also provides insight into what an adversary is trying to do. With tactic and technique detections, security analysts can spend more doing what matters: stopping breaches.

In a comparative chart like the one above, it isn’t possible to see if the capability provided is noisy annotated telemetry or important context added to a high-fidelity alert.

Closed-book testing for unknown attackers: MITRE’s Managed Security Services Providers test is a truer measure of how vendors will protect a customer in the real world, with no do-overs or chances to hunt for additional evidence. The only notification vendors receive in advance is a start date, with no visibility into the adversary being emulated or their TTPs. MITRE runs the test, and you get a coverage score.

Figure 2. CrowdStrike detected 99% of adversary techniques during MITRE ATT&CK Evaluations for Managed Security Services Providers.

To find the cybersecurity partner for you, it’s worth reviewing and correlating performance across many different tests that use different TTPs and force products to behave differently to find the true outcome of the platform. Ensure you look at the results of both open-book and closed-book tests, including those that measure false positives and performance, and know exactly what vendors did to achieve their results. Most importantly, make sure you can achieve those same outcomes in your enterprise. Sophisticated adversaries don’t provide the luxury of a heads-up, and customers won’t have potentially dozens of people working behind the scenes on their deployment in the real world.

Stopping Breaches Matters

Next, it’s critical to evaluate how effectively a vendor can stop adversaries without manual intervention. In the open-book Round 5 test, the average blocking rate was 86%, compared to CrowdStrike’s 100% protection. Even more important than the coverage is understanding how the scores were achieved.

1. Did they use easily bypassed signatures or custom detections requiring prior knowledge?
2. Are the analytic detections and protections high-fidelity and suitable at enterprise scale?
3. How can I reproduce this result in my own environment?

For comparison, the CrowdStrike Falcon platform stopped 13 out of 13 scenarios without any specialized knowledge using advanced AI and behavioral analysis. This suggests that AI-based prevention will be just as effective in your environment as it was in the MITRE test.

How Do You Bring It All Together?

Ultimately, how the platform achieved its results is just as important as the coverage itself. With open source tests like the Enterprise Evaluation Round 5, you can hire enough experts to manually add your own tagging, detection, and context to achieve perfect coverage. That's why you'll see vendors shouting about their coverage from all loudspeakers - because, on the surface, many of them have succeeded.

All comparative charts, including those above, show only part of the picture. It is important to pay attention to the details: How you do it is as important as what you do. If you can't achieve results in your environment, it's just a number on a comparison chart. It cannot stop attackers and it cannot prevent breaches.

Ask your vendor, including CrowdStrike, how they achieved their results - and make sure they didn't use titanic manual efforts that will never work in the real world. It's also important to understand exactly what the full bill of materials looks like to reproduce the results. Some vendors require complex point-to-point product deployments, others require an expensive combination of network security software and hardware, and still others require a significant investment in personnel.

Vendors that use special test configurations that cannot be replicated in a real production environment should be considered especially carefully. The CrowdStrike Falcon platform is always delivered via a single lightweight agent that is easy to deploy, easy to manage, and never requires a reboot. We strengthen cybersecurity, achieving better results with a much better ROI.

The company guarantees the quality of its platform and superior coverage of both MITRE’s open-bookand closed-book testing for known and unknown adversaries — providing true breach prevention for the real world.

iIT Distribution is an official partner of the CrowdStrikewhich is responsible for the distribution and promotion of their products in Ukraine, Kazakhstan, Uzbekistan, Georgia, Poland, Azerbaijan, Estonia, Lithuania, Latvia, Kyrgyzstan, Moldova, and Tajikistan. We also provide professional support in the design and implementation of these solutions. Our team is always ready to provide our partners and customers with all the necessary information support related to each product and solution. We are also ready to answer all your questions and advise you on all issues related to improving the efficiency of your IT infrastructure and ensuring its security.