Top 8 Metrics to Collect During a Red Team Test

Why should we collect metrics in a red team test?

Metrics are a valuable way of measuring changes and improvements over time. A red team test is conducted to assess the controls and lack of controls in place to stop an attacker achieving certain goals. By using metrics during red team testing, you can observe the performance of these controls to analyse which controls work effectively and what controls need to be implemented or improved.

The aim of a red team engagement is to highlight more than just the technical vulnerabilities, which by definition have technical solutions. A portion of what engagements show is procedural and many of the fixes aren’t black and white. Therefore the internal solutions that are identified from the testing must come in the form of metrics.

The following will cover the most important metrics to collect which includes metrics regarding discovery and mitigation of an attack to the metrics regarding control implementation. These metrics can be used to compare results with future and past tests (if available).

Discovery-based metrics

These metrics are concerned with how long an attack takes to discover, how it was discovered, and who by. It is unfeasible to expect all attacks to be discovered as they happen, however, decreasing this time using effective controls can limit the time an attacker has to do damage.

The first metric to collect is how long it takes to discover, which can be used to try and improve the effectiveness of discovering attacks. The measure of this is the difference between when (Bob) the attacker and (Jen), the new employee spots/reports the attack. By reviewing this against evidence of how the attack took place, you can analyse how effective your controls were in discovering the attack.

Next, is who/what discovered the attack. This will show the controls that work for spotting the attack used by the red team. To represent this numerically, you can use a percentage of all controls/personnel who should have been able to detect the attack. The results should be analysed to see if some controls didn’t work and why, as well as to see if there are control gaps, leading to the attack not being discovered in certain areas. So if all the senior staff spot the attack, but Jim, the new guy doesn’t, you might want to look into staff awareness training.

The final metric is strongly linked to the second. It is to note how the control discovered the attack. This can highlight why the control worked or how it can be improved to be even more efficient. The results can be tallied to show which aspects of the controls are most effective in detecting the attack that took place. In some cases, a control partially works meaning it does not provide the intended protection. By analysing how the new guard (Steve) spotted the attack but the seasoned vet (Tom) didn’t, you can get a better idea of what is working and why.

Mitigation-based metrics

This section covers metrics related to mitigation of the attack and the damage it caused. In an ideal situation, we could stop all attacks from happening. However, we all know to be 100% secure from a breach is not achievable. But metrics can be used here to see how effective controls are at stopping an attack and seeing what damage was done.

Similar to the discovery-based metrics, the first metric to collect is the time to stop the attack causing any more damage. This can help see how effective the controls are in stopping attacks in a timely manner. Calculate this by finding the difference between when the attack was first discovered and when the attack was mitigated by the security team.

Linked to the above metric, is how the attack was mitigated. This metric will help you discover ways to improve the controls in place. It is possible the control did not work as expected, possibly being more or less effective than expected. This can be measured by using a tally to see which aspects of the controls were most effective / least effective at stopping the attack.

The last metric for this section is about the cost of the attack. After the attack has been mitigated, the damage done if the attack was real, should be calculated as accurately as possible. This will help the CEO (James) to understand the risks associated with the current controls in place and how much damage they would have prevented if they stopped the attack from taking place.

Control implementation metrics

The last section is about post-test fixes and how long it takes. This is important as you are still vulnerable until effective controls are put in place to mitigate the risk of future attacks of the same type.

The first metric is how long it takes to put in the new controls. This can help identify what problems arise when controls need to be implemented and to see if they can be removed. All controls should be implemented properly as quickly as possible.

Secondly, the changes to current controls should be noted and recorded. This will be helpful to see if previous changes are working. If not, there may be a problem with the implementation of the controls. Future tests will help identify this if the same control issues reappear.

If you can’t measure it, then you can’t improve it

To conclude, metrics can be used to identify a variety of problems that arise during an attack and after. These can also be used to measure changes from each test to identify if the implementation of controls is working and if they have improved since the last test.