Tolerable security metrics

Like most pursuits in life, computer security does not have good measures of success. This annoys everyone involved.

Dino Dai Zovi said it well:

Good security is hard and the feedback loop on decisions is long and the signal is low fidelity. It's not clear how many incidents were prevented or mitigated from which foundational decisions years prior.

To me, the ideal overall metric would be something like:

% probability of the company or its users being hacked

Or maybe:

% probability of the company’s top five most damaging risk scenarios happening

Unfortunately, that is not achievable in any honest or useful way.

So here is a brainstorm of security metrics that have at least some value:

• % of bugs in first-party code found by your own organization vs. outsiders, such as bug bounty researchers
• Time to fix a bug, a security bug, and a high-priority security bug
• % of employees who clicked on an easy, medium, or hard phishing attempt
• How smoothly or poorly the organization responded to the last self-inflicted red-team operation
• Attacker interest. I don’t know how to make this real, and it is definitely not “number of threat intel feeds we pay for.” It is some blend of knowing the attacker landscape, their TTPs, the expected shape of attacks, and whether attacks shaped like that are actually happening.
• Log coverage from incident-relevant systems
• Cost of compliance, plus the times you have faced negative impact from being out of compliance, such as a GDPR fine
• % of total employees who have access to the most crucial business data, such as revenue, blueprints, the full user list, or whatever matters most for your company
• Total internet-accessible surface area, measured by lines of code, number of systems, or number of servers
• % of internet-accessible assets with strong baseline controls, such as logging, backups, encryption, and hardened configs
• Time since last breach. No joke, this is useful to track even if a lot is outside your control.
• % of key assets with an owner. A key asset might be an important database, a product made of code, or a business process like offboarding.

A good metric is a great thing. It lets you know if you are moving in the right direction.

Ultimately, metrics are consumed by people: our coworkers. So beyond being clear, they should ideally be motivating. The best case is that a metric gives you the freedom to throw some spaghetti projects at the wall and maybe move it with a novel project.