Information Security Metrics Conundrum

Me: “My patch compliance is at 99.75% and my user education percentage is 100.”

CEO:  “Wonderful, but we just got taken for 18 million records containing customer accounts.  Those metrics sure do look nice, make sure you and your dedicated pie chart creator take those reports with you on your way out the front door!”

To determine the effectiveness of any process used by a company it is always good to record metrics for future analysis.  In the InfoSec world it is also true that using metrics can be especially helpful in showing if your policies, procedures, and controls are actually keeping your network safe.  Unfortunately in InfoSec you can research and develop great policy, train your users to not be a victim and maintain vigilance, and reduce the known vulnerabilities on your network to a near perfect level, but you cannot always prevent an attack.  New exploits are constantly surfacing, and it only takes one slip up by a user to lead to data compromise.  And we all know that successful data attacks and compromise are the real metrics that leadership will be reviewing when determining InfoSec program effectiveness, not the multitude of pie charts you have detailing training status and patch compliance. 

I don’t make the argument against keeping InfoSec metrics, they are essential for tracking progress and status of the program, but I don’t know if they can really be relied upon when used to prove overall effectiveness.  You could reduce the number of successful attacks by 90%, but what if the attacks that did get through were more severe than anything before?  I think metrics for the most part should be kept internal to the InfoSec division.  Instead of trotting out charts with percentages and numbers to leadership, I think the InfoSec team should use their collected metrics to translate the numbers into a big picture view of the company’s security posture to prove that the program is truly effective and is constantly getting better.

 I guess the key here is to make sure you research and develop metrics that will actually contribute to a more secure network, while identifying and removing metrics that may generate interesting stats while not necessarily enhancing the security posture.  So you are at 100% for security training, but is the program adapting to the environment and still effective?  You have 99% patch compliance, but is our web server and database secure against SQL injection?  The metrics must be constantly analyzed for effectiveness and trends so effort is not wasted, because taking and evaluating metrics can be rather time intensive and focusing energy in the wrong spot can be fatal.