Keeping track of application security flaws

This article provides a granular view into how to track and visualise application security flaws. It builds upon a previous article that provided a high-level overview of how to keep track of your application security posture. We'll do a deep dive into a few metrics identified within the original article. This will help you to understand how to visualise the metrics and what to look out for when analysing / trending the data.

Before getting started its important to be aware that garbage in (flawed input) will lead to garbage out (flawed output). Ensure the flaws identified by your security controls are genuine. A combination of false positive (incorrectly identify a vulnerability) and false negative (incorrectly identify that a vulnerability does not exist) will distort your findings.

Reporting on flawed data can be particularly problematic as you may incorrectly prioritise and resource unnecessary mitigations or fail to act in situations where mitigations are required.

Finding / Flaw Creation Rate

Track the rate of newly created flaws over a set period. Flaws are often introduced due to:

• The deployment of changes;
• Newly identified vulnerabilities in utilised technologies;
• A failure to maintain technologies or adhere to security best practice standards.

What to look out for

Be aware that both upward or downward trends require further investigation as they can be considered positive or negative.

No change	If the number of identified flaws remains consistent this indicates that the security posture of your application/s is being maintained.
Upward trend	Positive This can indicate improvements in your capability to identify flaws. This may include the increased: Effectiveness in flaw identification tools / techniques Scope of systems covered in your programme Negative This can indicate declining security standards.
Downward trend	Positive This can indicate improving security standards. Negative This can indicate a reduction in your capability to identify flaws. This may include the decreased: Effectiveness in flaw identification tools / techniques Scope of systems covered in your programme

Finding / Flaw Remediation Rate

Track the rate of flaw remediation over a set period. Flaws are remediated due to:

• The deployment of changes;
• Patching of vulnerabilities in utilised technologies;
• Maintenance of technologies or alignment to security best practice standards.

What to look out for

Be aware that both upward or downward trends require further investigation as they can be considered positive or negative.

You will need to ensure that flaws marked as remediated have been fixed. Incorrectly closing flaws distorts the remediation rate as well as the overall security posture of the application.

No change	If the number of remediated flaws remains consistent this indicates that the resourcing level is being maintained.
Upward trend	Positive This can indicate an increased level of effort / resourcing or more effective enforcement of security standards. Negative This can indicate the potential gamification of the vulnerability management process. Check to ensure that flaws are not being incorrectly closed or suppressed.
Downward trend	Positive This can indicate a reduction in the total number of outstanding flaws. Negative This can indicate a decreased level of effort / resourcing or a decline in enforcement / adherence to security standards.

Flaw Growth Rate

The growth rate is derived from the flaw creation and remediation metrics. This is calculated by:

Flaw Growth Rate = Flaw Creation Rate - Flaw Remediation Rate

What to look out for

Upward and downward trends have a clear positive and negative correlation. From a security risk perspective, you want to see either no change or a downward trend to ensure that the associated level of risk is at least being maintained.

No change	A flat growth rate indicates that the security posture is being maintained at a consistent level. This may be an issue if you have a significant backlog of flaws.
Upward trend	Negative This indicates that the number of open flaws is increasing. An upwards trend can be a good indicator of increasing risk exposure.
Downward trend	Positive This indicates that the number of open flaws is decreasing. A downwards trend can be a good indicator of decreasing risk exposure.

Visualising the data

The following charts help to demonstrate how you can visual this reporting to support with the analysis of your data. The charts have been created based on the below table.

Flaw Creation & Remediation Rates

The below bar chart summarises the flaws identified and remediated over a period of six months.

If you were seeing this within your own data, you would want to determine why there is such a disparity between the rate of flaw creation and remediation. This is highlighting a concerning upwards trend.

Flaw Growth Rate

The below waterfall chart demonstrates the flaw growth rate over a period of 6 months. This clearly identifies a growth trend and shows that the total number of flaws have grown by 46 over that period.

Whilst the chart demonstrates a lagging (historic) indicator this can also be used as a leading (future) indicator in projecting trends. Given the identified average monthly growth rate of 8 flaws, you can predict that based on the current trajectory the backlog of flaws will end up doubling to 92 within the next 6 months. This provides a clear indication of increasing risk exposure.

What actions should you consider taking?

On the basis that the reported data is correct there are a couple of actions that you will want to take.

Reduce newly created flaws

Its always easier and more cost effective to address the flaws early in the software development lifecycle. You will want to consider:

• Defining and enforcing a secure coding standard;
• Integrating security tools (i.e. SAST, DAST) into the development lifecycle;
• Improving the security capability of your developers / testers
• Improving the security team engagement into the development workflow.

These actions will help to reduce the number of identified flaws within new deployments.

Remediate the flaw backlog

The flaw growth rate has led to 46 flaws in the production environment. The existing remediation priority / resourcing is insufficient to maintain the flaw backlog let alone reduce it. Investigate what can be done to increase the rate of remediation.

Correlating the flaw growth rate with the underlying risk will help you to indicate where the level of risk is outside of your company’s appetite. In doing so this may help you in getting increased resource allocation to address the flaws.