When security incidents from software defects happen, retrospectives often tell the story of heroic remediation in the form of a few hundred lines of code (or less) but maximum organizational disruption: all-hands-on-deck root cause investigations and executives working 24/7 to control fallout.
This reality has been the main drive behind the adoption of “Shift-Left” security—a proactive approach that integrates security testing like static application security testing (SAST) and software composition analysis (SCA) early in the development process.
Common pre-production security controls in a software development lifecycle (SDLC).
Catching and fixing security vulnerabilities before they reach build, test, release, and deployment—before they even exist seems easy to justify, right?
Turns out it’s more complicated than that:
- Return on Investment (ROI) arguments rely on speculation and fear: cost savings by avoiding unexpected losses.
- Business priorities typically focus on revenue-generating activities. The prospect of adding any extra step to pre-merge workflows when a company is already behind on deadlines is a tough pill to swallow.
- It’s hard to prove #1, very easy to prove #2.
Here, security leaders find themselves in an all-too-familiar paradox: if an ounce of prevention is worth a pound of cure, how is it so difficult to prove?
Rethinking return on security investment
Is a “return” on loss prevention the right justification for security investment? How do we quantify how well an organization outsmarts cybercriminals? How many organizations budget for expected data breaches as part of fiscal year planning?
Cybercriminal activity isn’t a predictable variable in business operations, but risk mitigation is. A more accurate framework for investment decision-making is return on mitigation (RoM)—the financial impact of preventing breaches, regulation penalties, and reputational damage.
Using the RoM model, we can estimate business impact for pre-attack mitigation. Imagine your team catches a high-severity SQL injection vulnerability during routine testing. Based on the $5.45M average cost of a data breach, its high severity, and exploitation likelihood (5.5%), the estimated mitigated losses is about $149,875.
For more, check out HackerOne’s RoM calculator to see how much financial damage your security investments prevent.
The true cost gap
Another important factor in investment justification is the delta between cost of remediation pre-production vs. post-production.
When a vulnerability is caught in production, fixing the code is a resource-intensive process that requires time and work from both security and engineering teams. How is the vulnerability possible? How and when did it get there? What development team, if any, is working on the application now? If we send the vulnerability report to the developers working on it, will they know how to fix it? Will attempt #1 to fix the vulnerability work? What about #2?
The below image from CISQ provides a good visual of how it plays out in practice: a trial-and-error feedback loop.
Dollar signs indicate where most effort/cost is concentrated.
At HackerOne, we see a median resolution lifecycle of 34 days for vulnerabilities reported to penetration tests.
Even when a patch involves an update to just a few lines of code, determining what lines of code can be like looking for a needle in a haystack. Fixing vulnerabilities discovered in production is roughly 30 times more expensive than finding and fixing them during development.
Source: NIST: The Economic Impacts of Inadequate Infrastructure for Software Testing
Fixing a vulnerability in coding/unit testing is more cost-effective because the context is immediately known, and the complexities involved in writing the code change have already been solved (e.g., navigating existing technical debt to “get it to work”). When a developer is tasked with a vulnerability report to fix, it creates an unplanned cycle of repeating this work. All of the context and technical debt navigation needs to be re-learned. In other words, code patches—even the ones with just a few changed lines of code—take a lot longer than they look.
How big of a problem is this? Just ask your engineering team. Developers spend 13.5 hours a week dealing with technical debt, which is the biggest obstacle to making changes to existing code bases and has an estimated economic impact of $1.52T in the US alone.
A majority of development teams already have a quality assurance stage built into their workflow: pull request review and approval. For most development teams, proposed changes remain in this stage for 47-50 hours, during which defects are caught and fixed prior to merge. During this stage, developers catch and fix about 3.4-4.7 defects per 1,000 lines of code with the help of peer review and automated tooling.¹ These existing development practices provide an opportunity for a non-invasive security policy that, if thoughtfully executed, avoids duplication of development effort and minimal impact on velocity. Ideally, none.
Revisiting our example of an SQL injection vulnerability caught in production, let’s assume the root cause was an implementation where a JavaScript template string was used instead of a parameterized query which would have escaped values properly. If the developer writing the code was informed with the right guidance in a pull request review, it would have taken them 30 minutes to address. Caught in post-production testing, perhaps months later, expect it will take 15 hours of triage, troubleshooting, translating application security terminology to software engineering terminology, etc.
Determining cost impact can be more directly measured based on known operational expenses and varies between organizations. For this example, let’s say the cost is $100 per hour.
Post-production remediation: $1,500
Pre-production remediation: $50
Total operational cost savings: $1,450
This brings overall financial impact to $151,325 when added to return on mitigation ($149,875). Or, 3,027 times cheaper. “An ounce of protection is worth a pound of cure” may be an understatement.
Conclusion
Whether caught pre-production or post-production, catching and fixing vulnerabilities before cybercriminals can exploit them prevents enormous losses. Forecasting business value is best achieved by using a return on mitigation (RoM) framework. The additive business value in pre-production code security comes from significantly lower remediation costs.
There’s no one-size-fits-all methodology for metric-based objectives to determine a “shift-left” security program’s success, but a good place to start is the volume of true positive reports received over the 12 months with a reasonable “prevention” rate prediction (i.e., expecting 40% fewer new true positive reports matching target CWE categories).
How HackerOne is reinventing security for developers
While there’s a clear investment justification for security in development, efforts to “shift-left” often backfire, creating frustration for developers, an unhealthy relationship between engineering and security, and overly strict friction inhibitors on velocity. HackerOne has been on a mission to understand why “shift-left” security isn’t working and to build a methodology-based solution that gets it right.
HackerOne PullRequest is a true, developer-first approach to code security. We combine AI with expert human manual code review. The output – remediation guidance embedded directly in the developers’ existing workflow – empowers them to write secure code and proactively address security risks in the tools they already use. With a developer satisfaction rate of over 96%, HackerOne PullRequest is trusted by development teams because it’s actionable, fast, and self-learned.
¹ Source: HackerOne PullRequest Benchmarks sample of 3,000 organizations and 115,000 repositories spanning Small, Mid-Sized, and Large development team size cohorts.