In my last post I discussed how developers can be your security secret weapon… but how to help them love doing security work? That’s a whole other challenge!
Stories of the tension between developers and security teams are a longstanding feature of the software industry, stemming from the friction that security is often perceived to create. Developers want to keep moving forward on valuable new features and many care deeply about the security of their code. But historically, security tasks are challenging to address. Tasks either lack context, clear ownership, or clear mitigation paths which ultimately drags down development velocity and allows security vulnerabilities to persist.
Against this backdrop, what can you do to improve the relationship between your developers and security teams and protect your product? It starts with understanding developer concerns.
Why some developers find security burdensome
Developers giving security the cold shoulder isn’t just a myth: Industry surveys have repeatedly shown that engineers try to avoid security work, while security teams become frustrated at engineers’ lack of action.
GitLab’s 2022 DevSecOps report found that 57% of organizations believe security is a relevant metric for assessing developer performance, but 56% also noted that developers rarely prioritize fixing security problems.
Unsurprisingly security teams, not developers, find 75% of vulnerabilities. However, security is often ill-equipped to patch flaws alone. And if developers are unable to prioritize newly found risks, vulnerabilities can remain in the product long after they’ve been discovered.
To break down these silos, it’s important to listen to developer concerns and implement security tooling that encourage and empower engineers to participate in security. Below, let’s start with a few of the top issues inhibiting successful developer-security collaboration.
Lack of business context
Security risks are rarely contextualized within organizations. When there’s no business case for security, why should we expect developers to care about strengthening it by proactively taking on recommended fixes from security teams? New features often appear more valuable than slowing down to inspect security issues, especially when the product has no history of known vulnerabilities.
Product leaders are great at tying development work to tangible impact being made by development teams. You should provide clear context – both within the security team and to the development team – on why a security task is important. If security isn’t providing this context, then developers are being asked to forgo new feature development in favor of security issues without clear understanding as to why the extra attention is required, such as when a payment system becomes in-scope for a specific regulatory framework.
Unclear prioritization
Developers often fear new security tasks because they can come out of the blue and are all treated as critical, without regard for developers’ other scheduled work. Security fixes are important, but are they more important than meeting existing product deadlines? Where product and feature work is usually perceived as revenue generating, security is often viewed as a cost. They may be more likely to prioritize features at the expense of security problems.
We cannot fall victim to “the security team that cried wolf.” Clear and consistent prioritization is key. Developers need to know when a vulnerability is truly critical or high and should take priority over product work, and why. They’ll also welcome assurances that time spent patching critical issues will be recognized in their overall productivity.
Disputes over ownership and responsibility
Most Dev-Sec disputes stem from doubts over who’s responsible for a vulnerability. Developers often feel like their role is to engineer new features, while security and DevOps should take care of aspects such as the software supply chain, developer access management, detection of hardcoded secrets, and environment hardening.
Yet security can credibly claim that developer input is required in many of these areas: developers add new packages to the software supply chain, and developers commit hardcoded secrets, for example. In the case of hardcoded secrets, developers also have the necessary context to quickly find and fix the secret where it exists.
This doesn’t mean developers are always at fault, though. The security team should have a clear remit for which risks should not be queued with developers, as well. If developers are constantly taking on all security tasks, it can be a sign that security should bring more of its work in-house. This can boost overall productivity and avoid engineers feeling like they’re controlled by the security team.
Long feedback loops
Developers hate being slowed down or interrupted. Unfortunately, legacy security testing systems often have long feedback loops that negatively impact developer velocity. Whether it’s complex automated scans or asking the security team to complete manual reviews, these activities are a source of friction. They increase the delay between making a change and verifying its effect.
Security suites with many different tools can result in context switching and multi-step mitigations. Additionally, tools aren’t always equipped to find problems in older code, either. Only scanning the new changes in your pipeline maximizes performance, but this can allow oversights to occur as more vulnerabilities become known.
Similarly, developers have to refamiliarize themselves with old work whenever a vulnerability impacts it. This is a cognitive burden that further increases the fix’s overall time and effort. All too often, these problems add up to an inefficient security model that prevents timely patches and consumes developers’ productive hours.
Insufficient ability to make change
Security might view developers as reluctant to prioritize fixes, while developers may not feel empowered to address the issue themselves. There can be many different reasons for this:
Developers may not have precise security-specific knowledge. It’s the role of the security team to define expectations and establish baseline standards. Non-security engineers may not have the ability to enforce good security practices if they’re unfamiliar with common attack vectors, are working in an unknown part of the code, or haven’t been introduced to the organization’s security standards.
Developers may struggle to stay updated. Developers have a full time responsibility in developing features and can fall behind on their security knowledge if they’re not given sufficient time and education to regularly hone their security skills. Regularly changing security expectations has the effect of pulling the rug from under developers, suddenly tasking them with addressing a slew of new failures in code that was acceptable yesterday, last week, or last month. Developers will feel like they’re constantly battling a crisis that they’re unable to influence or fully resolve.
External pressures can thwart good intentions. Even when developers want to prioritize security, they can be prevented from doing so by other stakeholders who have yet to recognize its value. Product managers and business execs could pressure developers to stay focused on new features, even when it means dropping planned security patches.
How to change developer minds
After listening to developers discuss their reluctance to work on security, you can alter your processes to change their minds. Aim to provide developers with support so they can contribute to fixing security problems, without detracting from their engineering productivity.
Provide a clear security toolchain
To start with, make sure developers have convenient tools for identifying and fixing risks. Automate as much of the security process as possible to save time and achieve results with higher accuracy. Your tools should be fully integrated into your processes while staying focused enough to run quickly in a tight feedback loop.
There are multiple elements you can include here:
- Static application security testing (SAST) to spot known problematic coding practices
- Software composition analysis (SCA) to identify vulnerabilities in the code
- Dynamic application security testing (DAST) to probe your application’s endpoints and find live vulnerabilities
- Scanning commits for exposed secrets to avoid exposure of sensitive credentials
- Monitoring systems to detect anomalous developer activity such as malicious code and spoofed identities
Your tools should provide developers with simple solutions for fixing discovered risks, as well as notifications for any fixes that were automated on their behalf. A list of problems without remedies will only slow you (and your developers) down and cause alert fatigue. Actionable insights and recommended resolutions will keep you moving forward.
Help developers maintain velocity
Disruption to development schedules is often an engineer’s biggest fear in relation to security. Address this by implementing mechanisms that allow security work to occur more efficiently, without impacting development velocity.
This can be only partially achieved via conventional security toolchains. Truly frictionless security requires a new approach that gives you rapid targeted feedback and full coverage as each change occurs.
A “pipelineless” approach to security allows you to easily establish and maintain 100% security scanning from day one and it opens the door to run security workflows earlier and more frequently. In contrast to traditional security tools, pipelineless security should drive an increase in development velocity because you’ve eliminated the need for code changes in the pipeline as well as indiscriminate security to-do’s in favor of security findings being sent directly to the appropriate person (or channel) best equipped to solve the security issue quickly. Not only that, you can automate mitigations based on policy or a single click from a developer in their chat tool of choice: “Fix it for me.”
Treating security as a check, rather than a CI pipeline stage, transforms it into a lightweight process that’s more scalable and maintainable. Security teams can register new checks without requiring CI config modifications, creating less work for developers.
Consider a pipelineless model to reliably enforce your organization’s security policies and maximize developer productivity.
Don’t shame and blame
As we’ve discussed, traditional security tools will often push security vulnerabilities into a security backlog that has broad visibility across the development organization. This is problematic for two reasons. First, the security finding itself may contain sensitive information that would be better suited for a direct communication to a single developer rather than across the whole team – eliminating an exposed secret, for example.
But more central to the topic of developer-security collaboration, the paradigm of broadcasting blame on a developer for a security risk being introduced is counterproductive to endearing developers to the cause. Rather for security to be truly developer-centric, it should provide them with actionable feedback – preferably direct feedback – and opportunities to easily improve code security within their regular workflows.
In the example of hardcoded secrets that we just mentioned, the secret that is detected on push should halt the push and alert the developer directly that they are about to push a hardcoded secret. Not only that: the developer should receive clearly laid out paths to manually fix the issue quickly or automate the fix.
In this scenario, the risk has been mitigated before it was ever introduced to production, the developer was empowered to fix the problem easily, and, critically, has avoided blame and shame for the possible risk being introduced.
Clearly classify ownership and prioritization for risks
Finally, seek to clear up developer concerns about security ownership, responsibility, and prioritization. Help developers understand security’s scope by contextualizing its importance at the organization, team, and project levels.
Success in this area is dependent on clear communication of what security and developers are each responsible for, but also of what security can automate on behalf of developers. Make sure you tell your developers about the “No New Hardcoded Secrets” policy that you’ve automated by detecting and mitigating secrets on push.
Documenting expected standards and the immediate consequences of a failure ensures clear and consistent accountability each time there’s a new vulnerability. This enables both development and security teams to move quickly to address the problem, without being distracted by questions of blame and prioritization.
Summary
Developers often (understandably) see security as a delay to new feature work. This perception produces unwanted and unnecessary friction in the software delivery process. Nevertheless, security is an essential part of all modern application development. Everyone has a role in maintaining good security—and that includes engineers.
You can help developers and security teams work better together by “shifting left” to fully integrate the two disciplines. But rather than shifting work left, shift detection and automated mitigation left instead. Shift clear prioritization left. Achieve this by setting up automated toolchains and establishing a clear classification system for new security threats. This will keep everyone informed about the ownership and priority of each risk.