More accreditation and compliance requirements have been added in response to cyber incidents. While these frameworks play an important role in establishing security baselines, true security is more than just achieving a perfect compliance score. As I often say, “policies and procedures won’t stop an attacker, they’ll just have more documents to exfiltrate when they breach us.”
Testing how our environments withstand a determined threat actor is the real validation of security posture. That’s where the annual manual penetration test comes in, with boards now demanding to see positive results.
There are, however, significant issues with manual penetration testing I have experienced, particularly when conducted only annually.
Speed, scope, and the human bottleneck
The constraints of manual testing became increasingly apparent as our environment grew more complex. Every engagement was bound by time and budget, forcing difficult trade-offs about what to test and how deeply. The quality and comprehensiveness of results varied significantly depending on which consultant we engaged, their individual expertise, their familiarity with emerging techniques, and how much they could accomplish within the contracted hours.
Traditional penetration testing delivered what I came to see as a fundamentally flawed value proposition. We’d invest significant budget to receive a snapshot of our security posture weeks after the test concluded and from that moment it began aging like milk. There was no ongoing feedback loop, no continuous validation of our security controls. We were essentially flying blind between annual tests, hoping our defenses remained effective even as the threat landscape evolved daily around us.
The remediation black hole
Perhaps most frustrating was what happened after we received findings. Our teams would work diligently to implement fixes, but we rarely had the budget or opportunity to bring testers back to validate remediation. We were left with uncertainty. This gap between identification and verification created a dangerous blind spot in our security program.
Traditional vulnerability assessments leaned heavily on CVSS severity scores that did not tell us how exploitable a vulnerability was in our specific environment or where it sat within a realistic attack path. We needed to understand what an attacker could actually accomplish by chaining vulnerabilities together.
A better way forward
Frustrated with these limitations, I explored automated penetration testing, a category that includes breach and attack simulation (BAS) and continuous automated red teaming (CART). Platforms like Pentera and Horizon3.ai’s NodeZero conduct continuous, on-demand simulations using real-world attacker tactics, techniques, and procedures.
They offer black box testing (simulating external attackers), grey box testing (simulating insider threats), and custom scenarios targeting specific risks like ransomware or zero-day exploits.
Most importantly, they deliver results instantly, no waiting weeks for reports, and enable immediate retesting to validate fixes.
The implementation and investment
We moved from $35,000 for an annual manual test to $90,000 annually for an automated platform, delivering over $1.3 million worth of equivalent testing. Our cadence jumped from one test per year to a minimum of 38, with unlimited flexibility for additional simulations.
We established a fortnightly rhythm of black box and grey box tests, supplemented by monthly custom scenarios targeting specific concerns like ransomware attacks. This gave our team two weeks to remediate before retesting confirmed fixes worked. These tools test more in a day than human testers accomplish in a week, rapidly adjusting to findings and leveraging gaps to probe deeper.
Unexpected lessons and team transformation
The platform delivered insights that fundamentally changed our understanding. Take password security: we’d adopted longer passphrases, confident that fourteen-character phrases would increase breach time from eight months to twelve billion years. The tool shattered that confidence, cracking a 23-character passphrase containing upper- and lower-case letters, numbers, and special characters in under half an hour. The lesson was humbling, humans are predictable. Attackers maintain wordlists and precomputed hash lists in rainbow tables specifically targeting common phrases. Passphrase length matters, but quality matters more.
The retesting capabilities proved game changing. Security teams could identify problems, remediate them, and immediately retest to verify fixes were effective. The platform generated both executive-level reports for board presentations and detailed technical reports for security teams to action instantly, not weeks later.
Perhaps most importantly, the platform elevated our team’s capability. Until your team experiences an automated penetration testing tool exploiting their environment, they won’t fully comprehend how to apply defensive concepts to their specific systems. Each simulated attack was fully documented, providing real-time learning opportunities. The teams began treating the platform as a game they were determined to win.
Rethinking prioritization: attack paths over severity scores
One of the most significant revelations was how automated penetration testing transformed our vulnerability management. We discovered that the critical-rated vulnerability receiving immediate attention might be buried five layers deep in an attack path, while a low-rated vulnerability we’d deprioritized could be the initial entry point attackers would exploit. More revealing still, the platform showed how seemingly low-risk vulnerabilities could be chained together to access critical systems.
This changed our patching strategy. Instead of reflexively addressing vulnerabilities by CVSS severity ratings, we focused on what attackers could actually use to establish a foothold. Given the overwhelming number of vulnerabilities requiring constant attention, this intelligence about actual attack pathways proved invaluable allowing us to focus limited resources where they’d produce the greatest security outcome rather than chasing severity scores that didn’t reflect real-world risk.
The gap between configuration and reality
We place enormous faith in our security tooling when we enable a feature, we assume it’s working. The automated penetration testing platform delivered a sobering lesson: test your controls, don’t just trust the GUI.
I experienced this firsthand when we enabled a functionality to mitigate a specific risk. It looked perfect on screen, but it wasn’t working. The platform methodically tested different attack types, including the scenario we thought we’d protected against. The attack succeeded, the security tool’s features weren’t functioning due to a bug. We didn’t have the protection we thought we did.
It reminds me of the defender’s dilemma: “Defenders have to be right 100% of the time; attackers only have to get it right once.” I’d much prefer our own testing tools highlight these gaps than have attackers discover them.
The ultimate validation: Testing your detection and response
Another powerful application is validating your detection tools and SOC. The first time I ran a proof of concept, I deliberately didn’t inform our third-party SOC. Our internal SIEM immediately generated numerous alerts. It took four hours for the external SOC to contact us — a lifetime in cybersecurity.
When you’re paying for a third-party service, validating their response is invaluable and I strongly recommend running at least one unannounced test. The results may surprise you, and it’s far better to discover gaps during your own testing than during an actual incident.
One final lesson: as your security resilience improves and you achieve consistently high scores, you reach a plateau. Moving to a new automated penetration testing platform can yield fresh findings, as each tool takes different approaches, providing opportunities to continue improving rather than becoming complacent.
The verdict: Evolution, not elimination
Should you replace manual penetration testing with automated platforms? The answer is nuanced. For ongoing security validation, continuous improvement, and operational resilience, automated testing should become your primary validation method. The ROI, learning opportunities, and continuous feedback loop far exceed what annual manual testing delivers.
However, I wouldn’t completely eliminate manual testing. There’s still value in bringing in specialized human testers for complex custom applications, critical infrastructure changes, or when you need creative thinking that only experienced security researchers provide. Think of automated platforms as your daily training regimen, with manual tests as occasional specialized assessments.
The real question is whether you can afford not to adopt continuous automated validation. The gap between annual manual tests leaves you vulnerable for 364 days a year. Automated penetration testing fills that gap, transforms your team’s capabilities, and validates your security posture continuously, not just once a year when auditors ask.
