Government agencies, private businesses and nonprofit organizations have spent decades trying to teach their employees not to click suspicious links or download untrustworthy files, but recent evidence suggests that this cybersecurity awareness training is largely ineffective and possibly even counterproductive.
Organizations rely on cybersecurity education, from phishing simulations to annual webinars, to train their employees to identify and block digital threats. The security industry tells organizations that people are their weakest link and emphasizes training as the solution, and a cottage industry of cybersecurity training programs has sprung up to meet that need. But these programs — a cornerstone of the modern security strategy — are missing the mark.
Common cybersecurity training methods do not significantly reduce people’s likelihood of falling for phishing attacks and in some cases actually make people more susceptible to those attacks, according to a Cybersecurity Dive review of more than a dozen studies and meta-analyses published since 2008. The studies cast doubt on the value of mandatory training, critique the lessons provided to people who fail tests and highlight methodological flaws in earlier research.
“Awareness training, as it is, is not a solution,” said Arun Vishwanath, a cybersecurity researcher and consultant who studies human behavior. “The analogy I use is, you go to the doctor’s office and he throws a pill at you, which is awareness training. And then you go back, and the patient’s still sick, and they give you more of it, and they keep giving you more of it, and in the end, they blame you.”
Unintended consequences
Most organizations that deliver cybersecurity training do so in one (or both) of two ways: periodic assessments, often conducted annually or monthly, and embedded training lessons, which are displayed after a user fails a phishing test. But recent studies have questioned the efficacy of both of these landmark tools.
In a widely discussed paper presented at multiple conferences over the summer, a team of researchers at the University of Chicago and the University of California, San Diego found “no evidence that annual security awareness training correlates with reduced phishing failures.” The researchers expected to see better performance from people who had recently completed the training, but their study found no significant connection between how recently a user completed a phishing training and how well they performed on a phishing test.
“Annual awareness training is not providing meaningful new knowledge or education to users,” said Grant Ho, an assistant professor of computer science at the University of Chicago and one of the study’s authors.
Ho and his co-authors wrote that the cybersecurity community “should re-examine whether such training, as delivered today, provides meaningful security benefits.”
There are equally serious problems with how remedial lessons are assigned.
For one thing, the lessons are presented only to people who fail a test, meaning that they exclude others who might be susceptible to future failure. “This design implicitly assumes that users who do not fall for one phishing lure do not need training to protect against future attacks,” Ho and his co-authors wrote. “Unfortunately, our results show that the majority of users at our organization will eventually fall for a simulated phishing attack given enough time.” Because embedded phishing lessons don’t reach everyone who needs to see them, the researchers said, they are an inefficient way of educating people.
Other research casts doubt on the importance of immediately presenting failing users with information about how to pass future tests. In a study published in late 2024, researchers at ETH Zurich found that “informing employees about the exercise and directing them to training material the day after the incident … seems to be as effective as ‘immediate’ embedded training.” In other words, there is no need to confront people right after they fail.
In 2021, the ETH Zurich researchers reported on even more concerning results from a study of embedded training. Not only does the approach “not make employees more resilient to phishing,” they wrote, but it can have negative side effects that “make employees even more susceptible to phishing.”
The researchers elaborated on this finding in their more recent paper, writing that embedded training “can make employees overconfident both in their abilities and in the fact that mistakes in phishing tests are without repercussions.” Because embedded training can “create misunderstandings and overconfidence,” they wrote, organizations should exercise caution in deploying it.
Studies also suggest that mandatory training is not an effective intervention for the people at the greatest risk of falling prey to phishing attacks.
In the 2024 ETH Zurich study, the researchers separated phishing test participants into two groups, and they warned the people in one group that two failures would result in mandatory training. They found no statistically significant difference between the performances of people in the two groups. “For the most susceptible participants,” the researchers wrote, “mandatory training did not provide additional benefits.”
Researchers at Harvard University and its affiliated health system found similar results in a 2019 study. They delivered 20 phishing campaigns to more than 5,400 employees at an unnamed healthcare organization, and, after the 15th campaign, they required people who had clicked on at least five of the lures to undergo mandatory training. The training “did not have a substantial impact on click rates,” the researchers wrote, “and the offenders remained more likely to click on a phishing simulation.”
Fleeting effects
Even when training seemed to improve people’s ability to spot phishing attacks, those effects did not last long.
“Evidence on the success of programs in driving sustained behavioral change is limited,” a trio of researchers at the University of Adelaide in Australia concluded in a 2023 review of dozens of studies of phishing awareness training programs.
According to another study, presented at a conference in 2020, people were significantly better at distinguishing between real and fraudulent emails immediately after training and four months later, but by the six-month mark, the improvement had disappeared.
“Our habits are stronger than the nudges we receive,” Vishwanath said, adding that preconceived notions about risk “have more inertia” than ephemeral training material.
Knowledge is not enough
One of the biggest obstacles to effective security awareness training is the difficulty of converting knowledge into behavior. Training sessions may teach people to identify phishing attacks but still fail to protect them from those attacks. Human behavior is a complicated mixture of knowledge, attitudes and incentives, and many training programs fail to recognize what makes people behave the way they do.
“While training significantly increases predictors of end-user behaviour, such as attitudes or knowledge, changes in behaviour can only be observed minimally,” researchers at Leiden University in The Netherlands wrote in a 2024 meta-analysis of 69 studies.
“We have become extremely good at changing these precursors to behaviour, but not the actual behaviour that is necessary to be secure,” said Julia Prümmer, a Leiden PhD candidate who co-authored the meta-analysis and another recent review of cybersecurity training studies.
Researchers at the University of Oxford came to the same conclusion in a 2019 paper.
“Knowledge and awareness is a prerequisite to change behaviour but not necessarily sufficient, and this is why it has to be implemented in conjunction with other influencing strategies,” they wrote. “Answering questions correctly does not mean that the individual is motivated to behave according to the knowledge gained during an awareness programme.”
The 2024 ETH Zurich study found that regular “nudges” — reminders about the dangers of phishing attacks and the importance of blocking them — were the main drivers of phishing training’s effectiveness, rather than the content of the training modules, which even “the most susceptible participants” described as unhelpful.
Kari Kostiainen, a senior scientist at ETH Zurich and a co-author of the study, said the finding about nudges and content should prompt organizations to “reconsider their phishing defenses as a whole.”
“Instead of emphasizing the testing/tricking aspect,” he said, “the emphasis could be on reminders and reporting.”
Critique of ‘contrived’ conditions
Numerous studies over the years have found that security awareness training improves people’s security practices, but because of methodological issues, those findings may not be as meaningful as they initially seem.
Many studies of phishing training involve volunteers taking tests in a research lab. This environment, in which participants are intensely engaged with training materials and alert to the dangers of phishing, may yield unrealistically “positive results about training efficacy,” the University of Chicago and the University of California, San Diego researchers wrote in their recent paper. They said their findings showed that “very few users engage with embedded training in-the-wild.”
The Australian researchers highlighted the limited value of studies conducted “under contrived experimental conditions” that “do not provide real insight on the success of programs in driving sustained behavioral change under naturalistic conditions.”
Some of the studies that yielded promising results acknowledged their work’s limitations. In a 2008 paper on the effectiveness of a phishing awareness video, a team of German and Scottish researchers noted that their participants’ performance “should definitely be considered a ‘best-case’ scenario” because of how they were primed to focus on security. “Their actual detection rates are likely to be poorer in the real world,” the researchers wrote.
Phishing-training research has suffered from other methodological limitations, according to a 2024 literature review by the Leiden University team. Some studies had a small sample size, while others didn’t test participants enough times to reliably determine if they were susceptible to being tricked. Other studies offered participants only one training session before evaluating their skills.
In some cases, according to the authors of the literature review, researchers focused on studying the wrong metrics. “Outcome measurements were often not concerned with cybersecurity behaviour, but focused instead on behavioural intentions, changes in attitudes and perceptions, or other metrics,” they wrote. While many theories of behaviour have identified these factors as predictors of behaviour change … this association is often weak.”
Understanding root causes and shaping habits
The current scholarly consensus is that “commonly deployed forms of training offer small or minimal protective benefits,” said Ho, the University of Chicago researcher. And improving training programs will require significantly changing how they are designed, delivered and evaluated.
The Oxford researchers said phishing training should focus on changing behavior by using proven persuasion strategies — and avoiding counterproductive ones. Among their recommendations:
- scaring or shaming people “is not an effective tactic”
- educational content should be “targeted, actionable [and] doable”
- organizations should provide “continuous feedback” to help users form good habits
Organizations should also prioritize training that shapes people’s attitudes about security, which will influence their motivations and then their behavior, the researchers said.
Vishwanath, who has created a model to categorize the human impulses that affect phishing susceptibility, argues that current awareness training ignores behavioral science in favor of easily scalable, one-size-fits-all knowledge transmission.
“None of these programs deal with correcting habits,” he said. They also fail to address misconceptions about security threats. “What you believe about the risk of what you do is very important. None of the security awareness programs address this.”
Prümmer agrees. “Embedded training often does not look at the root cause of why someone falls for a phishing email,” she said. “We first need to understand what leads to increased victimization online, before we can try and fix it.”
Ultimately, no one training regimen will suffice for all organizations and situations, because users exhibit a wide variety of concerning behaviors. “We need to find methods of training that are suitable for each cybersecurity behaviour we are attempting to address individually,” the Leiden University research team argued.
Going forward, Vishwanath said, researchers should focus more on fixing flawed training than on criticizing programs that many regulated companies are required to use.
For now, businesses and government agencies remain vulnerable to breaches that have skyrocketed even as awareness has also increased dramatically.
“I don’t think we’re any farther along when it comes to cyber resilience than we were before [awareness campaigns],” Vishwanath said. “We feel like we’re doing something about it. We’ve spent a lot of money doing this. But are we any better? I don’t think so.”
