Going Into the Deep End: Social Engineering and the AI Flood

It should come as no surprise that the vast majority of data breaches involve the “human element.”  The 2025 Verizon Data Breach Investigations Report cites that human compromise held relatively steady year over year at nearly 70% of breaches.  Human emotions and tendencies – and the massive variation in what influences each individual – are a massively dynamic vulnerability.  Most equate Social Engineering with vague promises of riches to be had, or urgent or even threatening missives that require immediate action to avoid consequences. On the plus side, increased awareness has brought about a healthy skepticism in individuals and organizations toward something unexpected from a not completely familiar source.

Unfortunately, with the rapid rise and advancement of Artificial Intelligence (AI), criminals have powerful new tools to boost not only the believability of scams, but also the volume of humans they can attack quickly – and as they say, the bad guys only need to be right once.  However, AI can also be an equally potent ally for defenders in accelerating their ability to identify and blunt the impact of human targeting and compromise.   While this may look like the age old, “cat and mouse” game between attackers and defenders, we’ve reached another crossroads, where an exponential jump in attack capability needs to be met with an equal jump in defensive response to at least keep pace.

Let’s look at the AI “pool” of capabilities and challenges available to attackers and defenders, and the AI development representing a springboard that can launch the bad guys onto a new level – Deepfakes. 

“Learning” to Sink or Swim

Systems that can learn “autonomously” have not only been a staple of Hollywood for decades, but also a capability touted by security vendors for many years.  Unfortunately, as with any new capability, there are many that overstate the capabilities to ride the wave of popularity and profitability.  So, while in the early days anti-virus vendors effectively leveraged machine learning to continuously improve and iterate on malware detection signatures, it of course wasn’t long before any learning capabilities were termed “AI.” 

While early AI capabilities may have more accurately been described as “Artificially Inflated”, the speed at which we’ve moved from more basic machine learning to AI based on powerful Large Language Models (LLMs), cannot be overstated, or underestimated.

To put it bluntly, with today’s LLMs everything can be better, faster, bigger, and more precise.  For attackers, they ae already aggressively leveraging AI for better attack lure crafting and automating attacks at scale.  They’ve even begun to use gen-AI for malware adaptation/evasion.  While not seen widely in the wild yet, these advancements portend an inevitable trend toward autonomous ransomware and malware in the not too distant future.

But fear not, or at least not yet, because AI can be a powerful tool for defensive purposes.  LLMs enable defensive “needle” hunting at much greater scale.  The speed at which LLMs can analyze massive “haystacks” of data and activity and find the anomalies has become exponentially more efficient.  Additionally, by drawing from the attackers’ own playbooks, Red Teams can and are using AI to craft and conduct more effective simulations and training.  However, there is something emerging quickly from the depths that warrants a healthy dose of fear, both of what’s already possible, and what will likely soon breach the surface in spectacular fashion.

In over our heads

The “White Whale” we have already begun to face are Deepfakes and real-time human imitation that represent a transformational change for attacks and attackers.  Attackers have already proven the unsettling effectiveness of pre-recorded deepfakes to more easily override the default skepticism by projecting not only the appearance of validity in a request, but cloning the complete likeness of a known requestor.

Deepfakes are in the proof-of-concept stage where the majority of attacks are still more along the traditional lines. But just an inventory of this year, we’ve seen:

From an end-user security perspective, the challenge of Deepfakes are not unlike the move to the cloud.  We moved from high control and visibility over “infrastructure and assets” — with enforceable guardrails in both tech and process – to wild west deployment of new assets where we were forced to rely too much on policy as the primary guardrail.  With regards to Deepfakes, the relative “rough around the edges” quality of current real-time deepfakes are like managing a hybrid cloud model, but full cloud native is on the horizon.

What’s most concerning about this is that defensive AI seems to be getting bogged down in automation and filtering, and inordinately focusing on indicators of compromise, not indicators of vulnerability.  To keep pace and hope to blunt the coming deepfake tsunami, we need more defensive AI development that is about human analysis and augmentation – with regards to both defensive and offensive testing of end user communications.

No lifeguard on duty – yet

To get right to the point, the bad guys have a major hand up in this race.  They have a range of easily accessible, open-source tools to choose from, and with which they can begin to act today and with minimal investment.

Conversely, the good guys have lost control of what were once foundational verification inputs in voice and image, and there are no reliable technical countermeasures that are widely available. There are for sure efforts underway that show promise, such as the DARPA SemaFor project. They are working furiously to train detections and remove workarounds.  But perfecting that will take time, and then broad deployment will take more.

Until reliable and repeatable tech is available, the best weapon in the defensive arsenal is situational awareness and continuous vigilance. Organizations need to be having discussions about this now and reorienting people and processes to create barriers to human exploitation.  I live in Arizona, and we are neighbors – and sometimes unwitting houseguests – with the Bark Scorpion, the most venomous scorpion in North America. They are nocturnal, so when they are most active is when you are least likely to see them. However, they have a natural “tell” that exposes their presence. UV light, even at a safe distance, causes them to glow. Why do I bring this up?  Because organizations can expose multiple “tells” associated with deepfakes, including:

• Just as in the Ferrari case, require multi-factor interactions thatgo beyond voice and image, and include elements like presence verification (e.g call back numbers), unique knowledge (e.g. shared personal details/experiences) and/or verbal queues (e.g. passphrases) for sensitive communications and tasks.
• In the same vein, something I’ll call “Egoless” Verification. Educate on and promote/encourage a more aggressive culture of skepticism and confirmation of requestors and requested actions.  If everyone – from administrative to the C-Suite – is subject to extra steps, no one can feel pressured to act faster than necessary.
• Undertaking Open Source Intelligence (OSINT) to inventory content that could serve to train deepfake models (public videos, live presentations, investor calls, podcasts, etc.) to understand those individuals in the organization who are most susceptible to deepfake creation.
• Investing time and resources more heavily in crisis management tools such as tabletop exercises to train both individual and team “muscle memory” in identification, escalation and response with regards to anomalous acts.

Learn More at the AI Risk Summit

Related: How Hackers Manipulate Agentic AI With Prompt Engineering

Related: How Agentic AI will be Weaponized for Social Engineering Attacks