Employees pasting company data into AI tools worry security teams more than AI-powered attacks do.
Most-concerning risks of AI agents (Source: O’Reilly)
AI-generated phishing, the attack they run into most often, made the list of top concerns for only a quarter of the 209 security professionals O’Reilly surveyed in September.
Mike Loukides, VP of content strategy at O’Reilly Media, believes the fear of AI wiping out humanity distracts from harm happening today. “We don’t need panic. We need an assessment of where we are: What are the threats? Where are the skills gaps? Where is the budget?” he wrote.
Little faith in vendor promises
Data leaking through commercial AI tools ranks among the main concerns for 49% of participants, and the same share worry about staff trusting AI output nobody has checked.
Most AI providers offer enterprise accounts with a promise not to train models on prompts or attached documents. Loukides doubts those terms can be trusted, noting that model makers are so hungry for training data that some have bought failing startups to get it. “Security professionals are paid to be cynical,” he added.
Shadow AI spreads where policies are weak
Nearly three in four organizations have rules for AI use, though fewer than half of the professionals polled described those rules as comprehensive. Over half said co-workers at least occasionally put sensitive data into unapproved AI tools.
The strength of the policy made a difference. At companies with a comprehensive policy, 17% described this shadow AI use as widespread, compared with 35% at those with a basic one, close to the figure reported where no policy exists.
Loukides argues that staff need to know which tools are approved and whether data may leave the company. Left without guidance, people turn to whatever gets the job done, opening the door to legal and reputational trouble.
Familiar threats draw less concern
In the past year, 55% of those polled encountered AI-generated phishing, ahead of vulnerability scanning and exploitation and AI-assisted malware. Loukides suspects familiarity explains the muted reaction, since employees routinely sit through phishing training and filters catch a large share of these emails before anyone sees them.
Prompt injection, where hidden instructions slipped into content trick an AI system, worried 11% of participants. Loukides considers that number low, citing a study that found hidden text meant to trick AI systems in about 1% of résumés.
AI agents and loose permissions
AI agents, software that carries out tasks on a user’s behalf, raise their own problems. The top worry was agents holding excessive permissions or lacking an obvious owner, followed by poor monitoring. Loukides wrote that today’s login and access systems assume identities stay stable over time, an assumption that breaks down when agents hand work off to other agents and models.
Defenders lean on chatbots
Two-thirds of security pros use chatbots or copilots for research and documentation, while about a quarter rely on AI for threat detection or incident response.
Teams using AI tools built for security reported faster alert triage and better detection accuracy than colleagues working with general chatbots alone. Just 7% said they had no skills gap, and the most common wish was to get more out of the AI tools already at hand.
Some experts argue that attackers hold the advantage for the moment, because finding a flaw and launching an attack is easier than stopping one. Loukides expects the balance to tip toward defenders once AI gets better at finding and patching vulnerabilities.
“Getting ahead of the curve on AI won’t happen if individuals work in isolation, adopting tools and ideas as they see fit. It’s an effort that requires the entire organization,” concluded Loukides.
