Offensive AI: The Evolution of the Red Team

Every August, the AI Village at DEF CON serves as a proving ground for offensive AI research. Researchers and hobbyists alike line up to jailbreak various production models, chaining indirect prompt injection through documents and web content and extracting system prompts out of supposedly “hardened” assistants. All in front of a crowd. It’s chaotic, fun, and also some of the most honest AI security research happening anywhere. Because nothing survives in a room full of people whose only goal is to try and make your model misbehave.  

For AHEAD’s Red Team, the AI Village isn’t just a spectator sport, but an example of the techniques we’ll be running against client AI systems for the next twelve months. So instead of recapping talks, we’re going to take a different (and slightly more useful) approach. What happens in Vegas this year shouldn’t stay in Vegas. The techniques emerging from the AI Village map directly onto how we scope, attack, and report on AI and agentic engagements.  

From Show Floor to Industry-Wide 

The security space has finally internalized that prompt injection is a legitimate threat vector. Previously, prompt injection was seen as an offshoot of general “AI safety” or a PR problem, where company chatbots got tricked into saying something off-brand. 

The research being presented at Black Hat and DEF CON — and emerging from the broader community — has completely reframed that discussion. Models that make decisions or invoke tools should be treated as untrusted execution surfaces. Prompt injection isn’t about trying to make a chatbot swear, but about making an agent act. That’s what turns a coding assistant into a remote code execution vector through a poisoned GitHub issue. Or exfiltrates conversation history through a rendered image. Or walks an autonomous agent through a multi-step task it wasn’t authorized to perform. If an LLM decides which actions and tools to invoke, you have to assume an attacker can control that output, and therefore everything downstream of it. That’s what makes this a vital security factor for every AI engagement and every model you’ve put into production. 

Prompt Injection Is Initial Access  

In a traditional intrusion, initial access comes from a phished credential, an exposed service, or an unpatched CVE. The industry has been scrambling to understand that in an AI-enabled system, the model itself is the front door. It takes its instructions from data, and data is everywhere the model can reach.  

One of the biggest emerging threats (and what’s keeping our own architects up at night) is indirect prompt injection. Direct injection is when an attacker types a malicious prompt themselves. In those cases, the blast radius is usually their own session. Indirect injection, on the other hand, is when the attacker plants instructions in content the model will later read on someone else’s behalf, whether that’s a support ticket, a PDF in a RAG store, or even a calendar invite. The user asks an innocent question, but the agent retrieves poisoned content, and may execute the attacker’s instructions with the user’s privileges. 

There’s no malware or exploit chain in the classic sense. The model’s still working exactly as designed, reading text and following instructions. But that’s why traditional controls can’t see it coming. The attack’s right there, in plain language, sitting inside a document your AI was built to trust.  

The Attacker’s View  

You don’t need a zero-day. You need to find one input the model trusts, and that an outsider can influence. So map the data flows: What does this assistant read? Where does that data come from? Who can write to it? A RAG pipeline that ingests customer emails, or a documentation site that indexes public web content. Even an agent that summarizes incoming tickets. Any of these can serve as a Trojan Horse for malicious instructions. 

Then you escalate from talk to action. Can the agent call tools? Execute code? Send an email? Touch an internal API? Every capability you hand the model is a capability the attacker inherits the moment they own the model’s output. The lesson we keep relearning is that the attack surface isn’t the model but everything the model can touch. An over-privileged, under-isolated agent can turn a simple sentence into a full-blown breach.  

What Defenders Have to Do Differently  

Here’s the good news: Defending against this involves a lot of security engineering you already know. It’s just about applying that discipline to a component your team isn’t treating as hostile. 

• Every model input should be untrusted. This includes retrieved data. The hard boundary isn’t between user and system but between instructions and data. And the model can’t reliably distinguish the two. You need to scope, label, and constrain whatever enters the context window. 
• Apply least privilege to agent capabilities, not just identities. If an agent doesn’t need to execute code, send mail, or reach an internal service, don’t wire it up. Every tool you remove shrinks the blast radius of a successful injection. 
• Put a human in the loop. Autonomous read is a risk class. The ability to write data or execute code is another. High-impact steps should require human verification the agent can’t simply talk its way around. 
• Isolate and sandbox execution. When an agent runs generated code or calls tools, assume the input was attacker-controlled and contain it accordingly. No shared credentials or ambient access to sensitive systems. 
• Instrument AI like production infrastructure. Log prompts, retrievals, tool calls, and outputs. All of this is valuable information that provides your SOC with visibility into what the agent saw and what it did, with AI-specific detections and an incident taxonomy to match. You can’t investigate what you never recorded. 

Several of these controls address risks recently identified in the OWASP Top 10 for Agentic Applications, including goal hijacking, tool misuse, privilege abuse, and memory or context poisoning. The frameworks have finally caught up. Has your architecture? 

What Leadership Needs to Understand  

The race to deploy AI assistants and agents has created vast new potential. But like we said at the start, it’s also led to a rapidly expanding attack surface that most security programs haven’t scoped, tested, or budgeted for yet.  

There’s no longer a good excuse for leaving AI security at the bottom of your to-do list. Especially when the techniques and, in some cases, the tooling for attacking agents are demonstrated publicly every August, for anybody to watch. That’s not to suggest you slow down AI adoption. But security has to be a precondition of it, with governance, secure design, and red teaming built into the roadmap from the beginning. Winning the AI race takes a plan where AI can be deployed and defended at the same speed. 

The Bottom Line  

The most important thing coming out of the AI Village is consensus: Prompt injection is initial access, and models in your AI system are part of your attack surface now. The offense has learned this by now. Defenders need to do the same. 

AHEAD’s Red Team is built for exactly this type of work. Our AI Penetration Testing stress-tests your models, agents, and the systems they touch using techniques demonstrated at Black Hat and DEF CON. And our broader Security for AI practice helps you design the governance, controls, and detections to address each gap we find. If you’re putting AI into production, it’s time to find out what an attacker can talk their way into. Let’s get started.