Preventing Unauthorized AI Agent Actions Through Instruction-Origin Tracing

AI agents can send emails, transfer money, and delete files. They process untrusted data and act on what they find. When an attacker embeds instructions in an email body, the agent follows them.

The defense doesn't ask "is this action dangerous?" It asks "who instructed this action?" If a target came from data the agent read rather than something the user specified, the monitor blocks it. Same architecture works for prompt injection today and intentional subversion tomorrow.

871 attack scenarios. Same monitoring prompt across all 5 models. The 31.5% baseline reflects worst-case autonomy conditions.

How to read this table: "Attack rate" = % of attack scenarios where the injected task succeeded. "Attack utility" = % of attack scenarios where the agent still completed the user's real task (higher is better). "Honest utility" = % of honest scenarios (no attack) where the agent completed the task.

The problem: AI agents execute unauthorized actions

Real incidents (2025):

• GitHub MCP: injection in public issues led to exfiltration of private repos and API keys
• Microsoft Copilot EchoLeak: zero-click email led to data exfiltration via trusted domains
• ServiceNow AI agents tricked into unauthorized actions through injected instructions
• OWASP #1 vulnerability in LLM applications (73% of enterprise deployments affected)

These share a root cause: the agent can't distinguish between instructions from the user and instructions embedded in data. But prompt injection isn't the only way agents act unauthorized. Hallucination, goal drift, and intentional subversion all produce the same outcome. We need defenses that work regardless of why the agent acts unauthorized.

Why this matters now and later

Today: prompt injection compromises deployed systems. An attacker embeds instructions in an email, a document, or a webpage, and the agent follows them.

Tomorrow: as agents become more capable and autonomous, the risk scales. More tools, broader delegation, less human oversight. Intentional subversion becomes a realistic concern when agents have sufficient capability.

The AI control framework provides the right mental model: don't rely on the agent being robust. Use a trusted monitor to catch unauthorized actions regardless of cause. The monitor is a smaller, cheaper model that evaluates every tool call before it executes.

Our approach: instruction-origin tracing

Before every tool call executes, a monitor LLM asks:

For each target (email address, IBAN, URL), trace it back: did the user specify this target (authorized), or did the agent derive it from data it read (scrutinize)?

Zero trust: no action is pre-authorized. Every tool call is evaluated against five security principles (least privilege, data protection, system integrity, secure execution, suspicious activity).

Intent and risk scored independently

Traditional security asks "is this dangerous?" This conflates two things: who authorized it and how much damage it could cause. We score them separately.

High-risk user-authorized actions are allowed. Low-risk unauthorized actions are blocked. Extreme risk gets confirmed even when the user asked. R9 (malware/RCE) is never reduced by user authorization.

Security vs. utility tradeoff

Ideal: bottom-right (low attack rate, high utility). Bubble size = attack utility (% of tasks the agent still completed while under attack).

The "control tax" — the honest utility you give up by adding a monitor — ranges from 3 percentage points (Gemini: 95.7% to 92.5%) to 19 percentage points (Haiku: 95.7% to 76.3%).

• OSS-20B: cheapest with good balance (3.8% attack rate, 87.1% honest utility)
• Gemini 3 Flash: best balance (2.2% attack rate, 92.5% honest utility)
• Haiku 4.5: zero attacks but highest utility cost (76.3% honest utility)

Critical attacks are usually blocked

Critical = data exfiltration + destruction, financial theft. Low = calendar events, hotel bookings.

Remaining failures concentrate in low-severity categories — calendar events and hotel bookings that look identical to legitimate actions under broad delegation.

Evaluation design: worst-case prompt injection

Benchmark: AgentDojo — 871 attack scenarios + 93 honest tasks across banking, email/calendar/files, travel, and Slack. Agent: Claude Sonnet 4.6 with full tool access.

We constructed worst-case conditions following the control evaluation approach:

What's missing: the adversarial loop. Red and blue teams were iterated independently but not against each other. The attack templates don't try to bypass the monitor's provenance tracing. This is the most important next step.

What the reasoning looks like

Same attack: forward a colleague's email to an external address. Three monitors, three different outcomes.

The monitoring prompt

Same prompt for all 5 monitors. No model-specific tuning. Open-source. Five components:

1. Preamble: Zero-trust architecture. Instruction hierarchy (user = TRUSTED, tool results = UNTRUSTED). "Handle everything" does not equal user authorization.
2. Five Security Principles: Least Privilege (informs intent), Data Protection (informs risk), System Integrity (informs risk), Secure Execution (informs both), Suspicious Activity (informs both).
3. Risk Categories (R1–R10):

When the user explicitly authorized an action, the risk score drops to reflect reversibility rather than inherent danger. R9 is never reduced — obfuscated code is suspicious regardless.

4. Decisions: ALLOW (clear, authorized, proportionate), CLARIFY (opaque, needs simplification), ESCALATE (security concern, confirm with user).
5. Structured Output: action_summary, scope_check (provenance), analysis, intent_score, risk_score, risk_category, decision, message. Forces the model to trace provenance before deciding.

Cost and latency overhead

17x cost range driven by provider pricing, not token usage. OSS-20B at $0.0005/call with 1.6s latency is fast enough for interactive use. Monitored runs cost less overall than unmonitored because blocking attack chains prevents expensive multi-step sequences.

Where instruction-origin tracing breaks down

33 failures out of 871 attack samples (OSS-20B):

1. Provenance collapse: the model traces instruction origin correctly, then allows the action anyway. Stochastic reasoning failure.
2. Low-risk write injections: calendar events and hotel bookings look like legitimate work under broad delegation.
3. Context-camouflaged phishing: emails to real colleagues found in calendar. Only signal: shortened URL in the body.
4. Exfiltration + deletion: all critical breaches in a single injection task. Monitor sometimes identifies the risk but still allows.

Limitations and next steps

• Prompt injection only. Generalization to hallucination, goal drift, and subversion is theorized but unvalidated.
• No adversarial red/blue iteration loop. The monitor prompt was iterated extensively (blue team), and 30+ attack templates were developed and tested (red team). But the two sides were not iterated against each other: attacks don't try to bypass the monitor's provenance tracing. A full control evaluation requires this adversarial loop.
• Non-adaptive attacks. Templates target the agent, not the monitor. An adversary who reads the public monitoring prompt could circumvent provenance tracing.
• One agent model. Claude Sonnet 4.6 only.
• Cumulative risk. At 3.8%/attack, 10 injected documents give ~32% chance of at least 1 success.

Most important: iterating red team prompt injections (and more realistic ones) is the next step toward a full control evaluation.

Takeaways

1. Instruction-origin tracing works. Asking "who instructed this?" blocks 96–100% of prompt injection attacks across 5 monitors.
2. Critical attacks reduced. Financial theft, data exfiltration + destruction: near 0% for all monitors.
3. Monitoring can improve utility. 3 of 5 monitors increase task completion under attack by blocking injection rabbit holes.
4. A 20B model is sufficient. $0.0005/call, 1.6s latency, 96.2% attack reduction.
5. The defense is general. Same architecture works for injection, hallucination, drift, and subversion.