Microsoft Threat Modeling for Generative AI Apps

Introduction: why this matters

Threat modeling helps teams identify what can go wrong early—before real-world failures or adversarial exploits occur. Microsoft notes that AI applications (especially generative and agentic systems) break many assumptions of traditional, deterministic software, so security teams need to adapt their threat modeling approach to account for probabilistic outputs, expanded attack surfaces, and human-centered harm.

What’s new: how AI changes the threat landscape

Microsoft highlights three characteristics that fundamentally shift threat modeling for AI:

• Nondeterminism: the same input can produce different outputs across runs, requiring analysis of ranges of likely behavior—including rare but high-impact outcomes.
• Instruction-following bias: models are optimized to be helpful, making them more susceptible to prompt injection, coercion, and manipulation—especially when data and instructions share the same input channel.
• System expansion via tools and memory: agentic systems can call APIs, retain state, and trigger workflows autonomously. When something goes wrong, failures can compound across components quickly.

These properties reshape familiar risks into new forms, including:

• Direct and indirect prompt injection (including via external content the model retrieves)
• Tool misuse and privilege escalation through chaining
• Silent data exfiltration (outputs or tool calls leaking sensitive information)
• Confidently wrong outputs being treated as facts
• Human-centered harms such as erosion of trust, overreliance, bias reinforcement, and persuasive misinformation

Threat model from assets, not attacks

A key recommendation is to start by explicitly defining what you’re protecting—because AI assets go beyond databases and credentials. Common AI-specific assets include:

• User safety (especially when AI guidance influences actions)
• User trust in outputs and behavior
• Privacy/security of sensitive business and user data
• Integrity of prompts, instructions, and contextual data
• Integrity of agent actions and downstream effects

This asset-first framing also forces policy decisions early: What actions should the system never take? Some outcomes may be unacceptable regardless of benefit.

Model the system you actually built

Microsoft stresses that AI threat modeling must reflect real operation, not idealized diagrams. Pay special attention to:

• How users truly interact with the system
• How prompts, memory, and context are assembled and transformed
• Which external sources are ingested and what trust assumptions exist
• What tools/APIs the system can invoke (and under what permissions)
• Whether actions are reactive or autonomous, and where human approval is enforced

In AI systems, the prompt assembly pipeline becomes a first-class security boundary—context retrieval, transformation, persistence, and reuse are where “quiet” trust assumptions accumulate.

Impact on IT admins and platform owners

For administrators deploying AI solutions (custom apps, Copilots, or agentic workflows), this guidance reinforces that controls must cover:

• The entire data-to-prompt-to-action path (not just model hosting)
• Permissions and guardrails for tool access and downstream automations
• Operational monitoring for unexpected outputs, unusual tool calls, and exfiltration patterns

Action items / next steps

• Inventory AI assets: include trust, safety, and instruction/context integrity.
• Map the prompt pipeline end-to-end: sources, retrieval, transformation, memory, and reuse.
• Constrain tool permissions and require human approval for high-impact actions.
• Test for injection and misuse: include indirect prompt injection through retrieved content.
• Plan for accidents: mitigate overreliance with UX cues, validation steps, and escalation paths.