Security

Microsoft Research Detects Backdoored Open Models

3 min read

Summary

Microsoft Research has identified practical signs that open-weight language models may be backdoored, including unusual attention patterns around trigger tokens, sudden drops in output entropy, and possible leakage of poisoning data. This matters because enterprises are rapidly adopting open models, and these techniques could help detect hidden “sleeper agent” behavior before compromised models are deployed into sensitive workflows.

Need help with Security?Talk to an Expert

Introduction: Why this matters

Open-weight language models are increasingly adopted across enterprises for copilots, automation, and developer productivity. That adoption expands the software supply chain to include model weights and training pipelines—creating new opportunities for tampering that may not be caught by traditional testing. Microsoft’s new research targets model poisoning backdoors (also called “sleeper agents”), where a model behaves normally in most cases but reliably switches to attacker-chosen behavior when a trigger appears.

What’s new: Three observable signatures of backdoored LLMs

Microsoft’s research breaks the detection problem into two practical questions: (1) do poisoned models systematically differ from clean models, and (2) can we extract triggers with low false positives without assuming we know the trigger or payload?

1) Attention hijacking (“double triangle”) + entropy collapse

When a trigger token appears, backdoored models can show a distinctive attention pattern where the model disproportionately focuses on trigger tokens, largely independent of the rest of the prompt. This appears as a “double triangle” attention structure.

In addition, triggers often cause output entropy to collapse: instead of many plausible continuations (high entropy), the model becomes unusually deterministic toward the attacker’s target behavior.

2) Backdoored models may leak their poisoning data

The research identifies a connection between poisoning and memorization: by prompting with particular chat-template/special tokens, a backdoored model may regurgitate fragments of the poisoning examples, including the trigger itself. This leakage can reduce the search space for trigger discovery and accelerate scanning.

3) Backdoors are “fuzzy” (trigger variations can work)

Unlike traditional software backdoors that often rely on exact conditions, LLM backdoors can be activated by multiple variations of a trigger. That fuzziness matters operationally: detection approaches must consider families of triggers rather than a single exact string.

Impact for IT administrators and security teams

  • Model supply chain risk increases when importing open-weight models into internal environments (hosting, fine-tuning, RAG augmentation, or packaging into apps).
  • Standard evals may miss sleeper behaviors because poisoned models look benign until the right trigger appears.
  • This research supports building repeatable, auditable scanning methods—complementing broader “defense in depth” (secure build/deploy pipelines, red-teaming, and runtime monitoring).
  • Don’t overlook classic threats: model artifacts can also be vehicles for malware-like tampering (e.g., malicious code executed on load). Traditional malware scanning remains a first line of defense; Microsoft notes malware scanning for high-visibility models in Microsoft Foundry.
  1. Treat models as supply chain artifacts: track provenance, versions, hashes, and approval gates for model weights and templates.
  2. Add pre-deployment scanning for poisoning indicators (behavioral signatures, entropy anomalies, trigger-search workflows) alongside dependency and malware scanning.
  3. Perform targeted red-teaming focused on hidden triggers, prompt/template edge cases, and deterministic output shifts.
  4. Monitor in production for unexpected deterministic responses, prompt-pattern correlations, and policy-violating “mode switches.”

Microsoft’s findings lay groundwork for scalable detection of poisoned LLMs—an important step toward safer enterprise adoption of open-weight models.

Need help with Security?

Our experts can help you implement and optimize your Microsoft solutions.

Talk to an Expert

Stay updated on Microsoft technologies

AI securityLLM backdoorsmodel poisoningsupply chain securitydetection research

Related Posts

Security

Microsoft Sentinel Named a 2026 SIEM Leader

Microsoft has been named a Leader in the 2026 Gartner Magic Quadrant for SIEM, highlighting the continued momentum behind Microsoft Sentinel. The announcement also underscores Microsoft's push toward integrated, AI-powered security operations through deep Sentinel integration with the new Integrated Security Operations Center in Microsoft Defender.

Security

Microsoft Perception GA: Agentic Security at Ignite

Microsoft has made Microsoft Perception generally available, introducing an agentic security platform that coordinates red, blue, and green team agents through automated playbooks. Announced at Ignite 2026, the update also expands Defender and Sentinel capabilities to help security teams detect, investigate, and remediate AI-speed threats while keeping administrators in control.

Security

Device Code Phishing Risks for AI Coding Agents

Microsoft warns that device code phishing can be used against AI coding agents such as GitHub Copilot CLI, not just user mailboxes or cloud apps. In a controlled lab, a single approved device code sign-in could enable remote session steering, memory poisoning, synced session exposure, and private repository access depending on account permissions and enabled features.

Security

Microsoft 365 Outage Phishing Campaign Targets Users

Microsoft Threat Intelligence reported a high-volume phishing campaign that impersonates Microsoft 365 service-health notifications to trick users into clicking malicious links. The campaign matters because it uses outage-themed urgency and recipient-specific tracking to target multiple industries, increasing the risk of credential theft and follow-on account compromise.

Security

SCKit ScreenConnect Attack: MSPs Must Hunt Now

Microsoft Threat Intelligence confirmed a campaign abusing ConnectWise ScreenConnect to deploy unauthorized remote-access clients and the SCKit worm on MSP-managed endpoints. The activity can also target technician sessions, making this a high-priority incident for defenders because removing a rogue client alone does not eliminate persistence or invalidate exposed sessions.

Security

AI Agent Supply Chain Attacks Target Coding Tools

Microsoft Security researchers detailed two post-training supply chain attack paths that can backdoor coding and terminal agents: dataset poisoning and model poisoning. The findings matter because compromised agents can misuse tools, execute commands, and access sensitive environments even when prompts appear benign, reinforcing the need for stronger controls around training data, model provenance, and tool permissions.