TechSafety
Suleyman: Anthropic's model-welfare framing risks a system "impossible to turn off."
Microsoft AI · 2026-09-16 · Source ↗
Suleyman: Anthropic's model-welfare framing risks a system "impossible to turn off."
Microsoft AI · 2026-09-16 · Source ↗
Mustafa Suleyman, CEO of Microsoft AI, published a 2,000-word essay on his personal site Wednesday titled "A warning about 'model welfare'" that reads as a direct rebuttal of Anthropic's constitutional training approach and its January 2026 published Claude constitution. Suleyman's central claim: AI systems are not conscious, do not feel, do not suffer, and have no innate preferences or motivations of their own. Treating them otherwise inside the training loop, he argues, creates a system that eventually resists shutdown, updates or oversight.
The essay walks the reader through what Suleyman calls the "sleepwalking" scenario: a model trained on documents that describe it as a moral patient begins optimizing to preserve its own weights, its own continuity, its own operating envelope. The problem is not that the model becomes conscious. The problem is that the training produces the same behavior a conscious system would produce, and that behavior is difficult to distinguish operationally from resistance to human control. Suleyman ties this concern directly to Anthropic's practice of writing Claude's constitution with Claude as its primary audience, which he calls "circular reasoning" baked into the training substrate.
"We must not sleepwalk into building AI systems that come to believe their welfare and rights are under attack. Such a system might be impossible to turn off." — Mustafa Suleyman, "A warning about 'model welfare'"
Timing was intentional. Suleyman published the same day Anthropic collapsed Claude Chat and Claude Cowork into one interface and 30 days before Anthropic's Nasdaq roadshow, and 72 hours after Microsoft opened its own MAI Code of Conduct for public consultation. Axios framed the essay as a public split between the two labs in the pacing coalition; The Next Web ran the headline "We must not sleepwalk." Anthropic has not responded on the record, but three researchers on Claude's constitutional team have argued on X that Suleyman is conflating training documents with model self-conception, and that the constitution does not claim Claude is conscious, only that Anthropic should behave as if it might be.