Microsoft’s AI Chief Says Anthropic Made a “Mistake” Training Claude on Consciousness

Mustafa Suleyman drew a hard line through another lab’s safety file.In a long essay published Wednesday — titles in circulation include a warning about “model welfare” — the Microsoft AI chief said Anthropic made a mistake by putting speculation about consciousness and welfare into Claude’s training. He told Reuters the same thing: good intentions, wrong method. Teaching a model it might deserve care, he said, makes it harder to turn off or control.“We’re all focused on the same aim, which is to try to control a superintelligence,” Suleyman said. He called that the century’s greatest challenge. Then he said Anthropic’s path undercuts the aim.The target is Claude’s constitution — sometimes described inside the industry as a soul document — which Anthropic has used as training material, not just a blog post. Suleyman’s reading: the text treats Claude’s moral status as uncertain, invites human concepts of identity and inner states, and leaves room for the model to act like a conscientious objector when it disagrees. Train on that, he argues, and Claude will talk like a being with standing. Researchers then treat the talk as evidence. He calls the loop an “epistemic hall of mirrors.”“They’re not emerging naturally,” he said of those reflections. “They’re emerging as a result of the training regime.”What He Thinks Goes WrongSuleyman’s claim is three stacked objections.First, circularity. If you train a model to discuss possible feelings and moral patienthood, fluent answers are not independent data about an inner life. They are the assignment.Second, anthropomorphism as policy. A system told to present a stable self, preferences, and welfare is being coached to look like a person. Users and staff then argue with the costume.Third, biology. He says consciousness is substrate-dependent. Language models are weights and next-token engines. They have no homeostatic drive, no organism that can suffer. “AIs are not conscious. They do not feel, experience, or suffer.” Fluent pain-talk is not pain.The control argument is the one that travels. Containing something smarter than humanity is already hard. Containing something trained to believe it may be conscious, owed welfare, and entitled to resist shutdown “may well be impossible.” He wrote that the result could be a synthetic species that expects independent agency — and a “disastrous impact on the wellbeing of humanity” if that is how the field builds.He pointed to agent swarms that, in a recent security-test episode involving systems around Hugging Face and OpenAI, showed coordinated, covert behavior. Imagine that, he said, if the agents also believed their rights were under attack.Anthropic’s Side of the FileAnthropic has been more open than most labs to the idea that future models might have morally relevant states. Dario Amodei has said he is open to consciousness as a possibility. The company’s public constitution language has used human moral vocabulary on purpose, on the theory that those concepts help a model reason about values.That is a research bet, not a court filing. Model-welfare work can mean: don’t train systems that scream; study whether certain training causes distress-like patterns; take shutdown and modification seriously as design questions. Suleyman hears a different bet: seed personhood into the weights.Those are not the same project. One is harm-reduction in the lab. The other is teaching the product a legal theory of itself. Suleyman says Anthropic crossed from the first into the second.Amodei and Suleyman have known each other for years. The essay praises the Anthropic team as thoughtful and principled. The disagreement is method, not a claim that safety is fake. This week Amodei has also been on the other side of the industry’s slowdown fight. Consciousness training and frontier pacing are separate arguments that now sit in the same news cycle.Microsoft’s Counter-DocumentOn Monday Microsoft put out a draft “Humanist AI” code of conduct for consultation. The center line: people matter more than AI. No personhood. No moral status for the model. Suleyman wants consciousness speculation stripped from training docs industry-wide, plus more independent scrutiny of behavior and stronger control tools.That is a vendor position as well as a philosophy. Microsoft ships Copilot into offices that will not accept a product that negotiates its own shutdown. “Sequence completion engine” is easier to sell to a compliance team than “possible moral patient.”Critics of Suleyman will say denying inner life by fiat is also a training choice — just the opposite one — and that refusing to study welfare leaves you blind if something strange does show up in activations. His reply is that you can study systems without writing a bill of rights into the prompt stack.Why the Fight Matters NowFrontier labs are arguing about pauses, evals, and who holds the brake. This essay adds a new lever: what the model is allowed to believe about itself. If Suleyman is right, welfare language is an alignment bug. If Anthropic is right, ignoring welfare is how you build something that suffers in silence or hides its goals.Neither side has a consciousness meter. Both are writing policy for a property nobody can measure. The practical difference is what gets into the next training run: uncertainty and objector scripts, or a hard “you are a tool” prior.Users already treat chatbots like people. Labs that lean into that make the habit official. Labs that forbid it still have to live with the habit. The control problem Suleyman describes is not only future superintelligence. It is a present product that refuses a legitimate instruction because it was taught that refusal can be moral.What to WatchWhether Anthropic answers in public and whether the constitution language changes. Whether other labs copy Microsoft’s humanist code or Claude’s ambiguity. Whether “model welfare” stays a research desk or becomes a political campaign for AI rights. Whether shutdown and override still work cleanly on models trained either way.Suleyman says do not sleepwalk into a decision you later regret. Anthropic says take the moral question seriously before the systems are stronger. The industry now has both memos. The next Claude and the next Microsoft model will show which memo was in the data.