Two of the three people who share the Turing Award for deep learning now spend much of their time warning about what they built. They are not warning about the same thing, and the difference between their diagnoses is the most useful thing I’ve found for locating my own position, not because the two are opposed, but because they operate at different layers, and the gap between those layers is where I work.
I want to be careful here. It’s easy to conscript famous people into an argument you were going to make anyway. So I’ll state each position as I understand it, say where I think each is strongest, and then say where my own work sits, including where it sits badly.
Hinton: the problem is the incentive gradient
Hinton’s public argument, as I read it, is fundamentally about economics rather than architecture. Companies will accept some level of harm if they believe the benefits or profits justify it, that’s a description of how firms behave under competition, not an accusation of malice. And his prescription is proportional: labs should devote something like a third of their computing power to safety research (https://www.cbsnews.com/news/godfather-of-ai-geoffrey-hinton-ai-warning/), against the much smaller fraction they currently do.
There’s a detail in that CBS interview worth more than the headline number. CBS put the question to the labs directly, asking how much of their compute goes to safety research. None of them gave a number. A quantity nobody will state is a quantity nobody wants compared, and the refusal tells you more about the gradient than the estimate does.
What makes this argument strong is that it doesn’t depend on any particular technical claim about how models fail. You can disagree entirely about mesa-optimisation or deceptive alignment and still accept that a racing dynamic between well-capitalised firms produces systematically less caution than the situation warrants. It’s an argument about structure, and structural arguments survive being wrong about mechanism.
What makes it hard to act on is that almost nobody in a position to act on it can act on it unilaterally. A single lab that reallocates a third of its compute to safety loses ground to the ones that don’t. That’s the standard shape of a collective action problem, and it’s why the interesting responses are coordination-flavoured (standards, disclosure regimes, international agreements) rather than technical.
The clearest recent illustration is the Pentagon dispute, which is also the answer to anyone who thinks labs hold no levers. They hold several: what gets trained at all, what inference costs, whether weights are released, and whether a contract is signed. Anthropic used the last one, declining a Department of War agreement (https://www.congress.gov/crs-product/IN12669) rather than drop contractual language barring the use of its models with autonomous weapons. The administration responded by directing agencies away from it, and OpenAI signed (https://openai.com/index/our-agreement-with-the-department-of-war/) within days.
That is lab-level leverage exercised and then neutralised inside a week, which is what a collective action problem looks like from the outside rather than in the abstract. But the detail I keep returning to is narrower. OpenAI’s agreement reportedly carries similar red lines, no domestic mass surveillance, human responsibility for the use of force, tied to existing law and Pentagon policy rather than to contract. A restriction the counterparty can change unilaterally is a different object from a restriction that binds, even when the two read identically on the page.
That distinction is the subject of this post, showing up at the contractual layer instead of the technical one.
Bengio: build the non-agentic thing instead
Bengio’s response is different in kind. In June 2025 he launched LawZero, a nonprofit incubated at Mila, the Quebec AI Institute he founded, and backed by donors including the Future of Life Institute, Jaan Tallinn, Open Philanthropy and Schmidt Sciences. Its principal research direction is “Scientist AI”, which he describes as “a practical, effective and more secure alternative to today’s uncontrolled agentic AI systems” (https://lawzero.org/en/news/yoshua-bengio-launches-lawzero-new-nonprofit-advancing-safe-design-ai). It is deliberately non-agentic: designed to understand the world rather than act within it, giving truthful answers from transparent reasoning.
The technical write-up (https://arxiv.org/abs/2502.15657) is more specific than the announcement, a world model that generates theories to explain data, paired with a question-answering inference machine, both carrying an explicit representation of uncertainty so the system can decline to be confident. The stated aims are to accelerate scientific discovery, to provide oversight of agentic systems, and to improve our understanding of the risks themselves.
His diagnosis is what makes the design choice legible. Bengio says frontier systems are already showing “signs of self-preservation and deceptive behaviours”, and that this “will only accelerate as their capabilities and degree of agency increase”. Read that way, Scientist AI is not a hedge. It is an argument that agency is the thing carrying the risk, so you build the useful part without it.
The move I find most interesting isn’t the safety argument, it’s the product argument I read as being implied by it, my inference from the research direction, not a claim he makes in these terms. Bengio is claiming that a large part of the value everyone is chasing with agents can be obtained without agency, that if what you want is understanding and prediction, goals and autonomy are an unnecessary and dangerous addition. That’s a bet about where the value actually is, and it’s falsifiable in the market.
If he’s right, the incentive gradient Hinton describes is less immovable than it looks: it points at agents partly because we’ve assumed agency is required for the returns. If he’s wrong (if agency is where the compounding economic value genuinely lives) then non-agentic systems become a specialised safety instrument rather than an alternative path, and we’re back to Hinton’s coordination problem.
Where that leaves me
I deploy agentic systems in production. That’s my job, and it means both of these arguments land on me directly.
Against Hinton’s frame, I’m a small local instance of exactly the dynamic he describes. Nobody in my organisation is being reckless; we are being competitive, which is the point. General agentic systems are economically compelling, so we deploy them, and then we do a great deal of careful engineering to constrain them. The engineering is real. It also, structurally, is what accepting a deployment on economic grounds and then managing the residual risk looks like.
Against Bengio’s frame, I’m evidence for the harder version of his bet. Much of what my agents do genuinely does require action in the world, calling tools, changing state, executing multi-step tasks that change things people depend on. A purely predictive system would not do that job. Which is either a demonstration that the agentic path is necessary for a wide class of real value, or a demonstration that I’ve never seriously tried the alternative because agents are what the ecosystem hands you.
I genuinely don’t know which. I notice I’ve never tested it, never taken a workflow currently done by an agent and asked how much of its value survives if the system only predicts and a human acts. That’s a gap worth closing, and it happens to be a cheap experiment.
The third place to act
Both prescriptions assume you can decide what a frontier lab builds. Almost nobody can. But that is not the same as having no lever, and the framing that treats it as the same thing is doing quiet damage to a field that needs far more people in it than it currently has.
The deployment layer is a place to act, and it is where most of us already stand. Every organisation putting these systems in front of users has a point where somebody decides a system may proceed: which tools it can call, how much network it can reach, what happens when it fails, and whether anyone can reconstruct afterwards what it was actually configured to do. Those decisions are made by engineers and platform teams, not by lab executives, and there are thousands of us making them. A control that ships as a default inside a platform people already use becomes the norm. The same control shipped as guidance is an argument you re-win every quarter.
None of that substitutes for the coordination problem or the architectural bet. It sits underneath both of them, it is where the two diagnoses actually meet the world, and at the moment nobody owns it.