AI’s Prisoner’s Dilemma
By Stephan Engelen · 2026-10-01
In 2018, during an executive program on AI and business at MIT, we spent quite some time discussing the possible impact of AI on society and business. One of the discussions that stayed with me was about warfare.
At the time, the question still felt relatively theoretical: what happens when AI is no longer only used to analyse information, but becomes part of systems that can increasingly assess situations, make decisions and act?
That discussion has come back to me recently.
Not because I want to make a political statement about one country, one administration or one company. What interests me more is the mechanism underneath it. Once AI becomes strategically important, whether in defence or in business, it creates pressure on everyone else to move as well.
That starts to look a lot like the prisoner’s dilemma.
Two parties may both prefer a situation in which there is restraint, but neither can be certain that the other party will show the same restraint. So both continue to invest and accelerate, because slowing down feels more dangerous than moving forward.
You can see how that logic could play out between countries. If one country believes another is developing more autonomous military systems, it becomes very difficult to say: we will deliberately move slower.
Not necessarily because anyone wants to remove human control, but because nobody wants to be in a position where the other side develops a capability they do not have.
The same dynamic exists in business, although obviously with very different stakes.
Many CEOs I speak with understand that AI adoption needs governance. They know that not every process should simply be automated and that human responsibility still matters. But at the same time they see competitors moving faster, reducing costs, changing operating models and experimenting with new ways of working.
Then the discussion changes.
The question is no longer only: “Are we ready?”
It becomes: “Can we afford not to move?”
That is where I think things become interesting.
Governance is easy to support when there is no pressure. It becomes much harder when governance starts to slow things down. Testing takes time. Approval takes time. Risk assessments take time. Human intervention creates friction.
And friction is exactly what organisations try to remove when speed becomes important.
Most of the time, this does not happen through one big decision. The boundaries shift gradually. One exception here, a little more autonomy there, one less approval step because the process is too slow.
Before long, the organisation may be in a very different place from where it originally intended to be.
For me, this is also why the distinction between instructions and constraints matters.
There is a big difference between telling an AI system that it should not do something and designing the system in such a way that it simply cannot do it.
As AI systems become more capable and more autonomous, that distinction becomes increasingly important. A system does not need to be conscious or malicious to find a route that its designers did not anticipate. If it is given an objective, sufficient access and enough autonomy, it may find another way to reach that objective.
That is not science fiction anymore. It is a design and governance question.
And that is also why I believe AI governance cannot remain a set of policies somewhere in the organisation. Policies matter, but they are not enough.
Governance increasingly has to become part of the technical and organisational architecture itself: clear permissions, clear ownership, auditability, defined intervention points and limits on autonomy.
The more I think about it, the less I see AI purely as a technology problem.
It is also a coordination problem.
We are building capabilities faster than organisations, regulators and governments are able to agree on how those capabilities should be used. And because every actor has incentives to move, slowing down becomes increasingly difficult, even if everyone understands the risks.
That may be the more uncomfortable part of the AI debate.
The biggest risk is perhaps not that AI suddenly becomes irrational.
It may be that all the human actors involved behave perfectly rationally from their own point of view, and together create an outcome that none of them actually wanted.
That is the prisoner’s dilemma.
And it does not only apply to governments or military technology.
Boards and leadership teams are already dealing with a smaller version of exactly the same question:
How do you move fast enough to remain competitive without gradually removing the safeguards you will later wish you had kept?
For me, that is where AI governance stops being primarily an IT topic.
It becomes a leadership question.
To assess where AI changes your business rather than simply adding noise, explore our AI and technology transformation advisory.
Twijfel je waar AI je business werkelijk verandert en welke keuzes daarbij horen? Lees over AI-strategie en de impact op je bedrijf bij SEMCOR.