A human veto is not a control if the action finishes before the human sees it.
That is an important lesson behind a reported coding-agent failure. The useful question is not whether the model was “rogue”. It is why an automated path had enough access to perform destructive database operations.
The question grows harder when software controls sensors and actuators. The World Economic Forum's physical-AI analysis describes systems in which a digital decision can produce a real-world effect.
And stronger account protection does not settle the issue. A well-protected account can still open an overpowered session.
These signals belong together because they expose the same design limit. Authentication establishes who or what presents an action. It does not decide how much consequence that action may create before an independent control intervenes.
My judgement is simple: authority should narrow as consequence grows. The same automated identity should not be able to damage a production asset and its path to recovery.
This week's Big Read, The consequence envelope for agentic and physical AI, develops the operating model: bound scope, cumulative magnitude, irreversibility, observation, interruption and recovery before granting autonomy.
| Context update | ||
What is the maximum consequence this system can create before an independent control stops it? | ||
| ||
| ||
|
In Focus
The consequence envelope
The reported coding-agent example makes the practical distinction clear. The consequence envelope defines how far one agent-enabled workflow may act before another authority must decide.
Start with scope. Name the systems, objects and environments the workflow can reach. Enforce those boundaries at the receiving API, database, cloud service or controller. A restriction held only in the orchestration layer is one component away from failure.
Then set magnitude. Limit both one action and the cumulative effect of retries, loops, delegated agents and parallel workers. Ten permitted changes can become ten thousand if each worker receives a new allowance.
Treat irreversibility separately. Bulk deletion, production deployment, safety-limit changes and access to recovery material should require fresh authority outside the context that proposed them.
Keep observation independent. The workflow should not be the sole source of its own history. Preserve the initiating identity, task, tool call, receiver decision, state change and cumulative-limit position under an authority the agent cannot alter.
Measure interruption. Add detection time, decision time, revocation time and the time for in-flight actions to cease. If that total exceeds the time needed to cross the unacceptable-consequence threshold, the control arrives too late.
Finally, separate recovery. Backups, restoration orchestration, evidence and emergency stops should sit outside the production identity and, where consequence demands it, outside the same management plane.
Choose one production workflow and tabletop a plausible wrong action. Do not ask only whether the model is likely to make the mistake. Ask what the architecture would allow it to amplify.
Explore the full argument in this week's Big Read.
Retrospective
Least privilege, separation of duties, rate limits, dual control and resilient recovery were not invented for AI. They exist because legitimate identities and useful software can still produce the wrong result.
Agents change the required speed of those controls. They can join several permitted actions into one consequence, retry without fatigue and distribute work across multiple processes.
My concern is autonomy granted before the organisation has named the maximum loss it will tolerate from one action or sequence.
We do not need to predict every model error. We need to decide which errors the surrounding system will refuse to amplify, then test that refusal under realistic timing and failure conditions.
Poll of the week
Does your highest-impact AI agent have a technically enforced consequence limit?
Opinion
Autonomy is useful when it removes unnecessary waiting. It becomes dangerous when the same design also removes every point at which consequence can be bounded.
A human-in-the-loop label is not enough in practice. The control must name the action, the decision point and the time available before the result becomes unacceptable.
The organisation does not have to make its agents incapable of error. It has to make a small set of unacceptable consequences structurally unavailable. Grant the autonomy only after you have bounded the consequence.
Pick one production agent and ask: can it change both the live asset and the last state from which you would recover?
The authority-to-consequence matrix turns that question into a short review and wrong-action tabletop.
Reply and tell me where the independent stop sits.
