Autonomy limits & human control

The state of it

One of 6 topics within Agent orchestration.

21 documents from 17 organizations address how far a system may act alone. The most usable framework in all of this research sits here, and it comes from a frontier lab's own security team rather than from a consultancy.

Anthropic's four questions - what untrusted content does it ingest, what actions can it take and under whose identity, what is the blast radius if it is misaligned, and what observability do you have - are answerable in an afternoon and produce a defensible decision. The principle underneath them is the useful part: the job is not zero risk but making risk legible and bounded, so exposure is accepted deliberately at a known scope.

Where the material is weakest is the middle. Almost everything here describes either full human approval or full autonomy; very little describes how to move between them as evidence accumulates.

The issues, by agreement

How many independent organizations name each issue as a problem. An issue is only as real as the number of separate publishers that identify it, so the count is the ranking. Bars are organizations, not documents. Where the count reads ours, no publisher here states the issue and the analysis is our own.

The issues in full

Each issue carries the organizations that name it, the numbers behind it, and the remedies proposed - with the concrete steps under each. Every citation points at a section of a named document, so any count here can be checked.

Issue 014 organizations name it2026 evidence

Risk to people rises with autonomy, and the trade is almost never stated

The more control is ceded to the system, the broader its action surface and the weaker the oversight. That relationship is documented, and it appears in almost no deployment decision.

The academic argument here is unusually blunt for this research. It holds that risks to people increase with the autonomy of a system, that increased autonomy brings broader action surfaces together with reduced oversight, and that even small miscalibrations of trust at a given level of autonomy multiply into much larger harm surfaces. Its recommendation is not prohibition but a spectrum: name the level, and accept the risk that goes with it deliberately. That is the same conclusion Anthropic reaches from operating experience, arrived at independently.

How to fix it — 1 approach, 3 steps

Write the autonomy level and the risk accepted with it

Record, per system, the autonomy level chosen and the specific harms that level makes possible. An unstated trade is an unmanaged one.

Done when Each production agent records its autonomy level on a fixed scale and the specific harms that level admits, including to people outside the organization, with a re-approval date within the last twelve months.

Name the autonomy level for each production agent using a fixed scale.0-30 daysRisk
Document the harms that level admits, including to people outside the organization.30-90 daysRisk
Re-approve the level annually rather than treating it as permanent.ongoingBoard Chair

The evidence — 6 documents

Organization	Document	Position
arXiv (research)Academic	Fully autonomous AI agents should not be developedOur reading Argues risks to people increase with the autonomy of a system - the more control a user cedes, the more risks arise - with safety, privacy and security risks growing together, and small miscalibrations of trust at a given level multiplying into larger harm surfaces.Risk characterisation across levels of autonomy	names it
CISAOther · April 2026	Careful adoption of agentic AI servicesOur reading Names both halves of the trade in one place: more independence means more the system can reach, and simultaneously less that an operator can see, because agents initiate their own follow-on work, spawn sub-agents and extend delegation chains in ways not surfaced to whoever is nominally supervising. The oversight does not merely fail to keep pace, it loses sight of what there is to oversee.Autonomy widens the surface and dims the light at the same time	names it
Cloud Security AllianceInstitution · July 2026	Defining non-human identityOur reading Gives the mechanism behind the curve: an autonomous identity executing thousands of operations a second removes the pauses in human work where a mistake would have been caught, so a misconfiguration compounds rather than surfacing.Why NHI risks differ from human identity risks	names it
IBMHyperscaler	Agentic AI risk and opportunityOur reading Names the autonomy trade directly - a system acting without approval at each step can reach unintended or harmful outcomes - and answers it with a staged framework that widens autonomy only as performance and quality measurements justify it, rather than setting a level once.Autonomous action	names it
AnthropicFrontier lab · July 2026	Anthropic’s CISO guide to agentic AIOur reading Reaches the same conclusion from operating practice: make agentic risk legible and bounded so exposure is deliberately accepted at a known scope.The principle of least agency	proposes a fix
arXiv (research)Academic	Fully autonomous AI agents should not be developedOur reading Recommends adopting an explicit spectrum of autonomy levels rather than a binary, so that trade-offs between benefit and risk are documented at each level.Adoption of a spectrum of AI agent autonomy	proposes a fix

Issue 022 organizations name it2026 evidence

Autonomy is being set once, as a binary, instead of earned incrementally

Systems are launched either fully supervised or fully autonomous, with no defined path between - so autonomy is granted by a meeting rather than by evidence.

How to fix it — 2 approaches, 5 steps

Define the ladder and the evidence for each rung

Write three or four levels between full review and full autonomy, and state the measured result that earns promotion to each.

Done when The rungs from propose-only to unsupervised are defined, each names the measured evidence that earns promotion and who signs it, and a threshold breach demotes automatically rather than opening a discussion.

Define the levels: propose only, act with approval, act with sampling, act unsupervised.0-30 daysCIO
State the measured evidence required to move up a rung, and who signs it off.0-30 daysRisk
Make demotion automatic on a threshold breach, not a discussion.30-90 daysRisk

Run the four questions before every approval

Untrusted input, actions and identity, blast radius, observability. Four answers produce a defensible decision in an afternoon.

Done when The four questions are the standing intake form for agentic use cases, and the answers are stored with the approval so the basis can be read back later.

Adopt the four questions as the standing intake form for any agentic use case.0-30 daysCISO
Record the answers with the approval so the basis is auditable later.0-30 daysRisk

The evidence — 5 documents

Organization	Document	Position
AnthropicFrontier lab · July 2026	Anthropic’s CISO guide to agentic AIOur reading Frames the task as making agentic risk legible and bounded so it can be deliberately accepted, with blast radius calculated as scope multiplied by severity, rather than as a single approval decision.Four questions; the principle of least agency	names it
IBMHyperscaler	Agentic AI risk and opportunityOur reading Describes agentic systems acting independently in ways whose inherent complexity produces unpredictable behaviour, and proposes goal-oriented guardrails that actively restrict and guide what the agent may do.Risks and key mitigations of autonomous action	names it
CISAOther · April 2026	Careful adoption of agentic AI servicesOur reading Sets out the incremental version six governments could agree on: begin on low-risk, non-sensitive work, widen access and independence only as the behaviour has actually been observed, and design every deployment so it can be reversed and contained from the outset rather than after the first incident. That last clause is what makes it a ratchet rather than a switch - reversibility has to be built while the system is small enough to change.Earn the autonomy, and build the way back in from the start	proposes a fix
McKinsey & CompanyConsultancy	McKinsey: Rethinking AI decision-makingOur reading Proposes measuring, monitoring and improving agents over time with underperforming ones retrained or retired, which implies a graduated rather than fixed level of trust.Governance and oversight of agents	proposes a fix
World Economic ForumInstitution · January 2025	The rise of AI agentsOur reading Sets out autonomy as a staged progression earned use case by use case - assistant, then recommendation, then automation, with the automation stage acting independently and referring only major exceptions - and illustrates it with a plant that ran the same agent as an adviser before letting any part of it act.The maturity of virtual AI agents	proposes a fix

Who is represented

This dossier is drawn from 18 organizations working on the subject, 7 of which are cited directly in the issues above.

Consultancy — 6

McKinsey & Company 3 Deloitte 1 EY 1 KPMG 1 PwC 1 UST 1

Institution — 4

Cloud Security Alliance 3 World Economic Forum 1 Association of Corporate Counsel 1 Citi 1

Academic — 1

arXiv (research) 2

Hyperscaler — 3

IBM 2 AWS 1 Google Cloud 1

Frontier lab — 1

Anthropic 1

Vendor — 2

LangChain 1 Sequoia Capital 1

Other — 1

CISA 1