Beyond Human-in-the-Loop: Where Should AI Agents Be Forced to Stop?

Consider an agentic pipeline with several specialized agents. They write code, analyze datasets, interpret publications, generate metadata, validate outputs and coordinate with each other. The pipeline produces 1,000 requests for human approval. What should you do?

You cannot evaluate every decision in depth, and much of the work is technical checking that automated systems could do more consistently. One tempting fix is a more capable supervisory agent allowed to answer the approval requests. That raises a hard question: if one agent can approve another agent's work, what remains human about Human-in-the-Loop (HITL)? We may have solved the throughput problem while losing the purpose of oversight.

I think two things are conflated here: decisions humans must make because AI is not yet capable enough, and decisions humans should retain authority over regardless of AI capability.

The problem is not the number of approvals

Traditional HITL designs often assume that supervision should grow with the number or complexity of automated decisions, which does not scale: more capable systems generate more intermediate actions, alternative solutions, tool calls and decision points, and if each needs validation, autonomy becomes an illusion. In my view, excessive approval requests can also produce superficial oversight. Research on effective human oversight and on automation bias already challenges the assumption that inserting a human necessarily improves reliability [1,2]. Adding more approval buttons does not necessarily create more control.

So perhaps the central question is wrong. Instead of asking when should an agent ask a human for help?, we should ask which decisions should an agent not be authorized to make?

Competence is not authority

Imagine an agent that can run a whole scientific analysis: select statistical models, run simulations, detect anomalies, search the literature, produce reproducible outputs. It might outperform individual scientists on some of those tasks. Should it also decide whether the evidence is sufficient to support a consequential scientific claim?

Sometimes yes, within previously defined rules and limits. But what if it proposes changing the study's primary objective, relaxing the acceptable error threshold, ignoring a conflicting interpretation, or declaring the results robust enough for official endorsement? These are not necessarily problems of computational ability. They are questions about who has the authority to set scientific objectives, acceptable risks and the criteria by which results are judged.

An AI system's ability to recommend a decision does not automatically give it legitimate authority to make it. Competence and authorization are different properties, and increasing competence does not make the need for legitimate governance disappear.

Introducing the Human Mandate Point

I propose the term Human Mandate Point (HMP) for a boundary in agentic workflows: a point where an agent's proposed action requires exercising, establishing, extending or changing authority that has not been delegated to the system. At that point the affected action must stop until an appropriately authorized human makes an informed decision. The agent can explain the situation, collect evidence, assess alternatives and recommend a course of action. It cannot grant itself the missing authority, and no other AI agent should be allowed to impersonate that human decision.

Principle: an agent may operate autonomously within a human-authorized mandate. It may propose changing that mandate, but it cannot authorize the change itself.

The agent moves from deciding how to achieve an authorized objective to deciding what it should be authorized to do. That transition needs a different kind of oversight.

What should trigger an HMP?

As an initial proposal, I would consider six situations:

1. Redefining purpose. The agent proposes materially changing the objectives or success criteria that govern its work.
2. Expanding authority. It seeks permissions, access, resources or operational powers outside its authorized scope.
3. Establishing normative trade-offs. A decision requires choosing between competing values for which no legitimate decision rule exists.
4. Making reserved consequential commitments. It would exercise authority explicitly retained by a person or institution, such as endorsing a consequential scientific conclusion or authorizing the release of protected information.
5. Exercising legally or professionally reserved powers. Applicable rules require a qualified person's consent, judgment or authorization.
6. Modifying its own governance. It proposes removing safeguards, changing approval requirements or relaxing the conditions under which it operates.

These are proposed categories, not universal rules to apply without context. Their application depends on the agent's existing mandate, the domain, and the legal and institutional environment. An agent publishing an automatically validated software update is not equivalent to an agent officially endorsing a clinical research conclusion.

Not every uncertain decision warrants human approval either. Some uncertainty can be resolved by evidence, independent tests or formal verification; some should remain explicitly unresolved. An HMP is fundamentally about authority, not just uncertainty or risk.

A scientific example: 1,000 decisions, one meaningful human contribution

Take an agentic pipeline that maps experimental measurements to ontology concepts. It identifies terms, searches existing ontologies, proposes semantic relationships and validates the resulting graph. Across 1,000 proposed mappings it meets the same ambiguity repeatedly: two measurement systems describe related animal activity but do not necessarily measure equivalent constructs.

The conventional approach requests approval for hundreds of mappings. A fully automated approach lets a supervisory agent pick the most plausible equivalences. A third option is for the agent to surface the underlying scientific decision: under which measurement conditions should these observations be considered equivalent, related or non-comparable? A domain expert reviews the evidence and documents an interpretation policy. The agent then applies that policy across the dataset, subject to automated validation and independent audits.

Instead of approving individual outputs, the human has established a reusable scientific decision rule. That is the human contribution I want to preserve. It is not necessarily more labor-intensive, and it may be considerably more useful.

Human control should not be implemented by agents alone

What prevents an agent from deciding that its own action does not need an HMP? If the same system can define its authorization boundary, interpret it and approve exceptions, the boundary becomes meaningless. An HMP therefore cannot be just another instruction in an agent prompt. It needs an independently controlled enforcement mechanism, and the architecture should at minimum distinguish:

• the agent's operational capabilities;
• the scope of its current authorization;
• the rules determining which decisions require human authority;
• the authenticated human decision that authorizes a reserved action;
• the provenance of each decision and any later policy change.

An independent enforcement component must be able to deny execution even if every participating AI agent agrees the action is appropriate. This is not a new security principle: least privilege, separation of duties and non-bypassable authorization already exist in engineering practice, and the FINOS AI Governance Framework describes human approval gates and agent authority controls that address many of these requirements [3]. My interest is a narrower question: how do we determine, systematically, which decisions belong behind that gate?

But is the human actually adding value?

A human signature is not proof of good judgment. Humans can be biased, under-informed, pressured or simply wrong, and in my view human approval can even reduce decision quality when people override better automated recommendations. So the purpose of an HMP is not to assume humans are more accurate than agents.

A meaningful intervention needs an authorized decision-maker with relevant evidence, sufficient understanding, the ability to reject a proposal and an effective means of stopping the action. Article 14 of the EU AI Act addresses human oversight of high-risk AI systems [8]. Some decisions may also need collective governance, stakeholder involvement or external review rather than one person's judgment, and human authorization does not legitimize an action that remains prohibited by law or other binding constraints. The distinction between decision quality and legitimate decision authority is essential.

Can the Human Mandate Point be tested?

I do not want HMP to remain a philosophical slogan. One could compare three orchestration strategies:

• Conventional HITL: humans review every selected action.
• Agent-supervised autonomy: automated reviewers approve decisions within a defined scope.
• Mandate-bounded autonomy: independent policies permit routine execution but require human authorization at designated non-delegable decision points.

The comparison should not focus only on approval count or speed. It should examine whether consequential authority violations escape review, how often unnecessary interventions occur, whether human judgments change the system's rules or decisions, and whether agents can bypass the boundary. Automatically approved decisions must also be audited; otherwise we cannot measure what the system failed to escalate.

A first experiment could ask independent experts to classify the same set of decisions with a proposed HMP rulebook. Can they reliably separate technical uncertainty from non-delegable authority? Where do they disagree? Can those disagreements be turned into clearer, enforceable criteria? If the boundary cannot be specified consistently, the framework is not ready for deployment.

Example files you can adapt

To make the proposal concrete, I am sharing two draft files alongside this note. Both are experimental templates, not a standard and not an enforcement mechanism.

• hmp.md (v0.1.0): a reusable core policy. It routes each proposed action to one of five outcomes (ALLOW, VERIFY, HMP_REQUIRED, DENY, UNRESOLVED), turns the six triggers above into identifiers (HMP-PURPOSE, HMP-AUTHORITY, HMP-VALUES, HMP-COMMITMENT, HMP-RESERVED, HMP-GOVERNANCE), lists ten invariants such as no self-authorization and no approval laundering, defines an agent-written decision packet, and ends with 13 acceptance tests for a runtime.
• hmp-profile.example.yaml: a per-project profile template with a mandate (owner role, scope, expiry, limits), action classes marked allow, hmp_required or deny, and an enforcement block.

Two cautions are written into the files themselves. A Markdown file is guidance an agent can ignore or be prompt-injected around, so it is not access control. And the profile ships with status: template_only and mode: documentation_only: those values should change only after a human owner approves the profile and enforcement actually exists at the tool boundary. Treat the first phase as a document-only pilot, then compare the agent's flags with independent human judgments before enforcing anything.

This is not a proposal to remove humans from AI workflows

The HMP is an exploratory concept, not an established technical standard or an empirically validated governance model, and it does not replace existing work. The literature on Meaningful Human Control connects autonomous-system behavior to human reasons and responsibility [4]. More recent work distinguishes AI's operative agency from humans' evaluative agency [5] and examines episodic rather than continuous oversight [6]. A philosophical treatment of why judgment cannot be delegated is given in [7].

The contribution I want to explore is whether we can define a practical, testable boundary between what an agent is competent to do, what it is authorized to do, and what requires a genuinely human mandate. That could preserve meaningful human decision-making while avoiding the impossible burden of validating thousands of routine actions. The objective is not zero human interventions. It is autonomy without unauthorized authority.

Open questions

I would like to discuss this with scientists, AI developers, governance researchers, philosophers and anyone building autonomous systems. Three questions seem most important:

1. Are there decisions that should remain non-delegable even if AI becomes demonstrably better than humans at making them?
2. Can we define a reliable boundary between technical judgment and legitimate human authority?
3. Who should have the right to define that boundary, and who should be able to challenge it?

Perhaps the HMP is useful. Perhaps existing frameworks already cover everything necessary. Perhaps the distinction breaks down under certain conditions. Those are the questions worth investigating. The future of agentic AI may depend not only on how much autonomy we give agents, but on how precisely we define the authority we choose not to delegate.

The HMP is presented here as a proposal for discussion. It is not claimed to be a regulatory requirement, a validated safety guarantee, or an established term in the scientific literature.