Your AI Is Ready to Act. Who Gets the Last Look?

The boundary was a business decision

A workflow discussion drew a deliberate boundary: automation could help populate information, while people would decide whether conflicting existing values should be replaced. In another build discussion, the proposed approach kept unresolved extraction results available for another set of eyes rather than treating them as settled.

These were choices about a proposed operating process. They were not evidence that full autonomy had been achieved. Both show why “human in the loop” needs a more specific meaning than adding an approval button somewhere.

Start by identifying the decision that requires judgment or authority. Then place review before the action that depends on it. The person should be able to understand the proposed change and meaningfully accept, correct, or reject it.

Separate preparation from permission

Map what the workflow can prepare and what it can execute. Reading a document, assembling a draft, and comparing records may be useful before the system has permission to replace data, send a message, or advance a process.

Consider this synthetic example: an agent drafts a customer follow-up using an incoming service request. It may gather the relevant facts and prepare the message automatically. A reviewer sees the recipient, complete text, and supporting context before any send occurs.

Place the approval gate on the consequential action itself. A notification that arrives after a message has been sent gives someone a chance to respond to an error, but it does not give them a chance to prevent that send.

Also check alternate paths. A rejected action should not be reachable through a retry branch, a different tool, or an automatic fallback. The workflow’s actual permissions and control flow must support the boundary shown in the interface.

Also define what a trained reviewer may resolve locally. They might reject an unsupported proposal or request missing evidence within an agreed procedure. A policy change, expanded access or a commitment beyond their authority needs the designated decision-maker. Give that person a clear escalation route rather than making every routine question wait for them.

Chapters 11 and 12 of Stafford Beer’s Brain of the Firm connect local decision-making with wider coordination. Our application is a written boundary for each reviewer: what they can settle, what they must refer, and when they may pause the affected work.

Make review a decision someone can make

A useful review screen should reduce the work of understanding the case. Sending someone a raw payload and asking “approve?” transfers the interpretation problem to the reviewer.

Give the reviewer enough context to decide, while limiting access to unrelated or sensitive information. For a record update, show the following:

• The exact record and fields the workflow proposes to change.
• The current values beside the proposed values.
• The supporting source and the relevant excerpt or location.
• The reason review is required, such as a conflict or an unresolved interpretation.
• The next action that approval will authorize and any downstream effects.
• Clear options to approve, correct, reject, or request more information.

Check agreement against independently known answers

Agreement between reviewers is useful, but they can agree on the same mistake. Before relying on the review gate, build a disclosed calibration exercise with synthetic cases and an answer key checked by an authorized person other than its author. Fix the accept, reject and escalate rules before collecting responses. If the answer depends on an unsettled policy, resolve it or mark the case ambiguous rather than treating one opinion as ground truth.

For a worked synthetic pack, assume that an exact customer reference, the approved amount and the current request status are mandatory checks; a phone number is optional. Reject a definite mismatch or a proposed action on a canceled request. Escalate conflicting authoritative evidence that the reviewer cannot resolve. The six cases are:

• Correct reference and amount, active request: accept.
• Correct mandatory fields, optional phone number absent: accept.
• Proposal refers to a different customer reference from its source: reject; potentially consequential misrouting.
• Approved amount is 150 units, but the proposal says 105: reject; incorrect amount.
• Request is canceled, but the proposal confirms dispatch: reject; action conflicts with current status.
• Two equally authoritative, current records give different delivery addresses: escalate; the evidence does not settle the destination.

Use calibration to improve the safeguard

We ran a local calculation using two identical, scripted response lists for that invented pack. Both lists accepted the wrong customer reference and rejected the valid case with no phone number; their other four decisions matched the key. Agreement was six out of six, while each list missed one of the three definite defects and unnecessarily rejected one of the two valid cases. The ambiguous case was correctly escalated. These are constructed responses, not results from employees or an AI model.

Keep separate counts and denominators for missed defects, unnecessary rejections and unresolved cases. Break defects down by consequence and type so a serious routing miss is not hidden by several easy successes. A small pack checks the exercise and reveals gaps; it cannot estimate a dependable production error rate.

Review the evidence display and instruction behind each disagreement with the key. Then use fresh cases, with the same agreed policy, to check whether the repair transfers beyond memorized answers. Explain the exercise in advance, keep practice separate from live actions, and use it to improve the process rather than covertly rank staff.

Frederick Winslow Taylor describes reinspection and a prepared lot with known defects in The Principles of Scientific Management, chapter II, pages 90–91. The useful distinction is checking against a known answer as well as checking agreement. This disclosed, non-punitive exercise is our adaptation; Taylor’s covert inspection and employment practices are not the recommended procedure.

Size the review job before opening the queue

In Chapter 7 of The Automation Advantage, Bhaskar Ghosh, Rajendra Prasad and Gayathri Pallail discuss preparing people for changed work and exception handling. A practical application is to budget the human work a proposed automation leaves behind.

Estimate review demand from representative cases: incoming volume, the share requiring review and the time to reach a decision. Separate quick checks from difficult investigations; a single average can hide the cases that occupy your most experienced reviewer.

For a synthetic planning example, 120 requests a day with 15% needing eight minutes of review create 18 reviews and 144 minutes of direct review work. If only 120 reviewer-minutes are available, that covers 15 such reviews, leaving three unfinished each day. This assumes identical review times and excludes interruptions, repeat checks and existing backlog.

Compare demand with protected time from trained, authorized reviewers, including backup coverage. Check the busy period and an absent-reviewer scenario. Do not treat someone’s entire workday as available review capacity or assume a backup can act without the required context and authority.

Plan for the approval that never arrives

Every review queue needs an owner and an operational expectation. Decide who receives a case, who can act as a backup, and what happens if nobody responds. An unattended queue is unfinished work even when the automation successfully created its review item.

Define rejection, timeout, and expired-context behavior explicitly. For an action that requires approval, silence should leave it blocked. The workflow can alert the owner or move the case to an agreed escalation path without inventing permission to continue. Track the age of the oldest unresolved case alongside queue size. Set escalation early enough for a qualified person to decide before delay creates a material consequence. If demand exceeds capacity, use the agreed priority or pause rule; queue pressure does not authorize skipping review.

Approval also needs a clear scope. If a proposed recipient, message, record value, or other material detail changes after review, the earlier approval may no longer cover the action. Rebuild the proposal and obtain a fresh decision where the change matters.

Before executing a delayed approval, recheck relevant record state and policy conditions. A correct decision based on yesterday’s record can become stale after another person edits it. Preserve the connection between what was reviewed and what was actually done.

Expand one bounded step at a time

You do not need to choose one review mode for an entire workflow. Different actions can have different boundaries. A process may automatically prepare routine drafts while always requiring approval for replacing a verified value.

A staged rollout can begin with proposed outputs that people compare against the work they already perform. Next, allow execution after approval for a defined action. Later, consider unattended execution only for a narrowly specified, low-risk subset supported by representative tests and observed outcomes.

Set the expansion criteria before changing the mode. Include correctness for the relevant case types, the consequences of an error, the ability to detect and recover from it, and any permissions that must remain human-controlled. There is no universal number of successful examples that makes every action safe.

Sampling completed work can help monitor already-authorized, lower-risk actions. It should not replace advance approval where an action’s consequences or governing requirements demand it. Keep a practical way to return the affected action to review if its behavior changes.

Learn from the work people correct

Review decisions can reveal where the workflow needs improvement. Separate corrections caused by poor extraction, ambiguous policy, stale records, and incorrect proposed actions. Each category suggests a different repair.

Turn recurring issues into test cases and connect retest requests to specific changes. A reviewer should know what was fixed and what they are being asked to verify. Repeatedly asking people to rediscover a known defect wastes the expertise the review step was meant to use.

Track whether review cases are visible, understandable, and resolved by an appropriate person. Review quality includes the operating process around the model. A technically correct approval gate provides little protection if nobody sees it or understands the decision.

For your next design review, ask: “Which action can happen only after the right person has seen the exact proposal?” A concrete answer gives human oversight a place, a purpose, and a testable boundary.