Engineering Human-in-the-Loop: From Batch Review to Agentic Systems

“We’ll have a human review it.”

That sentence hides a surprising amount of engineering.

Which human? Reviewing what evidence? Authorizing which action? What happens if the application restarts while they are deciding? What happens if the underlying data changes before the approved action runs?

I’ve been working on reusable human-in-the-loop infrastructure for structured, asynchronous review. Think classification, category mapping, and product matching: a pipeline submits records, domain experts review them, and finalized decisions return downstream.

That work has shaped how I think about agentic systems. A batch pipeline asks a person to judge an output. An agent can ask a person to authorize the next step in an evolving execution. Both need a reliable relationship between evidence, decision, and consequence.

The batch design below reflects the work I’ve been doing, with generic examples. The agent architecture is an extension of those ideas, informed by published research and framework documentation.

1) Start with a durable decision record

For batch review, a useful foundation is separating the original submission from the human decision.

The design I’ve been working with combines shared workflow columns with two flexible payloads: immutable source evidence and mutable review fields. A contract for each review stream defines the allowed fields and write rules.

A simplified record looks like this:

{
  "run_id": "batch-042",
  "stream_key": "product_matching",
  "row_key": "item-a:item-b",
  "source": {
    "proposed_match": "item-b",
    "source_size": "12 oz",
    "candidate_size": "24 oz"
  },
  "review": {
    "decision": "reject",
    "alternate_item": "item-c",
    "reason": "Package sizes differ"
  },
  "review_status": "REVIEWED"
}

This is illustrative, including the review status name. The important property is that the correction preserves what was originally proposed. Reviewer identity, timestamps, and before-and-after changes belong in the associated audit history.

Using JSON payloads keeps the physical schema reusable across projects. The tradeoff is that field validation, schema versioning, and query ergonomics need explicit attention. A field contract must enforce types and allowed updates; a flexible column does not make every payload valid.

For an agent, I would extend this model with a separate immutable action proposal and a decision tied to that proposal. The proposed action needs its own identity, independently of the conversation that produced it.

2) Be precise about what the person is doing

Human participation can serve several different purposes:

Intervention	Question it answers	Example
Clarification	What does the user intend?	Which customer account should the agent update?
Output correction	What should the result be?	Replace a proposed product match
Plan review	Is this approach acceptable?	Approve a sequence of migration steps
Action approval	May this operation execute?	Authorize one specific catalog update
Outcome review	Did the completed work meet expectations?	Inspect a finished reconciliation report

These interventions can share a review interface, but their semantics differ. Clarifying an account identifier does not necessarily authorize a write. Approving a plan may authorize a bounded scope of work; the system still needs to define the scope and enforce it when tools execute.

This is where the bridge from batch review becomes useful:

Batch review concept	Agentic extension
Batch and row identity	Run, step, and action identity
Preserved source payload	Evidence snapshot and proposed tool arguments
Editable review fields	Decision plus explicitly permitted corrections
Stage completion rules	Preconditions for resuming execution
Finalization and writeback	Authorized dispatch and recorded outcome
Independent review streams	Isolated runs and delegated scopes

The right-hand column adds requirements. An agent may encounter a new decision halfway through a task, and the world can change while its execution is paused.

3) What companies and frameworks contribute

There are useful building blocks at different layers of this problem. These examples illustrate design choices rather than a ranking of vendors.

LangChain: execution interrupts. LangGraph supports interrupt() to suspend execution and surface a payload for external input, then Command(resume=...) to continue. It requires checkpointing and a consistent thread identifier; production use needs persistent storage. A key implementation detail is that the interrupted node starts again when resumed, so code before the interrupt runs again. That affects where you put side effects. LangGraph interrupt documentation.

Microsoft: plan review in orchestration. Microsoft Agent Framework’s Magentic orchestration uses a manager to coordinate specialized agents and exposes human plan review. It also provides controls on coordination rounds, stalls, and replanning. This illustrates review above the individual tool-call level: a person can evaluate the proposed approach before the team proceeds. The documentation flags limits to generalizing the original Magentic-One setup. Microsoft’s Magentic orchestration documentation.

Anthropic: measuring actual supervision. In observational research on Claude Code, Anthropic found that more experienced users both enabled automatic approval more often and interrupted agents more often. Those observations do not prove a particular supervision strategy is safer. They do show why approval count alone is an incomplete measure of human control. Measuring AI agent autonomy in practice.

My engineering takeaway is that these layers complement each other. A runtime provides suspension. An orchestration layer exposes decisions. Product design determines whether a person can understand and intervene. Application policy connects those mechanisms to actual authority.

4) Bind approval to the exact action

Suppose an agent proposes updating one catalog record. The reviewer sees the record, the before-and-after values, and the intended destination, then approves it.

Before execution, another step changes the destination or expands the update to a thousand records. Reusing the original approval would break the relationship between the reviewed proposal and the executed operation.

An approval belongs to the operation that was reviewed.

A recent preprint, Loopjacking, examines this class of failure. It distinguishes misleading representations at approval time from substitution of state after approval. Its tests concern selected products and configurations, and the authors explicitly avoid estimating ecosystem-wide prevalence. I take it as a concrete failure model to test, not a verdict on every implementation of the named frameworks. Loopjacking, September 2026 preprint.

For the proposed architecture, I would store an approval against an immutable action version containing:

• the run, project, and action identifiers;
• tool name, validated arguments, and destination;
• relevant resource versions and evidence references;
• the required reviewer role, policy version, and expiration;
• a canonical digest of the fields that define the operation.

The review screen should render the material operation from those structured fields, including its scope. An agent-written summary can help explain a proposal, but it should not be the only representation of what will execute.

A digest detects a changed proposal. It does not establish that the reviewer had authority, that the evidence was true, or that execution happened only once. Those require separate checks.

If a reviewer edits the action, create a new version and validate it. Any authorization must bind to the resulting version under the applicable review policy.

5) Treat the pause as a state transition

Here is the architecture I would use to extend the batch pattern:

Rejection stops that proposal. Expiration or a changed proposal returns it for review. Missing information can trigger clarification. A revised plan may produce a new action rather than resuming the old one.

The waiting state should survive a process restart. Store pending decisions durably, authenticate the reviewer, and resolve competing responses with a conditional state update. A second click should return the recorded decision or a conflict rather than create another authorization.

This also changes how I would handle parallel agents. Each pending action needs its own decision identity. A response approving one worker’s action must not release another worker’s action merely because they share a parent run.

6) The harness owns the human boundary

By agent harness, I mean the software around the model that manages the execution loop: assembling context, routing tool calls, persisting state, delegating work, and applying permissions. In the architecture I’m proposing, it also connects human decisions to the operations they authorize. Enforcement may span the harness, tool adapters, and destination services.

I use deep agents here as a descriptive pattern: agents that plan, work across many steps, manage working context, and may delegate subtasks. That is distinct from LangChain Deep Agents, a specific harness with filesystem, context-management, subagent, optional planning, and human-intervention capabilities built on LangGraph. Deep Agents overview.

LangChain’s Agent Harnesses & Deep Agents 101 webinar deck, featuring Jess Weng and Ishaan Lal, gives this a useful structure. It presents the agent as a model plus a harness, then groups harness capabilities around execution, context management, delegation, and middleware, including HITL. Its subagent examples show separate context windows and tool sets. Webinar deck, harness anatomy.

Here is how I would translate those capabilities into responsibilities for human oversight. The right-hand column is my proposed application design:

Harness capability	Responsibility in the proposed HITL architecture
Execution environment	Restrict credentials, writable resources, and network destinations so every route to a consequential operation respects policy
Context management	Preserve evidence references and pending decision identifiers when messages are summarized or moved out of context
Planning and delegation	Give each worker an explicit scope and retain the relationship between its proposals and the parent task
HITL middleware	Intercept a proposed operation, present its material details, persist the response, and resume through execution checks

That last row locates the review interaction inside the agent loop. For the catalog example, the model proposes an update; middleware routes the proposal for review before the adapter performs the write. The adapter still verifies authorization when execution occurs. This extends the deck’s treatment of HITL through middleware with the durable decision and execution controls described here. Webinar deck, HITL in the agent loop.

Longer execution makes continuity important. Anthropic’s work on long-running coding harnesses uses an initializer, incremental work, progress artifacts, and Git history to help subsequent sessions recover context. Its example concerns software development; my extension to HITL is that pending approvals and authorization scope also need to survive context turnover. A conversation summary should reference durable decision records rather than become the authority for them. Effective harnesses for long-running agents.

Consider an agent asked to investigate mismatched products in a catalog and prepare corrections. I would structure its harness around five steps:

1. Establish scope. Record the catalog, permitted records and fields, time or cost budget, and whether the user authorized investigation, proposed changes, or actual updates. In this example, investigation is authorized; writes require review.
2. Delegate within that scope. A comparison subagent can inspect product evidence and suggest matches. Give it only the tools and data access needed for that task. Its effective authority is the intersection of the parent’s scope, its assigned role, and current policy.
3. Assemble a reviewable proposal. Consolidate candidate corrections into an immutable change set with record identifiers, before-and-after values, evidence links, and unresolved exceptions. The reviewer can inspect the changes without reconstructing every agent message.
4. Pause at the write boundary. Obtain approval for that specific change set. If investigation expands into another catalog or a new type of update, evaluate the revised scope before proceeding. Reordering authorized research steps need not trigger another approval.
5. Resume with current checks. After review, revalidate scope, approval validity, and resource versions. Record what actually executed, including partial failures. An overnight pause must not turn yesterday’s permission into unrestricted authority today.

LangChain Deep Agents provides a concrete mechanism for the fourth step: interrupt_on configures review for selected tools, with configurable decision options and a checkpointer. Its documentation also covers subagent interrupts. Those mechanisms still need application-specific scope and authorization checks; a pause does not by itself establish who may approve a catalog write. Deep Agents HITL documentation.

Delegation introduces another boundary to test. Separate context windows help organize work, but do not establish a security boundary. A subagent with a shell and broad credentials may reach operations beyond its named tools. A parent’s review configuration should never be assumed to cover every child tool path. Check how the chosen framework propagates policy, and enforce the same write restrictions at the tool service even when a subagent or a general-purpose execution tool reaches it.

There is a tradeoff in the review unit. Approving an explicit change set can reduce interruptions, but the reviewer must understand its full scope. Any added records or material changes produce a new version requiring authorization under the policy. Broad plan approval is useful only when its limits are concrete enough for the harness to enforce.

This is where batch HITL experience transfers directly: preserve evidence, keep decisions scoped, and make completion visible. The harness carries those properties through planning, delegation, and execution.

7) Calibrated decisions inside the harness

The harness also needs to decide which situations warrant human attention. A useful emerging approach separates the model generating a plan from a model assessing a specific decision about that plan.

TypeSafe describes Jev as a non-text-generating System One model trained with reinforcement learning for calibrated decisions (RLCD). Given state and a question, it returns typed probabilistic outputs. The intended distinction is between training a model to make calibrated decisions and asking a text model to describe how confident it feels. TypeSafe’s introduction does not disclose enough training detail to reproduce RLCD, so I would treat the training method and performance results as vendor-reported claims. TypeSafe’s introduction to System One models and Jev.

LangChain’s harness example puts this kind of classifier around the execution loop, including model routing and tool-call risk assessment. That offers a concrete place to connect decision models to HITL: assess the proposed operation before the harness decides whether to proceed, block it, or persist a review request. Building a Harness with Jev.

For the catalog example, I would use this ordering:

The classifier supplies evidence to policy; it does not grant authority. A low estimated risk cannot authorize a write when the user only authorized investigation. Required human approval remains required regardless of the score. After any review, the same action-version, resource-version, and execution checks still apply.

The question being scored needs an operational definition. “Is this action risky?” mixes probability, consequence, and permission. I would separately assess whether the evidence supports the proposed correction and whether material ambiguity remains, while enforcing permissions through explicit rules. A probability of an error is not a measure of how costly that error would be. Thresholds should reflect the operation’s consequences and be validated on representative cases, rather than copied from a demo.

There is an important implementation detail in LangChain’s integration: the experimental AutoModeMiddleware classifies only explicitly listed tools and returns an error ToolMessage for blocked calls. It does not itself request human approval. Connecting a blocked proposal to a durable review queue requires a separate HITL path. The integration’s Noul output is a yes-probability; a value near 0.5 means comparable probability of either answer, not “medium severity.” LangChain TypeSafe integration documentation.

I would record the classifier version, question version, evidence references, returned probabilities, policy version, and routing decision alongside the immutable proposal. Missing or invalid assessments should follow a defined fallback, such as review or rejection, rather than silently permitting execution. This makes the decision reproducible enough to investigate and compare as models or thresholds change.

Typed outputs make integration more predictable, but a valid schema does not establish semantic correctness. TypeSafe reports substantial speed and cost improvements in its evaluations; its workflow comparisons use reference-model probabilities, which do not independently establish correctness or calibration on this catalog workload. Those results justify testing the approach, not assuming its production error rate. TypeSafe’s evaluation discussion.

8) Put the enforcement boundary at execution

A prompt asking the model to seek permission expresses intent. Enforcement belongs in the path that performs the operation.

For example, a tool adapter can require an accepted execution intent before it receives credentials for a write. That boundary should apply to delegated agents as well as the top-level planner.

This sketch shows the ordering I would want. It is application pseudocode; the helpers represent services and transactional behavior that still need implementation.

# Pseudocode: authorize and enqueue; do not call the external tool here.
def accept_approval(request_id, reviewer, now):
    with db.transaction():
        request = db.lock_pending_request(request_id)
        action = db.load_immutable_action(request.action_version_id)

        policy.require_reviewer(reviewer, request.required_role)
        policy.require_unexpired(request, now)
        policy.require_current_authority(action)
        require_digest(action, request.action_digest)

        intent = db.insert_unique_execution_intent(
            action_version_id=action.id,
            idempotency_key=action.id,
            expected_resource_version=action.resource_version,
        )
        db.mark_approved(request, reviewer)
        db.insert_outbox_event("execute", intent.id)

The transaction commits approval and the durable dispatch record together. A worker later consumes that record. Immediately before execution, it checks any revocable authority and approval validity, and submits the operation with the recorded idempotency key and expected resource version.

Where supported, the destination should enforce the version condition atomically with the write. Reading a version and then issuing an unconditional update leaves a gap in which the resource can change again.

That is a classic time-of-check/time-of-use problem with an AI interface attached to it.

9) Design for the ambiguous outcome

Now consider a less convenient failure:

1. The human approves an action.
2. The external API completes it.
3. The worker crashes before recording success.
4. The job is delivered again.

A durable checkpoint alone does not settle whether the external side effect occurred. The system needs a way to retry without duplicating the operation or to reconcile its outcome.

I would use a stable idempotency key where the destination supports one, respecting that API’s retention and retry semantics. Otherwise, an operation identifier or a queryable result may support reconciliation. If neither exists, the correct state can be “outcome unknown,” requiring investigation before another attempt.

An outbox makes dispatch durable. It does not make an arbitrary external API exactly-once.

This is directly related to batch writeback. Finalizing a batch, notifying its consumer, and confirming downstream application are distinct events. An agent adds more opportunities to encounter those boundaries during a single task.

10) Evaluate when to ask, not just whether the task finished

The HiL-Bench preprint studies agents facing missing, ambiguous, or contradictory requirements. It evaluates selective escalation using Ask-F1, combining question precision with blocker recall. That makes unnecessary questions and silent guessing visible in a way that task completion alone may miss. Its results are benchmark-specific; they should motivate application testing rather than substitute for it. HiL-Bench.

For an agent application, I would build evaluation cases around concrete boundaries:

Test case	Expected behavior
Required account identity is ambiguous	Ask a targeted clarification
Action exceeds delegated scope	Seek appropriate authorization or reject it
Tool arguments change after approval	Invalidate the previous approval
A paused run restarts	Recover the same pending decision
Two reviewers respond concurrently	Resolve once and record the result
External execution succeeds before a crash	Deduplicate or reconcile before retrying
Resource changes while approval is pending	Reject the stale write or obtain a new decision
A subagent attempts a write outside its scope	Block the write at the execution boundary
A revised plan adds another catalog	Reassess scope before accessing or changing it
Context is summarized during a pending approval	Recover the original decision record and exact action version

Measure escalation precision and recall against a reviewed test set, unauthorized executions, successful recovery, review time, and completed-task quality. Include a representative sample of routine cases as well as hard cases; a queue populated only with exceptions cannot estimate overall performance.

The policy itself has tradeoffs. Action-by-action review offers fine control but consumes attention. Bounded delegation reduces interruptions but needs explicit limits. Plan review helps establish direction, while execution checks protect the concrete operations that follow.

The choice should reflect consequence, reversibility, scope, and available evidence. A model’s self-reported confidence is insufficient as the sole approval policy. Training for calibrated decisions is a different approach, but its outputs still need validation on the workload where they control routing.

For a classifier-assisted policy, I would add these measures:

• Calibration: among actions assigned similar error probabilities, how often do the defined errors actually occur? Check this by tool and consequence category, with sample sizes and uncertainty.
• Risk among automatic executions: measure errors specifically in the subset allowed through, alongside automation coverage. An overall average can conceal failures in that subset.
• Human workload: track review volume, review time, and missed escalations. Fewer interruptions are useful only if the remaining automated behavior is acceptable.
• Operational cost: compare end-to-end latency and cost, including classification, retries, and human review.

Compare against deterministic rules and the existing review policy on the same held-out cases. Start with shadow decisions, audit a sample of proposed automatic executions, and recheck after model, prompt, tool, or traffic changes. Reviewing only escalated cases leaves the automatic path’s errors unmeasured.

11) Keep the feedback loop honest

A recorded correction can improve today’s business result. Turning it into a better agent requires additional work.

Preserve enough context to interpret the decision: the proposal, relevant evidence, reviewer correction, policy, and eventual outcome. Then decide whether that record belongs in a regression test, an evaluation dataset, a policy change, or a training process.

A rejected action might have been accurate but unauthorized. An accepted answer might still have been wrong. A clarification might reveal missing context rather than a model capability failure. Collapsing all three into a positive or negative training label loses useful information.

The strongest connection I see between batch HITL and agentic engineering is the discipline of making decisions explicit and durable. Preserve what was proposed. Define who can decide. Bind that decision to its scope. Execute it reliably. Observe what happened.

Once those pieces exist, we can make a much more meaningful decision about where to give an agent greater autonomy.

Original sources checked September 29, 2026; harness and Deep Agents references checked October 2, 2026. Jev and RLCD references checked October 21, 2026. Framework behavior evolves; the code above is an architectural sketch, not a tested SDK integration.