Human-in-the-Loop Designer

Every checkpoint costs attention. Spend it where human judgment actually changes the outcome.

What this skill is

A workflow for placing and designing the moments where an agent pauses for a person: checkpoints, approval gates, escalations, and full handoffs. It treats human attention as a budgeted resource, sizes each touchpoint by its interruption cost and its error-catching value, and caps review load so approvals stay meaningful. Use it after autonomy levels are set (see agent-autonomy-level-mapper) to design how the human side of each gate actually works.

What it solves

• Reviewers approving 50 items an hour without reading, so the gate catches nothing
• Agents that escalate everything, or nothing, because escalation triggers are undefined
• Handoffs where the human receives a task with no context and starts from zero
• Workflows that stall for hours because the approver is unavailable and there is no fallback
• Checkpoints placed at the end of a long run, after expensive or wrong work is already done

When to invoke

• Designing a multi-step agent workflow with human review points
• Audit or compliance requires a human to sign off on agent output
• Approval rates are near 100 percent and reviewers report fatigue
• Customer-facing agents need a path to a human representative
• An agent run failed late and the team asks "why didn't we check earlier?"

Phase 1: Map the workflow and the humans

Lay out the agent workflow as steps, then list each human role involved.

Capacity limits are estimates to validate in research; start conservative.

Phase 2: Choose the touchpoint type

Place checkpoints after the step with the highest downstream cost, usually the plan, not the final output. Catching a wrong plan is cheaper than reviewing a wrong deliverable.

Phase 3: Score each candidate touchpoint

Touchpoint value = P(agent error at this step) x Cost of error x P(human catches it)
Interruption cost = Review time + Context-switch penalty + Wait time imposed on the workflow
Keep the touchpoint if value > interruption cost

Practical defaults:

• Context-switch penalty: 5-15 minutes for asynchronous reviewers pulled from other work; near zero if in session
• P(human catches it) drops sharply when the reviewer cannot see the evidence; if the gate shows only a summary, assume under 30 percent
• A touchpoint with high error cost but low catch probability needs a better review interface, not removal

Phase 4: Set review-fatigue limits

Rubber-stamping is the main failure mode of human-in-the-loop systems. Define limits per role:

Volume controls:

• Batching - group similar low-risk items into one review with exceptions highlighted
• Sampling - review a random 10-20 percent when a gate is mostly confirmatory, plus 100 percent of flagged items
• Exception-only review - agent surfaces only items that fail a rule or fall below a confidence threshold

Phase 5: Design escalation and handoff

Escalation triggers (define each explicitly):

• Confidence below the measured threshold for this action
• Policy or permission boundary reached
• Conflicting instructions or missing required data
• User expresses frustration or asks for a person
• Retry limit exceeded (for example, 2 failed attempts)

Every handoff package must include:

Define fallback when the human is unavailable: wait with a stated expected time, route to a backup role, or safely park the task. Never let the agent proceed past a gate by timeout.

Phase 6: Specify the touchpoint interface

For each kept touchpoint, write a one-row spec: trigger, who, channel (in-product, email, chat, queue), content shown, actions available, expected response time, and fallback. Tell the waiting user who is reviewing and when to expect an answer.

Output

• Workflow map with human roles and review capacity
• Touchpoint inventory with type, value score, and interruption cost
• Review-fatigue thresholds and volume controls per role
• Escalation trigger list and handoff package template
• Touchpoint interface specs with fallbacks
• Instrumentation list (approval rate, review time, edit rate, wait time)

Operating rules

Always

• Put the earliest checkpoint after the highest-leverage step
• Show the reviewer the evidence, not just the agent's conclusion
• Budget reviewer capacity and monitor approval rate and review time
• Include full context in every handoff so the human does not restart
• Tell the waiting user who is reviewing and the expected response time

Never

• Add a gate only to shift liability onto the reviewer
• Let an approval time out into automatic execution
• Escalate without stating why the agent stopped
• Treat a 99 percent approval rate as evidence the gate works
• Route every edge case to a human without a capacity plan