Human-in-the-Loop Designer

Purpose

Design human-in-the-loop (HITL) workflows: when humans intervene, how work is queued, and how feedback improves the AI system over time.

Input: AI use case, risk level, automation target, team roles, SLA requirements
Output: HITL Workflow Specification with triggers, queues, UI/API hooks, and metrics
Examples: See examples.md for worked input/output.

Workflow

Step 1: Assess automation boundary

Document what the AI may do autonomously vs what requires human gate.

Step 2: Define escalation triggers

Each trigger maps to queue priority and assignee role.

Step 3: Design review queue

Define states: pending → in_review → approved | edited | rejected | escalated.

Step 4: Design reviewer experience

• Diff view — AI output vs policy template
• One-click actions — approve, edit-and-send, reject-with-reason
• Keyboard shortcuts — for high-volume queues
• Audit trail — who decided what, when

Specify API or UI integration points (webhook, ticket system, Slack).

Step 5: Close the feedback loop

Route feedback to ai-evaluation-builder datasets and prompt iteration.

Step 6: Define HITL metrics and SLAs

Step 7: Validate

Run Validation checklist.

Decision Rules

Validation

• Risk tier and automation boundary documented
• ≥3 escalation triggers with thresholds
• Queue schema and state machine defined
• Reviewer actions and audit trail specified
• Feedback loop connected to eval/improvement
• SLAs per risk tier with alert thresholds
• User-facing messaging for pending human review
• Rollback path if HITL system fails

Anti-patterns

• HITL theater — queue exists but reviewers always click approve.
• No feedback capture — human edits discarded; system never improves.
• Unbounded queue — no SLA; users wait indefinitely.
• Reviewer without context — AI draft shown without inputs or policy.
• Same gate for all risk levels — over-reviewing low-risk kills throughput.

Best Practices

• Start with shadow mode: AI drafts, human sends, before any automation.
• Sample-review low-risk paths; full gate only where regulation requires.
• Instrument override reasons as structured tags, not free text only.
• Pair with guardrails-builder for automated pre-queue filtering.
• Reconcile HITL metrics weekly with product and safety owners.

Output Structure

# HITL Workflow: [System Name]

## Risk Assessment
| Tier | Autonomous scope | Human gate |
|------|------------------|------------|

## Escalation Triggers
| Trigger | Threshold | Queue | Priority |
|---------|-----------|-------|----------|

## Queue Specification
[State machine + fields]

## Reviewer UX
[Actions, integrations]

## Feedback Loop
| Action | Destination |
|--------|-------------|

## SLAs & Metrics
| Metric | Target | Alert |
|--------|--------|-------|

## Rollout Plan
[Shadow → partial → full automation]

Next Skills