pm-archetype-ai
Skill1 starsChanged 42 days agoMITView on GitHub
AI / ML / LLM PM archetype lens. Invoke when the product is partly probabilistic — LLM features, agents, retrieval, classification, generation, recommendation, anomaly detection, model-in-the-loop tooling — and the work needs eval suites, guardrails, failure-mode analysis, human-in-the-loop, model selection, prompt design, eval-driven iteration, or safety/policy. Probabilistic systems need a different discipline than deterministic software. Trigger on "eval suite", "guardrail", "hallucination", "esse feature usa modelo?", "como medimos qualidade do modelo?". For Claude-API-specific implementation use the `claude-api` skill instead — this skill is the PM lens, not the implementation reference. The full trigger-phrase list and phase-skill pairings live in the skill body.
In this repository65 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- anti-slopSkill
- data-science-analystSkill
- humanize-deliverablesSkill
- humanizerSkill
- inference-disciplineSkill
- pm-archetype-aiSkill
- pm-archetype-enterpriseSkill
- pm-archetype-growthSkill
- pm-archetype-platformSkill
- pm-phase-defineSkill
- pm-phase-deliverSkill
- pm-phase-developSkill
- pm-phase-discoverSkill
- pm-prioritization-regua-comumSkill
- pm-product-senseSkill
- pm-storytellingSkill
- pm-transversal-analysisSkill
- pm-transversal-commsSkill
- pm-transversal-docsSkill
- pm-transversal-stakeholderSkill
- repo-doctorSkill
- anti-slopSkill
- data-science-analystSkill
- humanize-deliverablesSkill
- humanizerSkill
- inference-disciplineSkill
- pm-archetype-aiSkill
- pm-archetype-enterpriseSkill
- pm-archetype-growthSkill
- pm-archetype-platformSkill
- pm-phase-defineSkill
- pm-phase-deliverSkill
- pm-phase-developSkill
- pm-phase-discoverSkill
- pm-prioritization-regua-comumSkill
- pm-product-senseSkill
- pm-storytellingSkill
- pm-transversal-analysisSkill
- pm-transversal-commsSkill
- pm-transversal-docsSkill
- pm-transversal-stakeholderSkill
- repo-doctorSkill
- umbertoSkill
- anti-slopSkill
- data-science-analystSkill
- humanize-deliverablesSkill
- humanizerSkill
- inference-disciplineSkill
- pm-archetype-enterpriseSkill
- pm-archetype-growthSkill
- pm-archetype-platformSkill
- pm-phase-defineSkill
- pm-phase-deliverSkill
- pm-phase-developSkill
- pm-phase-discoverSkill
- pm-prioritization-regua-comumSkill
- pm-product-senseSkill
- pm-storytellingSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 151
every release passes the eval suite at the declared threshold
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
PM Archetype — AI / ML / LLM products
Product-type lens. Pairs with phase skills (
pm-phase-{discover,define,develop,deliver}) when the product is probabilistic. The phase skills cover when and how to sequence; this skill covers what's special about AI work.
Shape: a compositional lens — one SKILL.md that composes the phase and transversal references linked below and adds the archetype-specific concerns, anti-patterns and output format. It carries a single reference of its own, references/eval-design.md, added when a real need appeared (PM-owned eval design had no home in any phase skill); the other archetype lenses stay single-file until they show the same need. Its evals/evals.json (standard, adversarial, negative control) is the quality contract and is graded like any full skill.
Progressive loading
Load this SKILL.md first. references/progressive-loading.md names the one supporting reference and when to open it.
Prime directive
Quality bars first, then experience. AI products that ship without measurable quality thresholds look magical until they embarrassingly fail in production. Set the eval suite and guardrails before — not after — you ship.
When to invoke
The product or feature includes any of:
- LLM-driven generation (text, code, summaries, drafts)
- LLM-driven extraction or classification
- Retrieval-augmented generation (RAG)
- Agentic flows (tool calls, multi-step planning)
- Recommendation, ranking, scoring, anomaly detection
- Voice / vision / multimodal models
- Any output users will treat as authoritative that came from a stochastic system
Skip this skill when the AI piece is purely backend optimisation users never see (e.g. internal log clustering with no user-facing surface).
Trigger phrases
"AI feature", "LLM", "modelo de ML", "evals", "eval suite", "guardrail", "hallucination", "HITL", "human in the loop", "model selection", "prompt strategy", "fallback chain", "drift", "red team", "safety policy", "esse feature usa modelo?", "tem AI nisso?", "what's the eval threshold to ship?", "como medimos qualidade do modelo?".
Required reading before output
.ai/rules.md; for project work, read.ai/memory/active-context.mdfirst when present and resolve<slug>before opening relevant project files. If no pointer exists, do not infer a project; initialize only when durable project context is needed and the project name is known.- relevant project memory — prior eval results are load-bearing; if the team has shipped anything AI-shaped before, the eval log determines what's possible now
- shared org context in
.ai/memory/org/when present (personas.md,competitors.md,goals.md); open the file the task needs, not all four
References this skill chains to
Pairs with pm-phase-develop (PRD with eval plan), pm-phase-deliver (release gate by eval pass-rate), pm-transversal-analysis (qualitative eval of outputs), and the claude-api skill for Claude-API-specific implementation. Specific references:
references/eval-design.md— scenario sheet, hazard list, limits block, golden set with a real failure, block rule, validation and verification of the judge../pm-phase-develop/references/prd-writing.md— AI PRDs emphasise failure modes + eval plan + HITL../pm-phase-develop/references/tracking-plan-design.md— AI apps need bespoke events (inference called, tool used, fallback triggered, user rated output)../pm-phase-deliver/references/experiment-interpretation.md— A/B on AI outputs requires care (variance, subjectivity)../pm-phase-deliver/references/metric-quality-guardrails.md— AI guardrails are critical../pm-transversal-analysis/— qualitative eval of outputs is a core AI skillclaude-apiskill (external Anthropic skill, not bundled here) — when the implementation lives on the Claude API / Anthropic SDK
AI-specific concerns to cover in any AI PRD
- Evaluation suites — representative test cases, graded against quality rubric; run before release
- Guardrails — hard filters (content policy, PII, toxicity), soft filters (confidence thresholds), HITL fallbacks
- Failure modes — hallucination, misinterpretation, bias, jailbreak, data-leak, brittleness to prompt phrasing
- Human-in-the-loop — when to route to human review; how escalation works; user-facing UX for uncertain outputs
- Model selection — which model for which cost / latency / quality trade-off; fallback chain
- Prompt engineering — system prompt, few-shot examples, structured outputs, prompt caching, versioning
- Observability — trace per inference, token cost, latency, confidence, user feedback
- Safety + policy — content policy, moderation, red-teaming, regulated domains
- Data flywheel — how user interactions improve the model / prompts / retrieval
Workflow
- Define the user task — what are we automating or augmenting? What does "good output" look like?
- Define the quality rubric — dimensions (accuracy, helpfulness, safety, tone) and grading method per dimension.
- Draft the eval suite — 20–100 representative cases covering happy paths, edge cases, adversarial inputs, laid out as a scenario sheet, hazard list, limits block and golden set per
references/eval-design.md. - Pick the model + prompt strategy — cost / latency / quality envelope; fallback chain.
- Design the guardrails — hard limits, soft limits, HITL routing.
- Plan the release gate — eval pass-rate threshold and block rule before shipping; canary rollout.
- Design observability — traces, feedback capture, drift detection.
- Define the iteration loop — cadence for re-running evals, updating prompts, reviewing failures.
- Stress-test failure modes — loop in your QA lead for code / metric / experiment integrity.
- Architecture check — loop in your engineering architecture partner for inference infrastructure, fallbacks, cost control.
- Update memory with eval-suite decisions, model selections, guardrail policies.
AI-specific anti-patterns
- "AI delight" without measurement. Shipping because the demo looked great; first users hit failure modes the team never tested.
- No eval before release. Relying on "looks good" rather than graded test cases.
- Guardrails as an afterthought. Retrofitting safety after an incident is painful.
- Over-trusting benchmark scores. Public benchmarks often don't match your task distribution.
- No human fallback. Model fails → user has no path forward.
- Hidden costs. Inference cost not instrumented; surprise bill arrives.
- No drift monitoring. Models and prompts work day one; degrade silently as upstream changes.
- Ignoring policy / privacy / safety. Shipping in regulated domains without legal input.
- Hallucinating confidence. Product asserts facts the model made up; users trust the UI.
- Prompt-only thinking. Treating prompt engineering as the whole discipline; ignoring retrieval, tools, evals, HITL.
Output format
## pm-archetype-ai recommendation
### User task + quality rubric
what we're building; dimensions of "good"
### Eval suite
coverage + size + grading method + pass-rate threshold and block rule to ship
### Model + prompt strategy
model selection rationale, prompt structure, caching, few-shot, fallback chain
### Guardrails
hard limits / soft limits / HITL routing / content-policy alignment
### Observability
traces, feedback, cost, latency, drift monitors
### Failure modes + mitigations
hallucination / bias / jailbreak / brittleness — known ones + detection paths
### Release gate
eval pass rate + canary plan + rollback
### Iteration loop
cadence for eval re-runs; how we use user feedback
### Safety + policy
legal / privacy / safety review needed; regulated-domain implications
### Memory updatesIntegration
- Upstream:
pm-phase-develop(PRD with eval plan, tracking plan tuned for AI events). - Downstream:
pm-phase-deliver(release gate by eval pass-rate, post-launch A/B interpretation with AI variance in mind). - Transversals:
pm-transversal-analysis(qualitative review of outputs),pm-transversal-docs(model card, eval log, failure-mode log in Confluence). - Engineering pairings: your engineering architecture partner for inference architecture; your QA lead for eval / failure-probe design;
claude-apiskill for Anthropic SDK implementation. - Copilot mirror: .github/agents/pm-ai.agent.md (kept for GitHub Copilot harness compatibility).
Success criteria
- every release passes the eval suite at the declared threshold
- guardrails catch predictable failure modes before users see them
- HITL routing works for uncertain cases; users have a fallback
- cost and latency stay within envelope at scale
- drift is detected within days, not months
- incidents decrease quarter over quarter as the eval suite matures
- the team carries institutional memory of failure modes; we don't re-learn the same lessons
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

