chat-quality

Biweekly loop closing the site chat's feedback cycle. The widget's thumbs verdicts land in Redis and, until this loop, nothing ever read them. Each run joins the verdicts to the exchange log, turns real failures into golden-set regression cases (the standing rule in evals/chat/README.md), runs the eval, and hands the fixes over on a local branch. Never push, merge, or deploy — the user approves every change.

When invoked

Run when asked to "run the chat quality loop" (/chat-quality), or by a scheduled task once one is created.

Scope guardrails (read first)

• The KV store is live. An Upstash Redis has been connected to the site since 2026-09-26, so the loop has real verdicts to read. The stop-and-ask rule in step 1 still stands as the safe failure mode if the credentials are ever missing.
• Read-only on the live Redis. SCAN, GET, LRANGE, LLEN against chat:* keys only — never SET, DEL, EXPIRE, or anything that writes. The logs are production evidence with a 30-day TTL; this loop observes them.
• Visitor privacy. Logged questions are visitors' own words and may carry personal details. They may appear verbatim in the local report, never in the committed diff: a golden-set case gets a paraphrase that preserves the failure, not the visitor's sentence. Visitor hashes never leave the report.
• One eval run per loop. npm run eval:chat spends real API budget (golden set × 3 repeats). Run it once, after the golden-set changes, not iteratively.
• Fixes stay inside the chat's answer pipeline: evals/chat/golden-set.json, corpus sources (page prose, corpus-facts() blocks, the generator's exclusions), the persona in website/src/app/api/chat/persona.ts, and the chat's lookup-tool layer — the CHAT_TOOLS definitions in website/src/app/api/chat/route.ts and their shared implementations in website/src/lib/site-tools.ts (a disliked prop or token answer can be a tool-description or lookup bug, and site-tools.ts also serves /api/mcp, so a fix there changes both surfaces). The engine is the package's, so a cause can also sit there: the follow-ups prompt, the handlers' notices and the ask_user guidance (DEFAULT_ASK_USER_DESCRIPTION, the authority for SPEC rules A1 and A2: a reply that asks when it should answer, or answers when it should ask) in src/chat/server.ts, the guardrail notices in src/chat/upstash.ts, the fallback follow-up pool in src/chat/followups.ts (and the site's own in website/src/lib/chat-followups.ts). Those are in scope, but a package edit changes every consumer's chat, not only this site's — say so in the finding, and prefer the site-side fix when one exists (for the asking guidance that is askUser: { description } in website/src/app/api/chat/route.ts). No UI, no components, no guardrail-cap changes (spend caps are the user's call — propose, don't edit).
• Local branch only. Never push or touch the user's working tree — do the work in a temporary git worktree on a fresh branch named chat/YYYY-MM-DD-<slug>, and remove the worktree when done (the branch survives). Skip the worktree entirely when there is nothing to encode.

The loop

0. Close the previous loop

Read the newest report in evals/chat/loop-reports/ (git-ignored, local-only). Note whether its branch was merged; don't re-propose a case a previous run already encoded. First run ever: say so and move on.

1. Pull the verdicts and the log

Credentials are KV_REST_API_URL / KV_REST_API_TOKEN in website/.env.local. They are often commented out there so local dev fails open — read the values from the file either way; never uncomment them or export them into a dev server. If the file has no values at all, stop and ask the user for read access; there is no loop without the data.

Query the Upstash REST API directly (plain curl): SCAN for chat:feedback:* and GET each verdict; LRANGE the day lists for the window since the last report (the key builders are owned by src/chat/upstash.ts and the entry shape by ExchangeLog in src/chat/server.ts — the site's website/src/app/api/chat/guardrails.ts only configures them — so read them there rather than trusting a remembered shape). Join verdicts to log lines by exchange id.

2. Triage

Every down verdict is a golden-set candidate: read the question, the answer, and the page it was asked from. The exchange's path field holds the request's whole context as the route checked it, so for the site chat it is a JSON string of the page and, when the visitor had highlighted something, the highlight with what it sits in (website/src/lib/site-chat-context.ts); a bare path is an older entry. A down verdict on a highlighted question is often about the highlight, so read it with the answer. Also skim the unrated exchanges for silent failures — invented paths, guardrail notices where a real answer was possible, questions the corpus plainly couldn't answer. The log does not record a question set the reply offered: an answer tapped on a question card arrives as the next exchange, its "question" being the plain-text answer ("Next.js", or one prompt per line for a set), so read those against the exchange before them rather than as visitor questions. For an asking failure, the cheapest assertion is the [question] line evals/chat/transform-response.js appends (contains when the reply should ask, not-contains when it should not). An empty or all-up window is a valid outcome: write the short report and stop. Don't manufacture cases from answers that were actually fine.

3. Encode the failures

Apply the standing rule from evals/chat/README.md: each real failure becomes a golden-set case before it is fixed — paraphrased question, its requiredFacts, and the cheapest assertion that would have caught it, with the case's description citing the rule ids it covers per evals/chat/SPEC.md (a failure no rule covers means the spec gains the rule in the same change). If a required fact is missing from the corpus, that's the actual bug: fix the source (page prose or a corpus-facts() block), and scripts/validate-chat-coverage.mjs will hold the new case to the regenerated corpus. The one exception: a failure whose facts live in the generated prop or token data is answered by the chat's lookup tools, not the corpus — its case is marked source: tools and carries empty requiredFacts (the standing-rule section in evals/chat/README.md and rule T4 in evals/chat/SPEC.md own the convention); never fix one by stuffing a prop fact into page prose.

4. Run the eval

Follow evals/chat/README.md exactly — it owns the procedure (the website-eval launch entry / npm run dev:eval -w website, the port-3000 gotcha, npm run eval:chat) and how to read the transcripts (seat by seat; the aggregate score is noise). The question being answered: do the new cases fail before the fix and pass after, and did nothing that previously passed regress?

5. Fix on a branch

Work in the temporary worktree on branch chat/YYYY-MM-DD-<slug>. One coherent batch: the new golden-set cases plus the corpus/persona fixes they demanded. A persona or tool-definition edit updates its matching rule row in evals/chat/SPEC.md in the same commit, and a new rule ships with either a tripwire or an explicit unenforced entry there. Do not build in the worktree: a fresh one has no node_modules of its own, so the prebuild chain fails wherever module lookup cannot climb to the main checkout's install (the corpus generator needs typescript), and a worktree that happens to sit inside the checkout only borrows that install, and installing there is what land forbids. The build check happens when the branch is landed, whose combined npm run verify builds it; when page prose changed, the corpus that build regenerates is committed with the landing.

6. Report and hand off

Save to evals/chat/loop-reports/YYYY-MM-DD.md and repeat in full in the final message: the window, verdict counts (up/down/total exchanges), each disliked answer and the diagnosis, what was encoded, the eval result, and the branch name. Plain English. End with the approval step: merging the branch (or saying ship) approves it; deleting it rejects it. Nothing is pushed or deployed.