voice-profile-test
Skill6 starsChanged 29 days agoMITView on GitHub
Test how accurately a voice profile lets a writer predict what a real person would say or write, on an occasion the profile never saw. Blind, held-out, multi-run, two-judge. Use when a request says "test this voice profile", "which profile is better", "does this profile work", "compare our profile with theirs", "score the voice profile", "held-out test", or before any voice profile is trusted, shipped, merged with another, or replaced. Works for any person and any profile format, spoken or written. The method is three prompts (extract, predict, judge) that run by copy-paste in Copilot, Quick or any chat. It carries the freeze-first rule, blind one-set-per-chat judging, the quote-or-it-doesn't-count rule, the two-judges-agree rule and the run-range tie rule. Optional rig scripts fill the prompts and add up the table for repeated validation runs.
In this repository30 other files it gives its agents.
- adversarial-re-readSkill
- adversarial-reviewSkill
- build-estimateSkill
- build-evaluateSkill
- build-executeSkill
- build-planSkill
- build-polishSkill
- build-prdSkill
- build-probeSkill
- build-validateSkill
- excel-connector-governanceSkill
- five-whysSkill
- insight-builderSkill
- insight-crystallizerSkill
- insight-promotionSkill
- multi-sessionSkill
- notion-governanceSkill
- oodaSkill
- prd-writerSkill
- promote-to-toolingSkill
- quality-reviewSkill
- ralph-implementSkill
- ralph-loopSkill
- search-governanceSkill
- session-outputSkill
- skill-validatorSkill
- task-planSkill
- verifySkill
- why-diagnosticSkill
- writing-qualitySkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 35
Hold out an occasion the profile never saw: If the profile, or anything it was built from, contains the held-out material, the test …
- Distinctive: line 116
A low score can be a talking point rather than a voice gap: If every profile missed the same conclusion, check whether the answer was that …
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Voice profile test
A voice profile is a claim: "a writer holding this document can sound and think like this person." This skill tests the claim the only honest way: take something the person actually said or wrote, hide it, ask writers to predict it from the profile alone, and have blind judges score the predictions against the real thing.
The method is the prompts. references/prompts.md is written to be pasted into Copilot, Quick,
Claude or any chat, with copy-paste, new chats and a small table filled in by hand. That is the
deliverable. Nothing in it needs Python, a repo or subagents, and every step has to stay that way.
The scripts are a rig for us, not the product. scripts/build_briefs.py,
scripts/build_packets.py and scripts/score.py fill the same prompts and add up the same table.
They make repeated runs cheap while we validate the method. A change that only works through the
scripts is not a change to the method. Port it into the prompts, or drop it.
The rules that make the result trustworthy
Each one exists because the test gave a wrong answer without it.
- Hold out an occasion the profile never saw. If the profile, or anything it was built from, contains the held-out material, the test measures recall rather than prediction. Grep every profile for a distinctive phrase from the held-out and record the result.
- Freeze first. Extract the questions and seal the real answers before building or editing any profile under test. A separate agent (or person) does the extraction. The orchestrator reads the questions only, and never the sealed answers until scoring is done.
- Match the held-out to the use. A spoken held-out (interview, podcast, talk) tests spoken prediction. A written held-out (posts, bylines) tests written prediction. A profile built for one can pass or fail the other for reasons that say nothing about its quality. State which you ran.
- Several runs per profile. One prediction run is noise. Run at least three per profile, each a fresh writer with the identical brief. A difference that stays inside the run-to-run range is a tie.
- Blind by construction. Each judge chat sees one set of predictions: one profile, one run. It is never told which profile wrote them. There is nothing to randomise or decode, so a marketer can run it by hand. (The first runs used blind pairwise packets with X and Y randomised per question. That needs tooling, so it was replaced on 2026-09-28. Whether single-set judging discriminates as well is being measured.)
- A score without a quote does not count. The judge quotes the real answer for every score, including a 0. Measured: a judge that skipped quotes on 20 of 32 lines scored one profile 13 against 2. The same packet under a quoting judge scored it 10 against 11.
- Two judges from different models. Trust only the items where they agree within 1 point, and list the disagreements rather than averaging them away. Different models differ in leniency. One judge was about 6 points more generous than the other, yet the two ranked the profiles the same, so rank matters more than level.
- Record by set. Log each judge's THINKING and SOUND per set in the table at the end of the prompts, then take the mean and range per profile.
Procedure
Work folder. Client material stays out of git. Use a gitignored folder with this layout. The scripts expect it:
<work>/heldout/source.* the held-out transcript or written pieces
<work>/heldout/questions.md SETTING: … / optional LENGTH: … / Q1–Q4 (from Prompt 1)
<work>/heldout/SEALED-answers.md A1–A4 (from Prompt 1)
<work>/profiles/<name>.md one file per profile under test
<work>/briefs/<name>.txt built by build_briefs.py
<work>/predictions/<name>-r<N>.md one file per profile per run, Q1–Q4
<work>/packets/PNN.txt built by build_packets.py
<work>/KEY-do-not-open/key.json built by build_packets.py
<work>/judges/<judge>/PNN.md one file per judge per packet- Choose the held-out. Pick an occasion of the right kind (rule 3), ideally recent, and one the
profiles did not use.
- Spoken: get a verbatim transcript. In Microsoft 365, the Teams or Stream caption export
(
-en-US.docx) is one. Outside it, use a good speech-to-text model: a fast one garbles enough words to move scores. - Written: collect the pieces verbatim, with URLs and dates.
- Spoken: get a verbatim transcript. In Microsoft 365, the Teams or Stream caption export
(
- Extract with Prompt 1A (spoken) or 1B (written), in a fresh agent or chat. Four items. The
extractor writes
questions.mdandSEALED-answers.mdand never quotes the answers back. - Put the profiles in
profiles/unchanged. If you are testing a merge, build it now, after the freeze, and tag every imported block with its source. - Build the briefs:
python3 scripts/build_briefs.py <work> --person "Name". This fills Prompt 2 with the setting and length fromquestions.md. - Predict: for each profile, run N ≥ 3 fresh writers on its brief. Each reads the brief only
and writes
predictions/<name>-r<N>.md. - Build the packets:
python3 scripts/build_packets.py <work> --person "Name" --runs 3. This gives one Prompt 3 per profile per run. Packet numbers are shuffled, and the key is kept apart. - Judge: each packet goes to a fresh judge chat, for each of two judges from different models.
Save each reply as
judges/<judge>/PNN.md, with the score table as markdown:| question | conclusion | route | evidence | sound |. A chat UI's copy button may flatten tables, so rebuild them from the page. - Score:
python3 scripts/score.py <work>. It reports:- per judge, per profile: thinking and sound per run, with mean and range;
- the agreed items, and every disagreement;
- for each packet: rows parsed, quotes counted, and whether the judge's own totals match its table.
- Report numbers with bounds. State the kind of held-out, the setting, the profile lengths (a longer profile is a confound), any topic overlap between the profiles' sources and the held-out, whether predictions and a judge share a model, the sample size, and the leak check. A result file with no bounds is not a result.
Reading the result
- Thinking is conclusion + route + evidence, out of 6 per question. Sound is out of 2.
- Within-judge ranges that don't overlap show a difference for that judge. Report it as real only when both judges show it, or the agreed items do.
- A low score can be a talking point rather than a voice gap. If every profile missed the same conclusion, check whether the answer was that season's messaging.
- A profile of rules (bans, framing rules, audience modes) with no evidence of how the person actually talks tends to score low on sound and can push writers to invent specifics. Look for invented names or numbers in its predictions.
Known results
- 2026-09-28, spoken held-out, 3 profiles × 3 runs, Claude and Copilot judges.
- A profile built from recordings, and the same profile merged with a communications team's rules, tied on every measure.
- The rules-only profile scored lower on thinking under both judges. On sound, the judges agreed on every item: 0.48 against 1.48 out of 2.
- The two judges ranked the profiles identically. Copilot was more lenient: it scored higher on 27 of the 31 items where the judges disagreed.
- 2026-09-28, written held-out (4 LinkedIn posts), same 3 profiles, single-set judging.
- A three-way tie, but in the opposite direction: the rules-only profile led on the Claude judge.
- The merged profile tied the best profile in both tests, so it is the only one that is never behind.
- Copilot scored near the maximum (21–24 of 24), so it barely separated the profiles.
- Copilot's own totals disagreed with its own table in 6 of 9 packets. Never let the judge add up. The person adds the table.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

