Build a baseline

Some questions get re-answered every session: which keywords matter, which channel to spend on, which feature to build next. Each pass gathers what it can, reaches a confident conclusion, and the conclusion is different from last time. That is not carelessness. It is what happens when the answer is produced by judgement over whatever happened to be in context, because context changes and judgement follows it.

The user experiences this as being unable to trust anything you say. They are right to, and the fix is not to try harder. It is to move the answer out of your judgement and into a rule that produces the same output from the same data.

One founder put the failure exactly: "you do research, you conclude something, then you look at more research, then change your mind. you have no baseline. so you just make it up every single time."

When this applies

Reach for it when any of these is true:

• The same question has had two or more confident, different answers.
• The user says you contradict yourself, or asks what your baseline is.
• A recommendation rests on a number from a vendor's tool that nobody has ever checked against anything.
• You are about to rank things (keywords, channels, features) by reasoning.
• New data keeps arriving and each batch feels decisive.

It does not apply to one-off factual lookups, or to genuinely new questions with no prior answer to drift from.

The five parts

A baseline is not a document of conclusions. It is four mechanical parts plus a rule about who may change them.

1. A fixed scoring rule, preferably somebody else's

Write the ranking as an arithmetic formula over named inputs. Prefer a formula published by an outside source over one you invent, for one reason that has nothing to do with it being better: a formula you invented is a formula you can quietly adjust, and the user cannot tell the difference between you refining it and you rationalising a new favourite answer.

If the outside source publishes more than one version, that is not a problem to resolve by taste. Test them (part 4).

2. The subjective inputs frozen in a file, with a date

Most scoring rules need at least one human judgement, like how relevant a thing is to this product. Put those numbers in their own file, date them, define the bands they come from, and write at the top: change a number only with a stated reason and a date, never to make a favoured option win.

This is the part that actually stops the drift. The formula was never the problem; the problem was re-deriving the inputs each time with a different mood.

3. An instrument whose limits you have validated

You will need to measure something at scale, and the cheap way to measure it is usually an approximation of the expensive way. Validate the cheap instrument against the trusted one on one case before using it on a hundred, and write down what it is not good for.

A worked example. A free public search endpoint returned substantially the same set of competing apps as the paid ranking tool, with correct numbers attached, but in a different order: it placed one app second that the paid tool did not show at all, and omitted two the paid tool did. So it is sound for "who competes here and how big are they" and must never be used for "who ranks where". Stating that limit in the script's own comments is what stops the next session using it for the wrong question.

4. Falsification thresholds, written before the test runs

For every input you are trusting, name the quantity that would show it is informative, the threshold it must clear, and what you will do if it fails. Write all of that down before running the test, in the script, so the result cannot be reinterpreted into a pass.

The worked example again: "does this vendor's difficulty score predict anything real? Test it against an independently measured quantity. Falsified if the rank correlation is below 0.4, in which case the vendor score is near noise for us and the measured quantity replaces it." It came back at 0.788 and monotonic across every band, so it passed, and that pass means something because it could have failed.

Expect to be wrong about which way a test will go. In the same session the assumption that a multiplicative formula would beat an additive one was tested and lost, 0.552 against 0.665. Being tested and losing is the skill working.

5. Two scores, never one, when attractiveness and feasibility differ

The most common silent error in a ranking is folding "how good would this be" and "could we actually do it" into one number. They are different questions, they are measured differently, and the second one is a gate rather than a term in a sum.

Keep them separate and make the feasibility check mandatory. In the worked example, even the better of the two published formulas separated achievable options from impossible ones at only 0.665, where 0.5 is a coin flip. That is the argument for the gate: a high attractiveness score is never sufficient on its own, and discovering that is only possible once the two are measured apart.

How to do it

1. Name the recurring question and find every previous answer to it. Those answers are the evidence that a baseline is needed, and the list of things it will supersede.
2. Find an outside scoring rule. Search for a published method before inventing one. Note every version it publishes.
3. Write the inputs file and freeze the judgements, with bands and a date.
4. Write the scorer as a script, not as a table in a document. A script reruns; a table rots. Have it read the measured data from wherever it already lands, so the baseline updates when the data does and not otherwise.
5. Validate the instrument on one case against a trusted source. Write the limit into the script's comments.
6. Write the validation script with thresholds stated in its own comments, and run it. Report n, the statistic, the threshold, and the verdict.
7. Fold the validated result back in. If an input failed, replace it. If a feasibility gate exists, make it change the tiers.
8. Write the baseline document, which carries: the vocabulary, the formula and its source, the inputs and where they live, the validation results, what the baseline currently says, and a list of the decisions it supersedes so they cannot be resurrected.
9. Point everything at it. The project's rules file, and any older document that used to hold the answer, should say the ranking lives in the baseline now.

The rules that go in the finished document

Put these in the baseline file itself, because the next session reads that and not this skill.

1. Check the feasibility gate by hand before acting on any item. The score cannot see it.
2. Never quote a number from the baseline from memory. Re-run it and say the date.
3. Do not change a frozen input to win an argument. Changing one changes every ranking derived from it.
4. One position at a time. A new measurement updates the baseline. It does not start a fresh opinion in conversation.
5. Record superseded decisions with the reason and the date.

What this skill will not do

• It will not make a wrong formula right. It makes a formula's output stable and its failures visible, which is a different and more useful thing.
• It will not remove judgement. It moves judgement into a named, dated, reviewable file instead of leaving it implicit in each session's reasoning.
• It will not tell you the answer is correct. A baseline that passes its own validation can still be measuring the wrong thing, which is why the superseded list and the stated thresholds matter.