agentleFS
Sign inStart free
gladosAagmangladosAagman/youtube-agent-skill/skills/yt-test/SKILL.md

yt-test

Skill1 starsChanged 5 days agoMITView on GitHub

Design and read a thumbnail or title A/B test properly - what to vary, how long to run it, and whether the result is a real winner or noise, using a two-proportion significance test on impressions and clicks. Use for "A/B test my thumbnail", "test and compare", "which thumbnail won", "is this result real", "should I swap the thumbnail", or a pasted test result.

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 1 is distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Read ~/.claude/youtube/voice.md if it exists.

    line 25
  2. Distinctive:

    Never invent a number. If the user has percentages and no counts, say what that can and cannot tell you (below).

    line 27
  3. Very specific:

    Check both windows before keeping the winner, and if the video's retention dropped, the fix is the opening, not another thumbnail: …

    line 62

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

yt-test

A thumbnail test on a small channel is usually decided by noise, then believed. 40 clicks against 52 feels like a winner. It is not one. This says whether it is.

python3 abtest.py --variant A:12400:496 --variant B:11900:571        # control first
python3 abtest.py results.csv                                         # name,impressions,clicks

Before you write

  1. Read ~/.claude/youtube/voice.md if it exists. Variant titles still have to sound like the channel.
  2. Never invent a number. If the user has percentages and no counts, say what that can and cannot tell you (below).

Designing the test

  • Change one thing. The face, or the text, or the colour. Two changes at once and the result cannot say which one did it.
  • Make the variants genuinely different. Two near-identical thumbnails produce a test that cannot be called at any sample.
  • Write them with /yt-package, and lint each with title.py --thumb.
  • Decide the stop rule first: the impressions per arm, or a date. Stopping when it looks good is how noise becomes a winner.

Reading the result

verdictmeans
WINNERunlikely to be noise, and both arms have 1,000+ impressions
LOSERthe challenger is reliably worse than the control
NOT YETthe gap might be real; the tool says roughly how many more impressions it needs
NO CALLtoo close to call at any realistic sample. Keep the one that fits the video better

With three variants, the significance bar is split between them, so a three-way test is not three chances to fool yourself.

YouTube's own Test & Compare

Studio picks its winner on watch time share, not click-through, and it shows percentages without the counts behind them. A percentage alone cannot be tested for significance, so abtest.py does not try. Use Studio's verdict for Studio's test, and say "Studio judged this on watch time".

CTR is the click, not the watch

A thumbnail that wins on CTR and loses average view duration over-promised. Check both windows before keeping the winner, and if the video's retention dropped, the fix is the opening, not another thumbnail: /yt-retention.

The gate

Nothing here publishes or swaps anything. This skill reads and you decide. The last line of every run is the question: ship it, or change it?

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free