Automate Tests

A test's scenarios are the spec. Your job is to make every scenario run as an automated test, or to say exactly why one can't.

First, load the team config with roost-config, and read its tests adapter (tests.kind). Each step below names one of the adapter's operations ("Read a test's scenarios", "Mark a test automated", …); the adapter says how to do it with this team's tests.

Leave manual tests alone, if the adapter has them ("Manual tests"). A manual test is run by hand on purpose and closes on a recorded run. Don't write automation for it, don't propose splitting it, and don't count it as missing automation; list it in the report as "manual".

Inputs

• The test keys, or the story whose tests to cover ("A story's tests").
• The branch to work on. Use the one checked out unless told otherwise.
• Any notes on which scenarios the other repo already automates.

1. Read the scenarios

"Read a test's scenarios", with any shared setup they depend on ("Shared setup"). Never print a credential the adapter uses.

2. Map every scenario

For each test, list each scenario by name with:

• Where it's automated, if anywhere ("Find a scenario's automation"; match what the repo's existing tests use).
• Its latest result, from a run you do now ("Run a test"), not CI history. Check whether CI actually skips it (a test against a live outside service, skipped in CI, isn't passing in CI).

3. Write what's missing

When the test is its own automation (an eval case, say): a test drafted on the story but not in the repo is committed first, as drafted, before the change that should make it pass; then make the change until it passes every run.

When scenarios are automated in a repo's suite:

• Put each scenario at the layer that can really prove it. Behaviour the browser can't see (a database row, an email not sent, a token never logged) belongs in the backend's tests; write the UI scenario verbatim with e2e.backend_marker and a comment naming the backend test.
• Follow the repo's conventions: feature and steps files under e2e.scenarios_dir, the test's key as e2e.tag_style says, and the flag tags (stack.flag_tags and similar) so gated scenarios skip in the wrong flag state. Choose the tags from the local stack's real flag state (stack.flag_state), not from the kickoff text; if they differ, say so in the PR.
• Start or stop parts of the local stack yourself when a test needs them, as stack.how says (never stack.never).
• Drive through the real app on the services in stack.services. Stub network calls only where the scenario itself describes a stubbed state (a pending or failing request, a 403 the clock would produce).
• Use the test helpers the suite already has (for example e2e.helpers_hint) before writing new ones.

4. Knowledge goes in the repo, not your memory

These tests run in CI and on teammates' machines. When something blocks a scenario (a missing capability, an unbuilt slice, a bug), mark it skipped in the repo the way its suite does (for example @skip in a feature file), with a comment giving the reason and the ticket. Setup knowledge (how to create a fixture, which flag state a scenario needs) also goes in comments next to the scenario.

5. When the shipped behaviour differs from the scenario

Don't edit a scenario to make a test pass. Only when the behaviour deliberately differs (a decision changed it), update the scenario ("Update a test's scenarios", saving the old text to a local file first and saying where), and tell the user why.

If a scenario should be split because part of it won't be automated, propose the split ("Split a test") and wait for the user's answer.

6. Mark it automated only when everything passes

Mark the test automated ("Mark a test automated") only when every scenario has automation that passed in your run, at any layer. If any scenario is skipped, pending, gated by an unbuilt slice, or a no-op with no backend test, leave the test unmarked. When the tracker requires Components (tracker.components_required), set them on each test that's a ticket as you finish it.

7. Report

Anything the user must do goes first, in its own block headed 🙋 Your turn: one line per action, saying exactly what and where. Leave the block out when there's nothing; never write "nothing needed". Don't hand the user anything you can do yourself (attaching files to the tracker, retitling a PR, checking a setting through a connected tool); only sign-ins, vcs.signing_blocker and decisions are theirs. Then start with the outcome: "4 of 6 tests automated; 2 blocked." Then one table:

Say plainly what passed and what didn't. List anything you couldn't automate and why.