e2e-pr-check
Skill829 starsChanged 3 years agoMITView on GitHub
Select, run, or debug Artemis Playwright tests for a branch or pull request, or write a layout contract that measures element sizes and alignment.
In this repository8 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- ci-triageSkill
- client-conventionsSkill
- liquibase-migrationSkill
- local-setupSkill
- server-arch-gatesSkill
- write-testsSkill
The file
We split this skill into the rules it gives an agent
6 rules. About 0 are common, 4 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 13
Use the CI resolver instead of selecting specs by inspection.
- Very specific: line 26
When RUN_ALL_TESTS=false, RELEVANT_TESTS is the space-separated set of paths relative to src/test/playwright.
- Very specific: line 67
Default to the single-node runner.
- Distinctive: line 74
Use the multi-node runner instead when the diff touches cluster-sensitive code, because a single node cannot reproduce cross-node failures …
- Distinctive: line 81
Treat a change as cluster-sensitive when it touches any of:
- Distinctive: line 89
If the change is specifically about the distributed data abstraction, run the suite on both providers.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Run the E2E tests this change affects
Select affected specs before running Playwright. Use the result and the runner's topology to classify failures.
Step 1: work out which specs are affected
Use the CI resolver instead of selecting specs by inspection. Its output depends on the base revision and committed diff:
./supporting_scripts/ci/E2E-tests/determine-relevant-tests.sh origin/developIt prints five OUTPUT: lines. The ones that matter:
RUN_ALL_TESTS=truerequires the full suite. This includes configuration and Playwright infrastructure changes, global/shared runtime changes, unmapped files, an invalid mapping, or an unavailable base comparison. Say why the selector chose a full run. Do not present a subset as sufficient coverage.- When
RUN_ALL_TESTS=false,RELEVANT_TESTSis the space-separated set of paths relative tosrc/test/playwright. It includes module tests, declared cross-module consumer workflows, changed specs and the always-run smoke specs (e2e/Login.spec.ts,e2e/Logout.spec.ts,e2e/SystemHealth.spec.ts). REMAINING_TESTScontains the disjoint rest of the discovered suite. Computing this set does not mean CI ran it. Ordinary eligible internal PRs run Phase 1 only and intentionally omit Phase 2. Full PR mode (run_full_suite=trueon the reusable workflow) runs Phase 2 after Phase 1 passes its gate. Locally, use selected scope only when the resolver authorizes it, including eligible application-source changes. Do not reduce the selection to the changed module's folder: the consumer workflows are part of its coverage.
Two things about the input:
- It diffs commits, not the working tree. The script compares
<base>...HEADfrom the merge base. Uncommitted changes are invisible; identify their affected specs separately. Do not commit only to make the selector work. No committed changes or a failed base comparison selects all tests. Renames classify both the old and new paths; deleted specs are omitted from test arguments. - Use the pull request's actual base. The base is the first argument.
origin/developis only correct for a pull request that targetsdevelop.
CI sets run_full_suite=false for ordinary internal PR runs. The selector can still require all
E2E tests, which run in one full-suite job for global runtime, helpers and unknown inputs. The mapping's
dependency contract must account for every discovered spec before partial coverage is allowed. Adding
a new unassigned spec requires updating the contract. Java/client unit tests and documentation alongside
a mapped source change do not independently force full E2E.
Fork PRs do not start E2E; draft and test/documentation-only PRs can also skip it through the umbrella
workflow's change filters.
Non-PR E2E runs use the full suite: integration-branch pushes, release events, merge-group events when a merge queue is configured, and manual CI runs. A green selective PR result does not prove a full E2E run, and merge-group coverage is not guaranteed for every PR. To request full CI for a branch explicitly:
gh workflow run ci.yml --ref <branch> -f force_build=trueThe PR comment and job summary identify selected versus full scope. Report the scope actually executed, including intentional Phase 2 omission.
Step 2: choose the runner
Default to the single-node runner. It is faster and it is what most changes need. When step 1
requires the full suite, omit --specs from the chosen runner command.
./supporting_scripts/e2e/run-e2e-tests-local-fast.sh --specs "<RELEVANT_TESTS from step 1>"Use the multi-node runner instead when the diff touches cluster-sensitive code, because a single node cannot reproduce cross-node failures at all:
./supporting_scripts/e2e/run-e2e-tests-local-multinode-fast.sh --specs "<RELEVANT_TESTS from step 1>"Treat a change as cluster-sensitive when it touches any of:
src/main/java/de/tum/cit/aet/artemis/core/service/distributed/or any caller ofDistributedDataProvidersrc/main/java/de/tum/cit/aet/artemis/core/config/cache/- the build job queue and dispatch in
src/main/java/de/tum/cit/aet/artemis/localci/ - websocket broker or scheduling configuration
If the change is specifically about the distributed data abstraction, run the suite on both
providers. Redis has to pass the same tests as Hazelcast, and with --middleware redis no Hazelcast
instance is created at all, which is what makes it a genuine test of the abstraction:
./supporting_scripts/e2e/run-e2e-tests-local-multinode-fast.sh --middleware redis --specs "<paths>"Step 3: re-runs
The runners keep services alive between runs. After the first run, reuse them:
./supporting_scripts/e2e/run-e2e-tests-local-fast.sh --skip-server --skip-client --skip-db --specs "<paths>"For the multi-node runner the equivalent is --skip-build --skip-up.
The runners can stop services they did not start. Use --stop only after you have identified
the services and confirmed that the runner owns them. It also tears down its database.
Step 4: interpret the result
A red spec is not automatically a bug in the change. Work through these before concluding anything.
Does it already fail on develop? Some specs fail only in develop's full-suite job, so a green run on a pull request proves less than it looks and a red one may be pre-existing. Check the same spec on develop before attributing the failure to the branch.
Is the client under test actually this branch? The runners reuse whatever is serving ports 9000 and 8080. A dev server left running from another branch will serve different code and the failure will make no sense. If a failure looks impossible, verify what is actually being served before debugging further.
Is the assertion counting shared state? Tests run across parallel workers against one server, so anything asserting on a server-wide counter must assert a lower bound, not an exact value or an exact delta. A single-worker local run hides this class of bug entirely, so a test that passes locally and fails in CI with an off-by-a-few count is usually this.
Does the test assert exact request counts for one navigation? The page fixture enables navigation recovery by default and may reload a slow route. Set test.use({ recoverNavigation: false }) for that contract and wait for the actual page content. This keeps automatic recovery from adding cancelled requests to the count; retain the exact request assertions and existing timeouts.
Never fix a flake by raising a timeout. Raising a timeout hides the race rather than fixing it, and the test stays flaky in CI where the machine is slower and more loaded. Find what the test is actually waiting for.
Is the failure just a bad --specs path? Paths are relative to src/test/playwright. A typo
makes Playwright print Error: No tests found. and exit non-zero, which the runner reports as a
failed run. So a red run with no test output at all is a path problem, not a test problem. Check
the executed count against what step 1 selected before reading anything else.
Writing a layout contract
To pin how big an element is, how elements line up or that a page does not scroll sideways, measure in the browser. Never compare screenshots, and never find an element by a styling class.
- State the rule as a number or as a relation between elements.
- Find the elements by
data-testid; add one to the template when it is missing. - Assert inside
forEachViewportfromsrc/test/playwright/support/layout.tswithexpectHeight,expectAligned,expectFillsParent,expectInside(with amarginfor air),expectInset(name the scroll containers asscrollers, so that their scrollbars are not counted as inset),expectGap(the divider between two panels),expectBelow,expectNoOverlap,expectInsideOrBelow(a layout that depends on the room),expectWithinViewport,expectNoHorizontalOverflow,expectNoHorizontalScrollAroundorexpectSameComputedStyle, and give each element aname. A page in a card that scrolls on its own needs both sideways-scroll helpers, because the document keeps the width of the window. A click that must not move the page runs insideexpectScrollPositionKept. A divider that is dragged keeps the width it was given in px, which a smaller window does not undo, so drag it after the viewport matrix. - For a new or changed helper, add a case that holds and one that must fail to
src/test/playwright/e2e/shared/LayoutHelpers.spec.ts.
A change under src/test/playwright/support/ makes the resolver in step 1 report
RUN_ALL_TESTS=true. The helpers, the viewport matrix and the exam example are in the section
"Layout contract tests" of documentation/docs/developer/e2e-testing-playwright.mdx.
Reporting back
State which specs ran, which passed, and which failed. If step 1 reported RUN_ALL_TESTS=true and
a subset was run anyway, say so plainly. Do not describe a change as E2E tested when the selection
was narrowed for time.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

