Write Robot Framework Tests

These rules apply to acceptance tests in the atest/ folder.

Rules

Tests and user keywords must be self-explanatory

Test case names and keyword calls must read like plain English. No reader should need to look at an implementation to understand the intent. Choose names that describe what is being verified, not how.

Good

*** Test Cases ***
Login Fails With Wrong Password
    Go To    ${LOGIN_URL}
    Fill Text    input#username_field    wrong_user
    Fill Secret    input#password_field    $wrong_password
    Click    css=input#login_button
    Get Text    css=.error-message    ==    Login failed. Invalid user name and/or password.

Bad — name says nothing specific, comment compensates for a weak name

*** Test Cases ***
Test Login
    # Verify that login with incorrect credentials shows an error
    Go To    ${LOGIN_URL}
    Fill Text    input#username_field    wrong_user
    Fill Secret    input#password_field    $wrong_password
    Click    css=input#login_button
    Get Text    css=.error-message    ==    Login failed. Invalid user name and/or password.

No [Documentation] in test cases or user keywords

Test cases and user keywords defined in .robot files must not use [Documentation]. The name and structure must be sufficient. Inline comments (#) are acceptable sparingly for genuinely non-obvious details — they are the exception, not the rule.

Good

*** Keywords ***
Login User
    Fill Text    input#username_field    demo
    Fill Secret    input#password_field    $mode
    Click    css=input#login_button

Bad

*** Keywords ***
Login User
    [Documentation]    Fills in credentials and submits the login form.
    Fill Text    input#username_field    demo
    Fill Secret    input#password_field    $mode
    Click    css=input#login_button

Library keywords may have documentation

Python keywords in atest/library/ and Browser/keywords/ can and should have docstrings. These are discoverable via libdoc and IDE tooling.

def child_test_status_should_be(
    result: Result, name: str, status: str, message_pattern: str | None = None
) -> None:
    """Verify status, and optionally the failure message, of a child run test."""
    ...

Complex logic belongs in library keywords

If a keyword requires nested branching, or loops, those belong to Python library. Complex data transformation, or non-trivial orchestration, should be also implemented as a Python keyword in atest/library/ — not as a user keyword in a .robot file. Robot Framework syntax is not suited for complex logic.

Good — complex assertion delegated to Python

*** Test Cases ***
Scope Settings Are Restored After Test
    Log All Scopes    exp_timeout=10    exp_strict_mode=True

# atest/library/scope_logger.py
def log_all_scopes(exp_timeout: float, exp_strict_mode: bool) -> None:
    b: Browser = BuiltIn().get_library_instance("Browser")
    timeout = b.scope_stack["timeout"].get()
    assert timeout == exp_timeout, f"timeout {timeout} != {exp_timeout}"

Bad — logic squeezed into user keyword

*** Keywords ***
Verify Scope Timeout
    [Arguments]    ${expected}
    ${actual}=    Evaluate    ...complex expression...
    Run Keyword If    '${actual}' != '${expected}'    Fail    ...

Library keywords must have type hints

All Python keyword functions in atest/library/ must have type annotations on parameters and return values.

Good

def start_test_server() -> str: ...


def child_test_status_should_be(
    result: Result, name: str, status: str, message_pattern: str | None = None
) -> None: ...

Bad

def start_test_server(): ...


def child_test_status_should_be(result, name, status, message_pattern=None): ...

Do not assert elapsed wall-clock time

CI runners stall, so a test that measures how long something took is flaky, and an upper bound like < 5000 proves little about the value it means to check. To check a timeout, read the value Playwright reports in its error, Timeout 1000ms exceeded. Where nothing reports the value, assert only a lower bound: a loaded runner makes a wait longer, never shorter.

Good — exact value from Playwright's error (atest/test/08_Scope_Tests/scope_keywords.resource)

*** Keywords ***
Timeout Should Be
    [Arguments]    ${expected_ms}
    Run Keyword And Expect Error
    ...    *Timeout ${expected_ms}ms exceeded*
    ...    Click    id=not_here

Bad — elapsed time with a range

*** Test Cases ***
Timeout Is Set
    Set Browser Timeout    1s
    ${elapsed} =    Milliseconds Until A Wait Fails
    Should Be True    500 < ${elapsed} < 1700

Avoid deeply nested user keywords

The Browser library already provides rich, high-level keywords. Avoid building deep stacks of user keywords that wrap other user keywords. Test-suite-level keywords (in a *** Keywords *** section of a .robot file) are fine for encapsulating setup/teardown or repeated actions within that suite.

Good — one thin suite-level keyword delegates directly to Browser keywords

*** Keywords ***
Open Browser To Login Page
    New Browser    ${BROWSER}    headless=${HEADLESS}
    New Page    ${LOGIN_URL}

Bad — unnecessary nesting: suite keyword → user keyword → user keyword

*** Keywords ***
Open Browser To Login Page
    Open Browser With Settings
    Navigate To Login

Open Browser With Settings
    New Browser    ${BROWSER}    headless=${HEADLESS}

Navigate To Login
    Go To Login Page

Go To Login Page
    New Page    ${LOGIN_URL}

File locations

Tags that keep a test out of a run

inv atest excludes tests by tag. Tag a new test that cannot run everywhere:

Running acceptance tests

inv atest

Lint after changes:

inv lint-robot