AGENTS.md — bulk_executor

Operating manual for AI agents working in this directory.

Layout

server/src/         Glue-job-side code (runs on Spark workers)
client/src/         CLI-side code (runs on developer machines)
tests/server/       Mirrors server/src/ tree
tests/client/       Mirrors client/src/ tree
Makefile            Single source of truth for test commands
pytest.ini          Root config; sets pythonpath so both src trees resolve
AGENTS.md           This file — the operating manual
CLAUDE.md           Committed symlink to AGENTS.md; one file, two names

CLAUDE.md is a committed symlink to AGENTS.md (mode 120000), so tools that look for either name read the same content. Always edit AGENTS.md — never replace the symlink with a copy, or the two silently drift.

Running tests

Always use the Makefile:

make install        # First time only; creates .venv
make test           # All tests + coverage summary
make test-server    # Server-side only
make test-client    # Client-side only
make coverage       # Full term-missing coverage report

Never invoke pytest with --cov=python_modules. The two source roots are server/src and client/src — coverage flags must use those paths.

AI lint (on-demand, agent-run)

ai_lint/ holds AI-driven checks for invariants that are easy to verify by reading code but very hard to express as a deterministic unit test (e.g. "the README, the role-creation code, and the custom-role validator must all agree on the Glue job's IAM permissions"; "every command that touches DynamoDB must be rate limited").

There is no runner — you (the agent) are the engine. When the user asks to "run the AI lint" (or names a specific rule):

1. Read every ai_lint/rules/*.md. Each file is one rule: the invariant, where to look, and how to report.
2. For each rule, inspect the current code with your normal tools — glob/grep/read. Rules that say "discover the verbs" mean it: enumerate live so newly added code is covered; never work from a stale hard-coded list.
3. Report per-rule findings, and state what you verified even when clean (so a pass is trustworthy).

This is advisory — never a merge gate, never part of make test. A false alarm is feedback, not failure: if a finding turns out fine, tighten the rule's .md so the next run is sharper. Add a rule by dropping a new rules/<name>.md (one per file keeps merges clean). See ai_lint/README.md.

End-to-end tests (real AWS — opt-in)

make test is offline (awsglue/pyspark mocked). The e2e suites are separate, hit real Glue + DynamoDB, and never run under make test:

make test-e2e-connector   # count/find/sql/load via the live DynamoDB connector
make test-e2e-commands    # fill/update/delete/copy/diff against transient tables
make test-e2e-security    # documented bootstrap IAM policy actually bootstraps

Before touching the e2e harness, read tests/e2e/AGENTS.md — it documents the invariants (real-AWS guardrails, the transient-table contract, the shared-Glue-job hazard, what makes a smoke trustworthy vs false-green). The e2e tests require AWS credentials and a one-time ./bulk bootstrap; they cost real money and run real Glue jobs (~2 min each), so do not run them in tight loops.

Updating the badges in README.md

The top of README.md carries three badges (test count, line coverage, branch coverage). They are static — not auto-generated by CI.

Do NOT update the badges inside a feature/fix/test PR. Lines 3-5 of README.md are a single hot spot that every PR would touch, so bumping them per-PR guarantees merge conflicts — and because PRs merge in an unpredictable order, the numbers a PR writes are stale by the time it lands anyway. Leave README.md out of your change entirely; slightly-stale badges are fine and expected.

Instead, refresh all three in their own dedicated badge-refresh PR (touching only README.md), done occasionally when someone wants the numbers current — never wired into a substantive change.

When you do a refresh PR:

1. Run make test (the full offline suite — not test-client, test-server, or any e2e target; e2e never counts toward the badges). Read the final lines for <N> passed, LINE COVERAGE: <X>%, BRANCH COVERAGE: <Y>%. Use <N> passed verbatim — do not add skipped tests to it. The numbers come from this fresh run, never from the badge's current value or a "+N" bump. Also do not use the percentages printed by make test-client / make test-server / make coverage — only the full make test totals feed the badges.
2. Edit lines 3-5 of README.md. URL-encode % as %25 and spaces as %20.
3. Pick a color: brightgreen for ≥90%, green for ≥75%, yellow for ≥60%, orange for ≥45%, red below.

Example (illustrative format only — always use your own fresh make test numbers, not these):

![tests](https://img.shields.io/badge/tests-1330%20passing-brightgreen)
![line coverage](https://img.shields.io/badge/line%20coverage-94.1%25-brightgreen)
![branch coverage](https://img.shields.io/badge/branch%20coverage-91.7%25-brightgreen)

Writing tests

Mirror tests/server/test_fill.py — it is the canonical pattern.

• pytest, never unittest.
• Multi-line module docstring at the top naming each function/branch the file covers.
• Group related tests in Test<FunctionName> classes.
• Test names: test_<what>_<when>_<expects>.
• Mock external deps with unittest.mock.patch / MagicMock. Use monkeypatch for env vars, tmp_path for filesystem, capsys/caplog for stdout/log assertions.
• Do not mock the function under test — only its dependencies.
• Don't add fixtures unless shared across 3+ tests.

conftest patterns

tests/server/conftest.py mocks awsglue and pyspark modules globally before collection, so any server-side test can import a verb without those packages installed. If you need a real subclass of pyspark.AccumulatorParam, use the _AccumulatorParamStub already defined there.

tests/server/fill/conftest.py stubs faker because multi_entity_relationship.py imports it at module load. Apply the same pattern (pre-collection module stubbing) when a source file's top-level imports require a heavy optional dependency.

Coverage rules

• Project target: ≥99% line, ≥96% branch.
• Never modify source code to chase coverage. Tests-only PRs must change zero files outside tests/, the Makefile, pytest.ini, .gitignore. (Do not touch README.md badges here either — see "Updating the badges".) Document any unreachable branches in the PR description and move on.
• When you find an uncovered line that's only reachable through a source bug (e.g. a regex group typo, dead elif), do not "cover" it — note the bug in the PR description. Fixes land in a separate source-only PR.

PR splitting rule

Tests-only PRs and source-refactor PRs must stand independently on main. Never stack one on the other. If you find tests that depend on an unmerged source change, either:

• Hold those tests for a follow-up PR that lands after the source PR, or
• Use the pre-refactor source code from git history when writing the standalone tests.

Known unfixed source bugs

Currently none. All six entries this table once held were resolved under #298 (#310, #311, #312, #313, #314). If you add one, follow the shape below plus three rules the old list taught the hard way:

• Verify an entry before acting on it. One of the six described a bug that did not exist: it claimed an if should be an elif because the else swallowed ExpiredTokenException, but exit() raises SystemExit, so the first branch always terminated first. It sat here for months.
• Check whether the code is even reachable. Another described dead code in a module nothing imported -- the fix was deletion. A third described a log-merging bug that instrumenting a real run proved unreachable: across 318 captured driver events, including a deliberately split 400KB line, it never triggered once.
• Name the function, not the line. That first entry pointed at 386-392, which by then was a different function entirely.

Entries here are excluded from coverage chasing -- document an unreachable branch in the PR description and move on rather than contorting a test to reach it.