texera

<!--
  ~ Licensed to the Apache Software Foundation (ASF) under one
  ~ or more contributor license agreements.  See the NOTICE file
  ~ distributed with this work for additional information
  ~ regarding copyright ownership.  The ASF licenses this file
  ~ to you under the Apache License, Version 2.0 (the
  ~ "License"); you may not use this file except in compliance
  ~ with the License.  You may obtain a copy of the License at
  ~
  ~   http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing,
  ~ software distributed under the License is distributed on an
  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  ~ KIND, either express or implied.  See the License for the
  ~ specific language governing permissions and limitations
  ~ under the License.
-->

# AGENTS.md

## Architecture Map

Apache Texera: Scala/sbt backend services + the Amber workflow execution
engine, an Angular UI, and the agent service. JVM modules wired in
[`build.sbt`](build.sbt).

| Area | Path | Detail |
| --- | --- | --- |
| Workflow execution engine (Amber) | `amber/` | [amber/README.md](amber/README.md) |
| Backend services | `config-service/`, `access-control-service/`, `file-service/`, `computing-unit-managing-service/`, `workflow-compiling-service/`, `notebook-migration-service/` | `build.sbt` |
| Shared Scala libs | `common/` (`auth`, `config`, `dao`, `util`, `workflow-core`, `workflow-operator`, `pybuilder`) | `build.sbt` |
| Frontend (Angular) | `frontend/` | [frontend/README.md](frontend/README.md) |
| Agent service (Bun/TS, LLM agents) | `agent-service/` | `agent-service/package.json` |
| Pyright language service | `pyright-language-service/` | [pyright-language-service/README.md](pyright-language-service/README.md) |
| Deploy scripts / Dockerfiles | `bin/` | [README](bin/README.md) / [k8s](bin/k8s/README.md) / [single-node](bin/single-node/README.md) |
| DDL, sbt plugins | `sql/`, `project/` | files therein |

### Amber breakdown

| Path | Role |
| --- | --- |
| `amber/src/main/scala` | Pekko actors, scheduler, reconfiguration, fault tolerance, gRPC/proto |
| `amber/src/main/python/pyamber` | Python engine (`pyamber`) — bridge to the Scala engine |
| `amber/src/main/python/pytexera` | Python operator SDK exposed to UDFs |

## Where Things Live

| Topic | Source of truth |
| --- | --- |
| Contribution / PR / lint / format / testing / license header | [CONTRIBUTING.md](CONTRIBUTING.md) |
| Reporting security issues | [SECURITY.md](SECURITY.md) |
| PR template | [.github/PULL_REQUEST_TEMPLATE](.github/PULL_REQUEST_TEMPLATE) |
| Issue templates | [bug](.github/ISSUE_TEMPLATE/bug-template.yaml) / [task](.github/ISSUE_TEMPLATE/task-template.yaml) / [feature](.github/ISSUE_TEMPLATE/feature-template.yaml) |
| License-header coverage; vendored `workflow-operator` | [.licenserc.yaml](.licenserc.yaml); [project/AddMetaInfLicenseFiles.scala](project/AddMetaInfLicenseFiles.scala) |
| Run the local dev stack (infra in Docker; backend/frontend/agent-service native) | [bin/local-dev.sh](bin/local-dev/README.md) |
| Single-node / k8s deploy | [single-node](bin/single-node/README.md), [k8s](bin/k8s/README.md) |

If a topic is above, **read that file** instead of asking here.

## Agent-Specific Rules

### Scope and safety

- Narrowly scoped changes. No unrelated rewrites or cross-service moves.
- `git status --short` before editing; don't revert unrelated dirty files.
- Never commit secrets / local config / build output / caches / binaries
  (`python_udf.conf`, `.env`, `target/`, `dist/`, `.pytest_cache/`,
  `.ruff_cache/`, logs).

### Develop in a worktree

Leave `texera/` on `main`. One worktree per PR, branched off a freshly
fetched `upstream/main`.

```
texera/                      # stays on main, never dirty
texera-worktrees/<branch>/   # one worktree per PR
```

Reset to `upstream/main` at start; `git log upstream/main..HEAD` should
contain only this PR's commits before pushing; remove the worktree after
merge.

Prefer [`bin/local-dev.sh`](bin/local-dev/README.md) to run the stack while
developing. Its native services bind fixed ports and share one PID/state dir,
so only one worktree's stack runs at a time: `bin/local-dev.sh down` in the
old worktree before switching, then `up` in the new one. Use the
non-interactive CLI subcommands (`up` / `down` / `status` / `logs`); the
interactive TUI (`-i`) is for humans, not agents.

### Environment

| Component | Version |
| --- | --- |
| Java | JDK 17 |
| Scala | 2.13 |
| Python | 3.12 |
| Node | 24 |

One Python venv shared across worktrees, sibling of the texera checkout:

```
<workspace>/
├── texera/                   # main checkout
├── texera-worktrees/<br>/    # per-PR worktrees
└── venv312/                  # shared Python 3.12 venv
```

```bash
python3.12 -m venv ../venv312 && source ../venv312/bin/activate
pip install -r amber/requirements.txt -r amber/operator-requirements.txt
# For pytest or running bin/python-proto-gen.sh, also install dev deps:
pip install -r amber/dev-requirements.txt
```

Tests that spawn Python workers need an interpreter path. Edit `python.path`
in [`udf.conf`](common/config/src/main/resources/udf.conf) or
`export UDF_PYTHON_PATH="$(pwd)/../venv312/bin/python"` (env var overrides).
Without it, `sbt` Python-integration tests fail to launch a worker.

[`.jvmopts`](.jvmopts) holds every `--add-opens` flag Texera needs for
JDK 17+, with each group annotated by its upstream source (Kryo,
Apache Arrow, Apache Pekko). sbt's launcher and the [`.run/`](.run)
configs read it automatically; for raw `java` launches, pass it as an
argfile: `java @.jvmopts -jar …`. If a future library version or a new
code path triggers an `InaccessibleObjectException`, add the open to
`.jvmopts`. [`project/JdkOptions.scala`](project/JdkOptions.scala)
will propagates the changed options to forked test JVMs, sbt-native-packager dist launchers,
and IntelliJ.

### Branch and commit naming

Short, **Conventional Commits**, same shape for branch and commit subject.

| Kind | Branch | Commit |
| --- | --- | --- |
| Feature | `feat/agent-workflow-edit` | `feat(agent-service): enable workflow edit` |
| Bug fix | `fix/marker-replay` | `fix(amber): marker replay during reconfiguration` |
| Tests | `test/pyamber-handlers` | `test(pyamber): add handler unit tests` |
| Chore | `chore/angular-21` | `chore(deps, frontend): upgrade to Angular 21` |
| CI | `ci/merge-queue-stacking` | `ci: stack merge-queue builds by module` |

Both ≤ ~60 chars. For code changes, if you use a scope, use the module name
(`amber`, `pyamber`, `frontend`, `agent-service`, `file-service`, …) — not
`amber-python`. No `Co-authored-by:` trailer for the repo owner.

**Choosing the type** turns on what happens to the behavior, not on how big
the diff is:

| The change | Type |
| --- | --- |
| Worked before, broken now | `fix` |
| Support never existed; adding it | `feat` |
| Support exists; removing it | `feat` |
| Reworked so user-facing behavior intentionally changes | `feat` |
| User-facing behavior unchanged | `refactor` |

Behavior is what the code does, not what a doc or an old PR description claims
it does: implementing something that was never actually there is a `feat`.

`refactor` claims the **user-facing** behavior is identical. Tests that pin a
user-facing API must pass untouched — editing one of those assertions means
the behavior moved, so it is a `feat` or a `fix`. Tests that pin internals (a
private helper's signature, call order between collaborators, the shape of an
intermediate value) mirror the implementation, so rewriting them alongside the
code they mirror is still a `refactor`.

**Tests.** A test-only PR is `test(<module>): ...`. Repairing a broken or
flaky test is a bug fix in test code: `fix(test, <module>): ...`.

**Dependencies.** `fix` only when the bump carries a security fix:

| Bump | Commit |
| --- | --- |
| Patches a CVE | `fix(deps, <module>): ...` |
| Everything else | `chore(deps, <module>): ...` |
| GitHub Actions | `chore(deps, ci): ...` |

Omit the module for cross-module bumps (sbt). GitHub Actions bumps take `ci`
as their module — that is what [`.github/renovate.json5`](.github/renovate.json5)
opens them with; a bare `ci: ...` is for hand-written CI and workflow changes.

**Backports.** A PR targeting `release/vX.Y` appends the version as the last
scope component — `fix(deps, frontend, v1.2): ...`. Version tags belong only
on release-branch PRs, never on one targeting `main`.

### Issues and PRs

Issue-first; both stay short.

```
issue (template + Type)  ->  PR (Closes #N, template)  ->  review  ->  merge
```

- Every change starts as an issue (minor typo / docs excepted). File against
  `apache/texera`, never a fork.
- Pick the right template **and** set the GitHub Issue **Type** explicitly
  (`Bug` / `Task` / `Feature`); the template's `type:` frontmatter doesn't
  always apply on creation.
- Reference the issue: `Closes #N` (or `Fixes` / `Resolves`, or "related to").
- Issue titles are **plain prose**; never use the Conventional Commits
  format (`type(scope): ...`) — that prefix is for commit and PR titles only.
- Task issues match `task-template.yaml` exactly.
- Prefer **tables** and small **ASCII diagrams** over long bullets. Don't
  restate the diff or the template.
- For bugs, lead with **root cause** and a **before -> after** sketch:
  ```
  Before:  reconfiguration -> replay marker -> worker hangs
  After:   reconfiguration -> replay marker -> resume from checkpoint
  ```
- **Frontend PRs**: any visible UI change requires screenshots / GIF,
  **before / after** side by side. For purely visual fixes that's the
  primary verification under "How was this PR tested?"; interactive flows
  also list manual steps (click path, browser, viewport).

### Tests come first

TDD. Write the test before the source change.

```
write/adjust test (red)  ->  edit source (green)  ->  refactor
```

| Situation | Order |
| --- | --- |
| New feature / behavior change | Failing test, then implement. |
| Bug fix | Regression test reproducing the bug, then fix. |
| Code with **no tests** | **Characterization tests** pin current behavior first; only then change source. |
| Refactor (no user-facing behavior change) | Tests stay green throughout. User-facing API assertions stay untouched; tests that mirror internals may be rewritten with the code. |

Every test must cover:

- **Both directions**: positive (valid → expected) **and** negative (invalid
  / error → specific failure mode).
- **Edge cases**: empty / null / zero / max / boundary, unicode,
  concurrency/order, missing or malformed config.
- **Don't assume valid.** External input (user / API / file / message) must
  be tested with bad input.

Don't claim "tested" without commands. Paste the exact `sbt testOnly` /
`pytest` / `yarn test:ci` / `bun test` invocation under "How was this PR
tested?".

### CI labels & gating

CI runs are **selected by PR labels**, not by file diff.

```
diff -> pr-labeler -> labels on PR -> required-checks maps labels to stacks -> CI runs
```

- Path → label rules: [`.github/labeler.yml`](.github/labeler.yml)
- Label → stacks (`LABEL_STACKS`, source of truth):
  [`.github/workflows/required-checks.yml`](.github/workflows/required-checks.yml).
  Read it directly; don't duplicate the mapping here.
- Need extra coverage the diff doesn't imply (e.g. a `common/` change you
  suspect breaks the frontend)? **Add the relevant label manually**.
- Empty stack union (docs-only / dev-only / `dependencies` / `feature` /
  `fix` / `refactor` / `release/*` only) skips every build stack on purpose.
- `release/*` labels nominate backport targets. A nominated target is
  backported only once that branch's release manager — listed in
  [`.github/release-branches.yml`](.github/release-branches.yml) — approves the
  PR, and the required `Backport Approvals` check blocks the merge until every
  `release/*` label on the PR is approved. A manager declines by removing their label, so
  the labels on a merged PR are exactly the branches it was backported to.