Gale Development Guide

Gale is a Wado-native ANTLR4-compatible parser generator. The rules below hold for every change; the rest lives in companion docs:

• antlr4-compatibility.md — the compatibility contract, prediction / codegen design, soundness invariants, descriptor pipeline, and triage.
• resilient-parser.md — error-resilient parsing and the flat CST.
• import.md — grammar composition: how import S; resolves and what a delegate contributes.
• testing.md — the test layers and the oracle procedures.
• debugging.md — gale dump, the trace option, and the Rust corpus check.
• perf.md — performance notes: budget, levers, and measured perf dead-ends.
• dead-ends.md — prediction, predicate and lexer approaches that failed. The static path always has edges; the complete answer is the runtime ATN simulator. Read the area's section before retrying anything in it.
• TODO.md — open work.
• README.md and WEP: Gale — overall design context.

Setup

The descriptor corpus, .g4 semantics, and test regeneration all depend on the ANTLR4 submodule:

git submodule update --init --recommend-shallow vendor/antlr4

Files headed // Do not edit by hand are generated by mise run extract-gale-descriptors from their source (e.g. status.toml).

The oracle scripts spawn java and javac, which core:process reaches only under /usr and /etc by default. A JDK elsewhere (mise's, say) is opened through [process] read in ~/.config/wado/config.toml (read = ["~/.local/share/mise/installs"]) or --process-read.

Compatibility Principle

Gale accepts every well-formed grammar upstream antlr4 accepts; a real-world .g4 that ANTLR4 accepts but Gale rejects is a Gale bug. The one exception is import Foo = Bar; (claim (a) in antlr4-compatibility.md).

• A bug in ANTLR4 itself is not reproduced: a test pins the corrected answer and says why, by an upstream report or the jar contradicting itself. See "Bugs in ANTLR4 itself" in antlr4-compatibility.md.
• Compatibility is a capability contract, not byte-for-byte output. Parse trees, tokens, and semantics must match; incidental rendering differences that carry no structure may diverge (e.g. the <EOF> marker in toStringTree()).
• Gale is a superset: it may accept grammars ANTLR4 rejects only when the meaning is uniquely determined by Gale's language model — never an invented behavior. When accepting would require guessing, reject loudly. Each such grammar is listed, with the meaning Gale gives it, under "Grammars Gale accepts and ANTLR4 rejects" in antlr4-compatibility.md; a new one goes there too, confirmed against the jar.
• TDD every g4 change with a unit test in src/g4/{lexer,parser}_test.wado. If an existing test encodes a wrong expectation, fix the test — the spec wins; confirm against the published jar as a black-box oracle.

License Hygiene

ANTLR4 is BSD-3; copying or paraphrasing its implementation risks making Gale a derivative work. So:

• Do NOT read ANTLR4 implementation source: vendor/antlr4/tool/**/*.{java,g} and vendor/antlr4/runtime/**/*.java (e.g. ParserATNSimulator.java, LL1Analyzer.java, the bootstrap .g grammars). Algorithmic ideas inferred from that source belong to ANTLR4, not Gale.
• OK to read: .g4 files anywhere under vendor/antlr4/; runtime-testsuite/**/*.txt descriptors; vendor/antlr4/doc/*.md (spec-like prose — the canonical .g4 semantics reference; a curated index is in antlr4-compatibility.md).
• OK to run: the published antlr-4.13.2-complete.jar as a black-box oracle.

The first rule is enforced: permissions.deny covers the Read tool, .claude/hooks/vendor-license-guard.mts covers Bash.

Codegen

• Backtrack only where it makes the parse faster — parser or lexer. The rule is about speed: a try-fail-retry loop used to settle what prediction cannot usually costs more than predicting. So disambiguate with static k-token lookahead, and route a decision static prediction cannot resolve in depth 5 to the runtime ATN simulator. A backtrack that replaces more expensive work is allowed, such as parsing directly and rewinding on failure instead of scanning ahead and then parsing again. Show it is faster with a benchmark, and keep the tree the same as the jar's. The repeat-exit probe also rewinds, but it decides nothing: it re-parses a failed element under speculating to record where the error is, and rolls back all but the message. Mechanics, soundness invariants, and ATN escalation: antlr4-compatibility.md (Prediction & codegen design).
• Keep generated code byte-identical for grammars that do not use a feature (actions, FOLLOW gates, ATN, trace) — gate every emit site on the feature.
• Keep the ATN off any shape the static path already lexes or predicts as the jar does: an ATN-class rule inlines about a thousand lines of runtime into the generated parser. A change to what routes to the ATN, or to a static prediction trigger, runs the shapes it newly includes and newly excludes through mise run oracle-gale <grammar.g4> <start_rule> < input (--tokens for the lexer; scripts/antlr4_oracle.wado) before it is reported. The script caches the jar under build/antlr4/.
• A lexer emit decision lives in lexer_rule_routes and lexer_rule_plan, which gen_lexer and gale dump --lexer both read. A new strategy is a new plan node, never a branch in one consumer. See debugging.md.

Inlined Runtime

The generated parser inlines the runtime fragments in src/runtime/*.wado (lex, diag, tree, tools always; follow / scan_memo / highlight / atn / atn_predict / atn_lr / latn gated per-feature). Each fragment is also a real module for dev / test. A standard-library use in one is hoisted to the top of the generated file. A type only the generator needs lives outside src/runtime/, as the .g4 lexer's Token does.

Every byte of these files lands in every generated parser, so:

• No comments. They would be copied into hundreds of generated files. State intent through names, decomposition, and asserts.
• Nothing test-only. String-level comparison helpers live in tests/support/tree_compare.wado; only a helper taking a generated type (to_lexer_string, over the generated TokenStream) has to stay.
• A map hashes under a fixed HashSeed (LATN_SEED in latn.wado) rather than taking one from new(). A seed from the host would make every generated parser import wasi:random/insecure-seed.

Tests

wado test package-gale                         # the whole package
wado test package-gale/src/codegen_test.wado   # one file

wado test package-gale is the whole check for Gale work; mise run test-wado leaves this package out. CI runs it as test-gale-o2, which also catches the miscompiles a compiler change slips past the e2e fixtures.

• Pass the package directory, never a hand-written glob. The descriptor corpus sits one directory deeper (tests/antlr4-compat/drivers/<Category>/), so a flat glob passes on about a third of the suite, and the fixtures it never reaches drift behind the generator.
• A test of anything that imports codegen.wado (generate, the dump, the entry points) belongs in src/codegen_test.wado, and one of a pass below it (lowering, the lexer emit, the ATN, the Java translator) in src/lower_test.wado. Each file a test reaches the generator from compiles that part of it again, so the two files pay for it once each. Nothing enforces this; a slow compile in wado test output shows the breach.
• A codegen test asserts what the generated parser does, with the expected trees taken from the jar. Match the source text only for what running cannot show: that an unused feature emits nothing, or which strategy the emitter picked.
• A new e2e grammar goes in tests/grammars/ with a parse test in src/g4/integration_test.wado and a driver test importing it through the generator. It opens with a comment saying which shape it pins and why that shape is hard. A grammar taken from elsewhere also carries // Source: (its URL) and // License:; one written here carries neither.