ta-lib

# CLAUDE.md - TA-Lib Code Generation Guide

## Architecture Overview

All indicator code is **generated** by a single generator, **`ta_codegen`**
(`ta_codegen/generator/`, Rust): it parses `ta_codegen/input/` → IR → renders
per-backend (C, Java, C#, Rust). The C backend is generated **in place** into
`src/ta_func` / `src/ta_abstract` (the shipped library); the Rust/Java/C#
bindings live under `ta_codegen/output/`. `generate` owns every committed file it
writes — the build-system source lists and the ta-lib.org function pages included
— so "regenerate, then `git status` is clean" is a total gate over the tree.

**Why the C is generated in place and not symlinked** to `ta_codegen/output/c`: a
whole-dir symlink breaks autotools' per-dir libtool recursion (`make` enters the
symlink's *physical* path, so the Makefile's relative `../../libtool` fails with
`Error 127`), and it would force a packaging dereference step. Real files in
`src/` avoid both — and downstream consumers (notably the PHP `trader` extension)
glob `src/ta_func/*.c` straight out of the released source tarball.

**Build separation (important):** the C build systems (CMake + autotools) build
**only C** — the library plus `ta_regtest` and `ta_bench`. `ta_codegen` is Rust,
built and run with cargo via `scripts/build.py`; **CMake never invokes cargo**, so
a C-only setup needs no Rust toolchain.

**Toolchain per backend.** Generating any backend's *source* needs only cargo.
Building and testing one needs that language's toolchain, and a missing tool is a
failure, not a silent skip — narrow with `--backend=` / `build.py --language=`
instead. Java needs a **JDK and `unzip`**: Maven comes from the committed wrapper
(`ta_codegen/output/java/library/mvnw`), which downloads the pinned,
SHA-256-verified Apache distribution itself. `ta_codegen build-libraries
--backend=java` runs `./mvnw clean package` and tests *that jar*, so nothing
tests a class directory and every machine builds with the same Maven. No
credentials are involved (signing and the Central upload sit behind the pom's `release` profile);
only the wrapper's first run needs the network. C# builds with the .NET SDK the
root `global.json` pins, or a later patch of its band when the pin is not
installed. Keep `rollForward` at `patch`: every other value but `disable`
prefers a newer installed SDK over the pin, so CI would build with whichever
newer SDK the runner has, and a different compiler can change the DLL.

Correctness is verified against three baselines: the in-process C library, which
`ta_regtest --codegen` and `--xlang-hash` diff every language server against; the
frozen releases in `ta_ref/` (`scripts/build.py ref`); and the hardcoded
`ta_regtest` expected values.

See `ta_codegen/generator/CLAUDE.md` for generator internals,
`src/tools/ta_regtest/CLAUDE.md` for the test-runner spec, and
`website/src/contribute/README.md` for the process of adding or changing an
indicator, which the `/new-ta-func` skill drives.

### Source of Truth: ta_codegen/input/

One directory per indicator, and the single source of truth for ALL generated
code.

- **YAML** = data, config, enums, IDL. Pure definitions with no logic.
- **C source files** = logic. Anything with computation.
- **No logic in YAML, ever.** No hand-coded string literals for type definitions
  or scaffolding in the codegen either.

`generate` **writes back into this directory**, so a hand edit here is not always
the last word:

- Every `.c` is re-indented in place by a format pre-pass before anything else
  runs. It changes whitespace only, but it does overwrite hand indentation — so
  when the layout comes out wrong, fix `ta_codegen/generator/src/formatter.rs`,
  not the file.
- `internal_error_ids.yaml` hands each C internal-error guard its
  `TA_INTERNAL_ERROR(<id>)`, so the number a caller reports keeps naming the same
  guard across releases. Append-only — a full C run assigns the new keys and
  drops the dead ones; never renumber an existing entry by hand.

Do not hand-edit **generated** files under `ta_codegen/output/` — they are
overwritten on the next `generate`. The converse trap: some hand-written source
lives under `output/` too (the Java shared types, `Core.java` outside the GENCODE
markers, the test suites, the C# `TALib.csproj`, `pom.xml`); the generator
preserves those and never overwrites them. Of `pom.xml` and `TALib.csproj` only
the version is written for you, by `scripts/sync.py`.

### API tiers and entry points

The managed backends have **two** batch tiers: `public OutRange <N>(...)`, which
validates array lengths and throws on a rejection, and `<N>_Impl` — the
transcribed numerics, package-private in Java and `internal` in C#, keeping C's
`RetCode` + out-param shape. There is no third, catch-and-convert tier.

**Two suffixes, one meaning each:**

| suffix | means |
|---|---|
| `_Impl` | the numerics — a transcribed body, nothing else |
| `_Internal` | a **variant** of an entry point, not a tier: the `startIdx`-anchored seams `_OpenInternal` / `_OpenAndFillInternal` |

That is the whole vocabulary, deliberately — one word per concept. C spells them
`<N>_OpenImpl` / `<N>_StepImpl` (plus its own `TA_<N>_ReleaseImpl`; Rust has
`Drop`, the managed backends have GC) and stays the reference spelling below.
Rust, Java and C# recase `<N>` and the verb to their own idiom rather than
mirroring C: Rust `<n>_open_impl`, Java `<n>OpenImpl`, C# `<N>OpenImpl` (`<n>` =
camelCase, `<N>` = PascalCase, the acronym single-capitalized — `Sma`, not
`SMA`). Neither tier is public anywhere, so no runtime gate can see the spelling:
`the_transition_tier_is_step_impl_in_every_backend` pins the *shape* — one word
per concept, in that backend's casing — not one literal string.

**The `_Open*` family is five methods, symmetric, two hops deep** (C's spelling;
the other three carry the same five hops in their own casing):

```
PUB  <N>_Open(in, params)                   -> <N>_OpenInternal(in, 0, params)
PKG  <N>_OpenInternal(in, startIdx, params) -> one sink per output; <N>_OpenImpl(.., 0)
PUB  <N>_OpenAndFill(in, params, outs)      -> aliasing guard; <N>_OpenAndFillInternal(in, 0, ..)
PKG  <N>_OpenAndFillInternal(in, sIdx, ..)  -> <N>_OpenImpl(.., 1)
PRV  <N>_OpenImpl(sp, in, sIdx, params, outBeg, outNb, outs, outStride)
```

Both public entries delegate at anchor 0, so **no seam is emitted unreachable**.
The guard sits on the public frame because that is the only one handed an array
it did not vet: the plain open sinks into fresh arrays, and a composed call's
destination is already proved disjoint by `SubCallStep::is_fusable`. Rust emits
none — two aliasing `&mut` slices are unspellable in safe code. `MA` (Dispatch)
and `MAVP` (PeriodBank) are exempt and hand-roll a body per entry point — theirs
differ by which callee tier they call and by an anchor clamp, not by a stride —
so their `_OpenImpl` takes no `outStride`, which is the discriminator the gates
key on rather than a name list.

A cross-call inside a body calls the callee's *public* tier, in all four
backends, and lets its rejection surface: a throw in Java and C#, an
`Err(RetCode)` in Rust, a returned code in C. The callers that need a code back
convert it themselves: the JSON-RPC servers, and C#'s `ParamHolder.TryCall`
(whose conversion is the direct path's too — its thunk calls the function's own
public overload, like C's frames and Java's `Dispatch`).

**Every metadata tier calls the public tier.** C (`ta_frame.c`), Java
(`Dispatch`), C# (`FunctionCatalog`'s `invoke` thunk) and Rust
(`ParamHolder::call`) all do, so binding a leg shorter than the requested range
is `TA_BAD_PARAM` in three of them and inexpressible in the fourth — C's setters
take a bare pointer and carry no length.

**And a rejected setter leaves the holder as it found it** — the price setter
validates every consumed component before it writes any. The case that makes it
worth a rule is a *re-bind*: on a fresh holder a partial write is masked by the
unbound-component report, but over a bundle that already worked, the rejected
call leaves the components at and after the offending one holding the previous
bundle, and the next call succeeds over a mixture of the two — in C# with no code
and no exception. C is deliberately NOT the ports' whole-struct assignment: its
macro keeps the flag guard so an unconsumed component is skipped rather than
clobbered, which makes the fix a no-op on every call that succeeds.
`metadata_price_setter_validates_before_writing` pins all four on the PR gate,
because the four runtime probes are nightly-only.

## Comments and docs: guidance, not narration

Assume the reader is an AI that can read the code faster than the prose about it.
A comment or a `.md` earns its place only by saying something the code cannot:

- **Default to none.** Code shows *how*. Write a comment only for a *why* a
  careful reader would not reach from the code in a minute.
- **Guidance, not description.** Which invariant is load-bearing, what breaks if
  you change it. Not what the lines below do.
- **Never write these.** A restatement of the lines below. The reasoning that led
  here. An alternative that was rejected. What the code used to be. "Fixed X",
  "per review". A pointer to a doc instead of the fact itself. All of it belongs
  in the issue and the commit message, which are timestamped and do not pretend
  to be current.
- **Never re-explain another file.** That copy goes stale the moment the original
  moves, and costs a reader either a wrong belief or a re-verification. When you
  find one that has drifted, delete it — do not re-sync it. Drift is the symptom;
  the duplication is the defect.
- **A pitfall earns a sentence only if it fails SILENTLY** — and state the rule,
  not the history. "Keep every `(int)` cast the whole right-hand side" beats a
  paragraph naming the emitter that used to get it wrong. Loud failures get
  rediscovered in minutes and need nothing.
- **Named internals are the tell** — a function, a struct field, a symbol out of
  a compiler error. Most likely to be renamed out from under the comment, and
  naming them is usually description wearing a rule's clothes.
- **Doc lines outnumbering the code they describe means cut, not balance.**
  Writing less is the weaker half; re-reading and deleting before you commit is
  the half that works.

Applies to `ta_codegen/input/**` headers and `.md` files as much as to the
generator's own source. `ta_codegen/generator/input_synth/README.md` states the
gate-fixture form of the rule.

## Quick Reference Commands

```bash
# Build (from any directory in the repo; binaries land in bin/)
scripts/build.py                # C library + all C tools (CMake)
scripts/build.py ta_regtest     # Just the C test runner (CMake)
scripts/build.py ta_codegen     # Rust codegen tool (cargo)
scripts/build.py generate       # Regenerate every committed source for all backends —
                                # libraries, JSON-RPC servers, benches (cargo; writes only,
                                # so no JDK or .NET SDK for the Java/C# sources)
scripts/build.py servers        # Generate + compile the JSON-RPC language servers (cargo),
                                # and refresh bin/ta_regtest so bin/ can be driven by hand
scripts/build.py libraries      # Build the publishable Java jars + C# library from the
                                # committed source and run their suites against them

# Test
scripts/build.py regen-check    # The PR gate: regenerating must change nothing
                                # (cargo + Python only; the same command CI runs)
scripts/build.py test           # C reference tests only (quick)
scripts/build.py ref            # C vs each frozen release in ta_ref/ (C-only);
                                # --versions=X_Y_Z[,...] narrows it
scripts/build.py ta_bench_icount # Instruction-count bench (dev-nightly's icount job)
scripts/bench_icount.py         # Run it under callgrind and compare against the
                                # committed baseline (needs valgrind; the script
                                # header says what a count cannot see)
scripts/build.py regtest        # Servers (cargo) + C tests + cross-language verification.

# ta_codegen (run from ta_codegen/generator/)
cargo run -- generate                            # Generate everything, all backends
cargo run -- generate --func=SMA --backend=rust  # Specific function + backend
                                                 # (whole-corpus files — Core.java, the
                                                 # servers, the benches — are skipped)
cargo run -- generate-servers                    # Only the JSON-RPC servers (a narrowing
                                                 # of `generate`, for `build`)
cargo run -- build                               # Compile servers into bin/
cargo run -- build-libraries                     # Build the publishable Java/C# libraries
                                                 # and run their suites (no server, no bench)
cargo test                                       # ta_codegen's own test suite

# ta_regtest directly (from bin/)
./ta_regtest                                     # C reference tests only
./ta_regtest --codegen                           # C tests + all-language codegen verification
./ta_regtest --codegen --language=c,rust --function=RSI,SMA
```

## Cross-Language Regression Testing

`ta_regtest` is the **universal test runner**: rather than linking each language's
compiled code, it drives one generated JSON-RPC server per language over pipes and
compares every call against the C reference.

A **correctness** request goes through each language's PUBLIC API, and the server
normalises the exception back into the `retCode` / `outBegIdx` / `outNBElement`
wire shape; that is the server's job, not the library's. In **Java and C#** a
request declaring itself timed (`"timed":1`, sent only by `ta_bench`) calls the
BODY instead: these servers double as the cross-language benchmark, and nothing
measured may quietly acquire the public tier's argument checks. Rust cannot split,
`tools` being a separate crate.

A new ta_regtest source file must be registered in BOTH `CMakeLists.txt` and the
autotools `Makefile.am` — the dist-verification CI path builds with autotools, so
a missing entry there breaks the nightly. `scripts/build.py check-source-lists`
verifies the two agree. `scripts/synth_gate.py` (nightly, and runnable locally)
covers generator constructs no shipped indicator uses.

## Rust Backend

Generated Rust lives in `ta_codegen/output/rust/` — a Cargo workspace: `library/`
is the shipped `ta-lib` crate, `tools/` holds the JSON-RPC server and bench.
Indicators are methods on a `Core` struct, one file per indicator.

- **The public batch API is `pub fn <N>(...) -> Result<OutRange, RetCode>`, and it
  owns the argument contract** — index range, parameters, then every buffer
  length, answering `BadParam`. Its input bound takes no sub-lookback escape, so
  it is strictly stronger than each body's bounds-assert preamble: a `pub fn`
  call cannot reach an assert that would reject it, on the direct or the composed
  path.
- Indexing is safe: the crate is `#![forbid(unsafe_code)]`, so a violated bounds
  precondition panics rather than being undefined behavior. Only `pub fn <N>` and
  the phantom-I/O sweep reach `<N>_Impl`, so a panic there is a generator bug.
- Rustdoc, including a runnable doctest per function, is generated from each
  function's canonical `<name>.md`. Verify with `cargo doc --no-deps`
  (warning-free) and `cargo test --doc` in the crate.

## Versions and the shared library soname

Edit only `VERSION`. Everything derived from it, plus the shared library version
(`ABI.manifest`, `ABI.released`, `TALIB_LIBRARY_VERSION`), is written by
`scripts/sync.py`: run it any time, as often as you like, and commit what it
changed. It stops if public API that shipped was removed or changed: put it back,
or run `scripts/sync.py --accept-break` when the break is intended, and the soname
changes at the next release.

## Build flags that must stay in step

The generator's flags live in one place (`COMMON_GCC_FLAGS` and `X86_GCC_FLAGS`,
`main.rs`); four are set by all three build systems (CMake, autotools, the
generator) and must stay in step:

- `-ffp-contract=off` — load-bearing for the FMA contract, **not** a performance
  knob.
- `-fno-math-errno` — purely a performance knob, and the one part of
  `-ffast-math` that cannot change a value. Do **not** weaken it to `-ffast-math`
  on the strength of that: the same output-hashing harness shows `-ffast-math`
  changing 70 functions. It is not effect-free either — it lets STDDEV's sqrt map
  vectorize and raise `FE_INVALID` on lanes the scalar guard skipped (values
  unaffected). That, and why clamping the radicand does not fix it, are in
  `CMakeLists.txt` next to the flag.
- `-falign-functions=64 -falign-loops=64`, x86 only — performance knobs that
  change no value. Every function, and every hot loop gcc enters by falling
  through, starts a 64-byte line, so a function's code layout depends only on
  its own code after inlining, and a relink no longer moves it. That makes a
  link-shift sweep a no-op: to separate layout from an algorithm change, shift
  function entries instead (`-fpatchable-function-entry=K,K`), which models an
  edit at the top of every function.

## Benchmarking

The `ta-bench` skill covers it: `ta_bench`, `ta_bench_direct`, `ta_bench_stream`
and `scripts/stream_ab.py`; what each ratio actually compares (the same source
builds six different binaries); streaming vs batch; and the `--shape=` input
corpus.

When several sessions share a machine, one session's build skews another's
timing run without any error. Run anything that measures time as
`scripts/quiet.py measure <session> <secs> --queue=<max> -- <cmd>`: it waits its
turn, first come first served, for at most `<max>` seconds (900 at most), and
exits 75 if the window stays taken. Without `--queue` it exits 75 at once.
Either way, do other work and retry rather than wait in a loop of your own.
It also refuses while other work keeps the machine busy, whoever started it
(`status` shows the load; `TA_QUIET_MAX_LOAD=<cores>` sets the limit).
Run heavy jobs (`generate`, `build.py` targets, raw cargo/cmake builds) as
`scripts/quiet.py noisy <session> --defer=60 -- <cmd>` so measurers back off:
`--defer` first waits, at most that long (60 at most), while a measurement runs
or is queued, because back-to-back heavy jobs otherwise starve a queued measurer.
A deferred job starts after 60 s even inside a running measurement, so keep each
`measure` window short and split a long campaign into several.
`scripts/regtest.py` both builds and times, so split it: a heavy job with
`--no-perftest --no-direct-bench`, then `measure` with
`--test-only --no-regtest`. `bench_icount.py` counts instructions, which load
cannot move, but it builds first: run it as a heavy job. `scripts/quiet.py status`
names the holder and the queue.

More agent context in TA-Lib/ta-lib

5 other files this repository gives its agents.