Gradients and learning

Autodiff lives in the standard library rather than in a framework. Forward mode uses dual numbers; reverse mode has its own module.

Requires FLOW_HOST=python.

Find the current surface first

The module names are stable but the function names move. Check before writing:

ls lib/stdlib/ | grep -E "autodiff|nn|ml_|tensor"
grep -n "^function" lib/stdlib/autodiff.flow | head -40
grep -n "^function" lib/stdlib/autodiff_reverse.flow | head -40
grep -n "^function" lib/stdlib/ml_opt.flow | head -30

lib/stdlib/nn_xor_loss_*.flow are checked-in generated gradient sources. They show what code generation produces and are worth reading before hand-writing a backward pass.

Shape of the work

1. Write the forward function in ordinary Flow.
2. Get its gradient through the autodiff module, or through generated gradient code where the example does that.
3. Drive an optimiser loop.
4. Assert that the loss actually fell, and that the gradient matches a finite difference.

The check that matters

Never report a training run without a gradient check. A wrong gradient still produces a loss curve that goes down sometimes.

# finite difference against the analytic gradient
let h: f64 = 1e-6
let numeric: f64 = (f(x + h) - f(x - h)) / (2.0 * h)
let analytic: f64 = grad_f(x)
if fabs(numeric - analytic) > 1e-4 {
    println("FAIL: gradient mismatch")
    return 1
}

Then assert the training outcome, for example that XOR loss ends below a threshold, and return a distinct code otherwise.

Worked examples

ls examples/ml/
ls examples/ml/models/
FLOW_HOST=python flow run examples/ml/models/mlp_xor.flow

The ML demos are built to train on CPU in seconds. If something is running for minutes, the model or the step size is wrong, not the machine.

MLIR and GPU paths

flow ml run program.flow      # MLIR-first ML workloads
flow ml bench program.flow
flow gpu program.flow          # emit @gpu kernels as Metal sources

MLIR needs LLVM installed via flow setup. Do not claim an MLIR or GPU result you have not run. Report the backend in the command.

Reporting

State the loss before and after, the number of steps, the learning rate, the gradient check result, and the exact command. A number without a method behind it is not a result.