agentleFS
Sign inSign up
docxology

template

docxology/template/AGENTS.md

This document provides documentation for the Research Project Template system, ensuring understanding of all functionality, configuration options, and operational procedures. Title: A template/ approach to Reproducible Generative Research: Architecture and Ergonomics from Configuration through Publication DOI: 10.5281/zenodo.19139090 · Record: zenodo.org/records/19139090 template/ applies Infrastructure as Code to the research lifecycle: version-controlled manuscripts, tests, provenance, and a declared pipeline DAG. Layer 1 (infrastructure/) is separated from Layer 2 (self-contained projects under projects/). Current measured counts and stage facts live in docs/_generated/COUNTS.md; re-derive…

AGENTS.md20 starsChanged 33 days ago
  • Reads credentials
  • Deletes or force-pushes
  • Installs packages
View on GitHub Discuss
# 🤖 AGENTS.md - System Documentation

## 🎯 System Overview

This document provides documentation for the Research Project Template system, ensuring understanding of all functionality, configuration options, and operational procedures.

### 📄 Publication

**Title**: *A template/ approach to Reproducible Generative Research: Architecture and Ergonomics from Configuration through Publication*
**DOI**: [10.5281/zenodo.19139090](https://doi.org/10.5281/zenodo.19139090) · **Record**: [zenodo.org/records/19139090](https://zenodo.org/records/19139090)

`template/` applies Infrastructure as Code to the research lifecycle: version-controlled manuscripts, tests, provenance, and a declared pipeline DAG. **Layer 1** (`infrastructure/`) is separated from **Layer 2** (self-contained projects under `projects/`). Current measured counts and stage facts live in [`docs/_generated/COUNTS.md`](docs/_generated/COUNTS.md); re-derive them instead of copying literals into prose. Each directory carries `README.md` + `AGENTS.md`; infrastructure packages usually add `SKILL.md` for agent routing. Full paper, metrics, and claims: Zenodo record above and root [`README.md`](README.md).

### Documentation map

| Entry | Role |
| --- | --- |
| [`README.md`](README.md) | Onboarding, documentation hub links, exemplar table |
| [`.cursorrules`](.cursorrules) | Cursor agents: layer rules, CI scope, editing discipline |
| [`CLAUDE.md`](CLAUDE.md) | Command cheat sheet, patterns; keep in sync with this file for pipeline wording |
| [`START_HERE.md`](START_HERE.md) | Agent entry point: install, first pipeline run, orientation ladder |
| [`STATUS.md`](STATUS.md) | Per-subsystem verification ledger (last-verified dates, health) |
| **This file (`AGENTS.md`)** | Full reference: stages, validation, modules, troubleshooting |
| [`docs/documentation-index.md`](docs/documentation-index.md) | Flat index of long-lived docs |
| [`docs/_generated/active_projects.md`](docs/_generated/active_projects.md) | Authoritative public CI/documentation project names — never hard-code rotating private paths in docs |
| [`docs/_generated/COUNTS.md`](docs/_generated/COUNTS.md) | Measured coverage and counts; refresh after changing gates or discovery |
| [`.github/README.md`](.github/README.md) | GitHub: CI overview, templates, Dependabot |
| [`.github/AGENTS.md`](.github/AGENTS.md) | Actions job names, coverage gates, local reproduction commands |
| [`TO-DO.md`](TO-DO.md) | Current backlog, scoped by acceptance line |
| [`CHANGELOG.md`](CHANGELOG.md) | Release history |

## For assistants and automation

**Read order:** [`README.md`](README.md) → [`CLAUDE.md`](CLAUDE.md) → this file for anything not covered there.

**Ground truth:** Public CI/documentation project names come from [`docs/_generated/active_projects.md`](docs/_generated/active_projects.md) (`infrastructure.project.public_scope`). Runtime `discover_projects()` remains broader for local private symlinked workspaces. Measured numbers for documentation claims belong in [`docs/_generated/COUNTS.md`](docs/_generated/COUNTS.md); avoid inventing statistics.

**Definition of done (code):** Ruff clean on the full public lint surface from `uv run python -m infrastructure.project.public_scope lint-paths`, and mypy clean on the import-safe source paths from `... source-paths`; tests exercise real behaviour, introduce no prohibited mock framework, and do not increase semantic dependency-replacement debt; coverage still meets 60% (infra) / 90% (project `src/`) unless CI documents a rotating-project exception ([`.github/AGENTS.md`](.github/AGENTS.md)).

**Hooks:** [`.pre-commit-config.yaml`](.pre-commit-config.yaml) — commit stage runs Ruff and mypy; pre-push adds no-mocks verification, a short pytest smoke module, Bandit (`-c bandit.yaml`, exclusions in YAML), and `infrastructure.skills check` + `check-all-exports`. Install: `pre-commit install` and `pre-commit install --hook-type pre-push` after `uv sync`.

**Architecture:** Business logic only in `infrastructure/` or `projects/{name}/src/`. Scripts orchestrate; violating this breaks the test and documentation contract ([Thin orchestrator pattern](#thin-orchestrator-pattern)).

**Optional CodeGraph:** CodeGraph may be used as a local agent navigation index
for source-code questions and impact analysis. It is not a dependency, pipeline
stage, CI requirement, or manuscript evidence source. `.codegraph/` is ignored
and rejected by the generated-artifact guard if force-added. Use
[`docs/guides/codegraph-local.md`](docs/guides/codegraph-local.md) and
`uv run python scripts/maintenance/codegraph_local.py commands .`; initialize private
associated projects from their canonical private checkout, not through public
`template/projects/<name>` symlinks.

**Optional LEANN:** LEANN may be used as a local semantic-retrieval companion for
agent navigation. It is not a dependency, pipeline stage, CI requirement, MCP
default, or manuscript evidence source. `.leann/` is ignored and rejected by the
generated-artifact guard if force-added. Use
[`docs/guides/leann-local.md`](docs/guides/leann-local.md); build indexes from
tracked public files or from the canonical private checkout, not through public
`template/projects/<name>` symlinks.

**Local agent memory (gitignored):** When present, read
`.cursor/hooks/state/continual-learning-memory.json`
for durable user preferences and workspace facts. Never commit this file or write
Learned sections into root `AGENTS.md`. Schema:
[`.cursor/hooks/state/continual-learning-memory.example.json`](.cursor/hooks/state/continual-learning-memory.example.json).
Load/save helpers: `infrastructure.core.agent_memory`.

**Public output hygiene:** Canonical public exemplars may track deterministic
publication evidence and release contracts under `projects/templates/<name>/output/`:
final PDFs, figures, analysis data, hydrated manuscripts, and stable
validation/publication registries. Runtime checkpoints, `.pipeline/` state,
logs, telemetry, stage snapshots, pipeline reports, and LaTeX/slide build
intermediates are local-only and ignored. Recreate them through the canonical
pipeline when diagnosing or rendering; do not publish them as exemplar
evidence.

## Template authoring & operational notes

Generic, Layer-1 facts for working in this repository.

- **Manuscript variables are injected, not hand-authored.** Per-project metrics, counts, and variables come from `output/data/manuscript_variables.json` at render time. For `template_code_project`, the default pipeline calls `generate_variables(..., require_analysis_outputs=True)` via `projects/{name}/scripts/z_generate_manuscript_variables.py` and fails when `output/data/optimization_results.csv` is absent; pass `--allow-draft` only for intentional early drafts. PDF Publishing Information reads `publication.doi`, optional `publication.repository_url`, and `publication.repository_label` from `projects/{name}/manuscript/config.yaml` via `infrastructure/rendering/_pdf_latex_helpers.py`.
- **Validation & rendering pitfalls.** Content-validation diagnostics use stable dotted IDs from `infrastructure/validation/content/diagnostic_codes.py` (`MarkdownCode`, `BibtexCode`); every new `DiagnosticEvent` must pass `code=…`, and renaming an existing code is a breaking change for downstream `jq`/`rg` filters. Fast manuscript pre-flight: `uv run python -m infrastructure.validation.cli prerender projects/<project>/manuscript --repo-root .`. Multi-pass PDF rendering continues when pass 1 wrote output despite recoverable `Missing $` errors so later passes resolve forward references. Mermaid: unquoted `//` line comments; stadium nodes `[/label/]` close with `/]`; combined-PDF Mermaid via Chrome headless or `mmdc`, else verbatim figure fallback. `FIGURE_WIDTH_*` values must be bare fractions (e.g. `0.9`); the alt-text comment belongs before `\begin{figure}`; prefer inline `$...$` over `\(...\)` in Markdown list items.
- **Entry points & gates.** `run.sh` and `secure_run.sh` source only [`scripts/shell/shell_bootstrap.sh`](scripts/shell/shell_bootstrap.sh); menu and argparse live in `infrastructure.orchestration`. [`scripts/shell/bash_utils.sh`](scripts/shell/bash_utils.sh) serves backup/health scripts and tests, not pipeline entrypoints. Exemplar doc/code drift: `scripts/audit/check_template_drift.py` → `infrastructure.project.drift.run_drift_checks()` on `PUBLIC_PROJECT_NAMES` (`--project`, `--strict`). Layer 1 module size: `scripts/gates/module_line_count_check.py` and `uv run python -m infrastructure.core.health` (`module-line-count`). The health registry also runs the executable methods contract and `scripts/gates/public_capabilities.py` across the canonical public roster. Opt-in gates under `scripts/gates/` report `status: "skipped"` under `skipped_tools` when tools are missing. `bandit.yaml` `exclude_dirs` skips rotating/private trees so CI stays strict on `infrastructure/`, `scripts/`, and public exemplars.

## Confidentiality invariant (this is a PUBLIC repo)

`.gitignore` ignores `projects/*` and negates **only** the public canonical exemplar trees under `projects/templates/` (plus the repo-level `projects/*.md` docs). The public exemplar roster is derived from `infrastructure.project.public_scope.PUBLIC_PROJECT_NAMES` and documented in [`docs/_generated/active_projects.md`](docs/_generated/active_projects.md); do not hand-maintain a second allowlist in prose. Those public canonical exemplars are the **only** project trees ever git-tracked/pushed. Confidential/private work lives in a **separate, external private repository** whose location is configured with `TEMPLATE_PRIVATE_PROJECTS_ROOT` or `.private_projects_root`; the simplified sidecar normally has `working/` and `archive/`, with optional `ongoing/` (long-lived projects with no publication target) and optional legacy `active/` folders still supported by the linker. `run.sh`/`infrastructure.orchestration` sync existing lifecycle folders into matching typed subfolders under `projects/`: `working/*` → `projects/working/*`, `ongoing/*` → `projects/ongoing/*`, `archive/*` → `projects/archive/*`, and optional `active/*` → `projects/active/*` (rendered). Only `projects/templates/*` and optional `projects/active/*` are discovered/rendered by default.

Every path under `projects/` other than `templates/` — especially the local-only `working/`, `archive/`, and optional `active/` mirrors — is **local-only and must never be committed**. This is enforced, not conventional: `scripts/audit/check_tracked_all.py` fails the CI `lint` job and the pre-push `pre-push-quick` hook if any non-template project path is tracked (a `git add -f` cannot slip past it). Negative control: `test_offending_tracked_projects_flags_non_exemplar` and `test_offending_tracked_projects_flags_unknown_templates_toplevel_doc` in [`tests/infra_tests/project/test_git_guards.py`](tests/infra_tests/project/test_git_guards.py) construct known-wrong tracked paths and assert the guard fails. Consult [`docs/_generated/active_projects.md`](docs/_generated/active_projects.md) before hard-coding any project path in docs.

The same invariant covers three sibling top-level resource-pool directories — `fonds/`, `rules/`, `tools/` (each analogous to `projects/`: only their `templates/` subfolder is git-tracked, `working/`/`archive/` are LOCAL-ONLY, and each gets the same `run.sh`/`infrastructure.orchestration` sidecar auto-sync, independently overridable via `TEMPLATE_FONDS_ROOT`/`TEMPLATE_SKIP_FOND_LINK_SYNC`, `TEMPLATE_RULES_ROOT`/`TEMPLATE_SKIP_RULE_LINK_SYNC`, `TEMPLATE_TOOLS_ROOT`/`TEMPLATE_SKIP_TOOL_LINK_SYNC`). `scripts/audit/check_tracked_all.py` runs all four confidentiality checks (`offending_tracked_projects/fonds/rules/tools` in `infrastructure/project/git_guards.py`) in one pass; the narrower `scripts/audit/check_tracked_projects.py` still runs standalone but is no longer wired into CI or pre-commit.

Operational gotchas: running **all** `projects/*/tests/` in **one** pytest process fails when projects each ship `tests/conftest` under the `tests.conftest` package name — run **one project test directory per pytest invocation** (with `--cov-append` to merge coverage), or follow `.github/workflows/ci.yml`. `resolve_project_root` accepts a qualified `<subfolder>/<name>` path (e.g. `templates/template_code_project`, `active/demo`) and resolves it directly under `projects/`. For a bare name it prefers `projects/active/<name>/` (the hot seat) when that tree has project markers, then `projects/working/<name>/`, then a flat standalone `projects/<name>/`, falling back to `projects/active/<name>/` for error messages.

**Agent skills hub:** [`docs/prompts/SKILL.md`](docs/prompts/SKILL.md) (`template-workflows`); regenerate via `uv run python -m infrastructure.skills write` and `uv run python -m infrastructure.skills write-index`. Repository-scoped context-engineering skills live under [`.agents/skills/`](.agents/skills/) with a pinned source lock and cross-runtime sync/check command. Each canonical exemplar under `projects/templates/` also ships its own `.agents/skills/<name>/SKILL.md`; both public lanes are included in `.cursor/skill_manifest.json`, the generated skills index, MCP `list_skills`, and Hermes/agentskills.io runtimes.

## 📋 Table of Contents

**Assistants:** [For assistants and automation](#for-assistants-and-automation)

1. [Core Architecture](#core-architecture)
2. [Directory-Level Documentation](#directory-level-documentation)
3. [Configuration System](#configuration-system)
4. [Rendering Pipeline](#rendering-pipeline)
5. [Validation Systems](#validation-systems)
6. [Testing Framework](#testing-framework)
7. [Output Formats](#output-formats)
8. [Advanced Modules](#advanced-modules)
9. [Troubleshooting](#troubleshooting)
10. [Maintenance](#maintenance)

<a id="core-architecture"></a>

## 🏗️ Core Architecture

### Two-Layer Architecture

#### Layer 1: Infrastructure (Generic - Reusable)

- `infrastructure/` - Generic build/validation tools (reusable across projects)
- `scripts/` - Entry point orchestrators (core pipeline or full pipeline via `./run.sh`)
- `tests/` - Infrastructure and integration tests

#### Layer 2: Projects (Project-Specific - Customizable)

- `projects/{name}/src/` - Research algorithms and analysis (domain-specific per project)
- `projects/{name}/tests/` - Project test suite
- `projects/{name}/scripts/` - Project analysis scripts (thin orchestrators)
- `projects/{name}/manuscript/` - Research manuscript
- `projects/{name}/output/` - Working outputs during pipeline execution
- `output/{name}/...` - Final deliverables after pipeline completion

### Thin Orchestrator Pattern

**CRITICAL**: All business logic resides in `projects/{name}/src/` modules. Scripts are **thin orchestrators** that:

**Root Entry Points (Generic):**

- Coordinate build pipeline stages
- Discover and invoke `projects/{name}/scripts/` for specified project
- Handle I/O, orchestration only
- Work with ANY project structure (single or multi-project)

**Project Scripts (Project-Specific):**

- Import from `projects/{name}/src/` for computation
- Import from `infrastructure/` for utilities
- Orchestrate domain-specific workflows
- Handle I/O and visualization

**Violation of this pattern breaks the architecture**.

### Multi-Project Support

The template now supports **multiple independent projects** within a single repository:

**Project Discovery:**

- Projects are discovered automatically from `projects/` directory
- Each project must have `src/` and `tests/` directories
- Projects are validated for structural completeness

**Project Isolation:**

- Each project has its own source code, tests, manuscript, and scripts
- Working outputs are stored in `projects/{name}/output/`
- Final deliverables are organized in `output/{name}/...`

**Orchestration Options:**

- Run individual projects: `--project {name}`
- Run all projects sequentially: `--all-projects`
- Interactive project selection menu
- Backward compatibility with single-project workflows

**Active projects** (under `projects/`): the set **rotates** as workspaces are promoted, archived, or moved. Authoritative names **at any moment** are only in [`docs/_generated/active_projects.md`](docs/_generated/active_projects.md) (regenerate after layout changes). The **only** path **guaranteed** to remain the **control-positive** exemplar for docs and commands is `projects/templates/template_code_project/` (optimization research exemplar).

Private projects normally live in a separate external private repository
(location configurable via `TEMPLATE_PRIVATE_PROJECTS_ROOT` or `.private_projects_root`) and are symlinked by lifecycle into matching typed subfolders under `projects/`.
The simplified private sidecar uses `working/` and `archive/` by default; optional `ongoing/` (long-lived projects with no publication target) and legacy `active/` folders are still recognized when present. `working/*` mirrors into `projects/working/*`, `ongoing/*` into `projects/ongoing/*`, and `archive/*` into `projects/archive/*` for explicit inspection/rendering; optional `active/*` mirrors into `projects/active/*` for discovery/rendering. Use
`uv run python -m infrastructure.orchestration link-projects --dry-run` to
inspect the planned links, `TEMPLATE_PRIVATE_PROJECTS_ROOT` or
`.private_projects_root` to override the sibling repo, and
`TEMPLATE_SKIP_LINK_SYNC=1` to disable auto-sync for a command.

**Note:** Archived and in-progress trees live under the local-only `projects/archive/` and `projects/working/` mirrors. Rendered names are listed only in [`docs/_generated/active_projects.md`](docs/_generated/active_projects.md).

## 📂 Project Organization: Rendered vs Non-Rendered Subfolders

### Rendered Projects (`projects/templates/` + `projects/active/`)

Projects under `projects/templates/` (tracked exemplars) and `projects/active/` (hot-seat set) are **actively discovered and executed** by infrastructure:

- **Discovered** by `infrastructure.project.discovery.discover_projects()` with qualified names `templates/<name>` and `active/<name>`
- **Listed** in `run.sh` interactive menu
- **Executed** by the canonical pipeline stages under `scripts/pipeline/` (for example, `stage_01_test.py` and `stage_02_analysis.py`)
- **Outputs** generated in `projects/<subfolder>/{name}/output/` and copied to `output/<subfolder>/{name}/`

### Non-Rendered Projects (`working/`, `ongoing/`, `archive/`, optional legacy mirrors)

Projects under `projects/working/`, `projects/ongoing/`, and `projects/archive/` are **preserved for explicit targeted work but not executed by default**:

- **NOT discovered** by default infrastructure discovery functions
- **NOT listed** in the normal `run.sh` menu
- **NOT executed** by all-project pipeline scripts
- **Preserved** for in-progress work, long-lived no-publication-target work (`ongoing/`), historical reference, and explicit qualified commands such as `--project working/<name>` or `--project ongoing/<name>`

### Project Lifecycle

**Retiring in the sidecar:** Move `working/{name}/` → `archive/{name}/`
**Resuming in the sidecar:** Move `archive/{name}/` → `working/{name}/`
**Explicit render:** From the template checkout, run `uv run python scripts/pipeline/stage_03_render.py --project working/{name}` after `link-projects`.

Projects are automatically discovered when a deliberately restored sidecar
`active/{name}` entry is synced into `projects/active/{name}`. Normal sidecar
work can also be rendered explicitly with a qualified name such as
`working/{name}` without moving it into the default render set.

### In-Progress Projects (`projects/working/`)

An intermediate staging area for projects that are under active development but not yet ready to run through the full pipeline. Projects here:

- **NOT discovered** by infrastructure discovery functions
- **NOT listed** in `run.sh` menu
- **NOT executed** by any pipeline scripts
- Useful for drafting new project scaffolding before explicitly rendering with a
  qualified name or deliberately restoring into optional `projects/active/`

**Current in-progress projects:** the roster rotates every checkout, so it is deliberately **not** hard-coded here — run `ls projects/working/` for the live set (not executed by default by `./run.sh`; render explicitly with a qualified project name such as `working/<name>`). Rendered projects are listed only in [`docs/_generated/active_projects.md`](docs/_generated/active_projects.md).

**To render by default:** restore/sync the project through the sidecar's optional
`active/{name}` folder so it appears as `projects/active/{name}`. Keep ordinary
backburner work in sidecar `working/` or `archive/`.

## 📚 Repository Structure

The template separates **generic infrastructure** from **project-specific code**:

```mermaid
flowchart TB
    ROOT[template<br/>Generic template repository]

    ROOT --> INFRA[infrastructure<br/>Layer 1 · generic build · validation tools]
    ROOT --> SCRIPTS[scripts<br/>Pipeline stage orchestrators]
    ROOT --> TESTS[tests<br/>Infrastructure test suite]
    ROOT --> DOCS[docs<br/>Documentation hub · 300+ files]
    ROOT --> PROJECTS[projects<br/>Typed subfolders · templates+active rendered]
    ROOT --> OUT[output<br/>Final deliverables · organized by project]

    INFRA --> I_DOCS[AGENTS.md · README.md · SKILL.md]
    INFRA --> I_CONFIG[config<br/>Repo-wide configuration]
    INFRA --> I_DOCKER[docker<br/>Container specs]
    INFRA --> I_SUB[Layer 1 packages listed in<br/>`infrastructure/AGENTS.md` —<br/>counts from COUNTS.md]

    PROJECTS --> P_README[README.md · multi-project guide]
    PROJECTS --> P_STUB[_test_project<br/>Stub · output/ only · not discovered]
    PROJECTS --> P_CODE[template_code_project<br/>Guaranteed control-positive exemplar]
    PROJECTS --> P_OTHER[&lt;name&gt;<br/>Additional discovered projects]

    P_CODE --> P_C_SRC[src · tests · scripts · manuscript · output<br/>+ pyproject.toml]

    OUT --> O_CODE[template_code_project<br/>Project outputs]
    OUT --> O_DOTS[&lt;other projects&gt;]

    classDef root fill:#0f172a,stroke:#0f172a,color:#fff
    classDef l1 fill:#1e3a8a,stroke:#0f172a,color:#fff
    classDef l2 fill:#0f766e,stroke:#0f172a,color:#fff
    classDef gen fill:#7c2d12,stroke:#0f172a,color:#fff
    class ROOT root
    class INFRA,SCRIPTS,TESTS,DOCS,I_CONFIG,I_DOCKER,I_SUB,I_DOCS l1
    class PROJECTS,P_CODE,P_OTHER,P_STUB,P_README,P_C_SRC l2
    class WIP,ARCH,OUT,O_CODE,O_DOTS gen
```

<a id="directory-level-documentation"></a>

## 📚 Directory-Level Documentation

Each directory contains documentation for easy navigation:

### Generic Infrastructure (Reusable)

| Directory | AGENTS.md | README.md | Purpose |
| --------- | --------- | --------- | ------- |
| [`infrastructure/`](infrastructure/) | [AGENTS.md](infrastructure/AGENTS.md) | [README.md](infrastructure/README.md) | Generic build/validation tools (Layer 1) |
| [`scripts/`](scripts/) | [AGENTS.md](scripts/AGENTS.md) | [README.md](scripts/README.md) | Generic entry point orchestrators |
| [`tests/`](tests/) | [AGENTS.md](tests/AGENTS.md) | [README.md](tests/README.md) | Infrastructure test suite |

### Project-Specific (Customizable)

Public tracked exemplars live under `projects/templates/`. The live roster,
qualified names, and per-exemplar pointers are generated in
[`docs/_generated/active_projects.md`](docs/_generated/active_projects.md)
(`uv run python scripts/docgen/active_projects.py`). The control-positive
walkthrough remains [`projects/templates/template_code_project/`](projects/templates/template_code_project/).
Rotating private workspaces use typed lifecycle folders; see
[`docs/maintenance/private-projects-repo.md`](docs/maintenance/private-projects-repo.md).

**In-progress projects** live under [`projects/working/`](projects/working/) and are not executed by default pipeline discovery. Render one explicitly with a qualified project name such as `working/<name>`, or deliberately restore it through optional sidecar `active/` only when it should enter default discovery. The roster rotates every checkout, so it is deliberately **not** hard-coded here (cf. the rotation rule above — hard-coding rotating project paths is the recurring staleness defect this guidance prevents): run `ls projects/working/` for the live set, and see [`docs/_generated/active_projects.md`](docs/_generated/active_projects.md) for the rendered roster.

**Ongoing projects** live under [`projects/ongoing/`](projects/ongoing/) when present — long-lived work with no publication target, not discovered or executed by default. Render one explicitly with a qualified project name such as `ongoing/<name>`. The roster rotates every checkout and is deliberately **not** hard-coded here: run `ls projects/ongoing/` for the live set.

**Archived projects** live under [`projects/archive/`](projects/archive/) when present, but the roster is checkout-specific and is not discovered or executed by default. Resume by moving them back to sidecar `working/`, render explicitly with `archive/<name>` when appropriate, or deliberately restore through optional sidecar `active/` for default discovery. Use `ls projects/archive/` for local inspection. The authoritative list of rendered projects is in [`docs/_generated/active_projects.md`](docs/_generated/active_projects.md). Regenerate it after layout changes: `uv run python scripts/docgen/active_projects.py`.

### Documentation Directories

| Directory | AGENTS.md | README.md | Purpose |
| --------- | --------- | --------- | ------- |
| [`docs/`](docs/) | [AGENTS.md](docs/AGENTS.md) | [README.md](docs/README.md) | Project documentation hub |

### Documentation Navigation

**For detailed information:**

- Read directory-specific **AGENTS.md** files for details
- Each AGENTS.md covers architecture, usage, and best practices

**For quick reference:**

- Check directory-specific **README.md** files for fast answers
- Each README.md provides quick start and essential commands

**Root documentation:**

- This file (root **AGENTS.md**) - System overview
- [README.md](README.md) - Project quick start and introduction

### Directory Structure

```mermaid
flowchart TB
    ROOT[template<br/>Generic Template]

    ROOT --> INFRA[infrastructure<br/>Layer 1 · generic build/validation tools]
    ROOT --> DOCS[docs<br/>Documentation hub]
    ROOT --> CUR[.cursor<br/>Editor configuration]
    ROOT --> SCR[scripts<br/>Pipeline stage entry points]
    ROOT --> TS[tests<br/>Infrastructure tests]
    ROOT --> PR[projects<br/>Typed subfolders · templates+active rendered]
    ROOT --> OUT[output<br/>Final deliverables · disposable]
    ROOT --> PYPROJ[pyproject.toml<br/>Root configuration]

    INFRA --> I_DOCS[AGENTS.md · README.md · SKILL.md]
    INFRA --> I_CFG[config<br/>.env.template · secure_config.yaml]
    INFRA --> I_DOCKER[docker<br/>Dockerfile · docker-compose.yml]
    INFRA --> I_MODULES[build_verifier.py · figure_manager.py · ...]

    DOCS --> D_FILES[AGENTS.md · README.md ·<br/>CLOUD_DEPLOY.md · PAI.md · RUN_GUIDE.md]

    CUR --> C_FILES[.cursorrules · .cursorignore · README.md]

    SCR --> S_DOCS[AGENTS.md · README.md]
    SCR --> S_STAGES[scripts/pipeline/stage_00_setup.py<br/>stage_01_test.py<br/>stage_02_analysis.py<br/>stage_03_render.py<br/>stage_04_validate.py<br/>stage_05_copy.py]

    TS --> T_FILES[AGENTS.md · README.md · test_*.py]

    PR --> PR_CODE[template_code_project<br/>Optimization exemplar · active]
    PR --> PR_OTHER[&lt;name&gt;<br/>additional discovered projects]
    PR_CODE --> PRC_LAYOUT[src · tests · scripts ·<br/>manuscript · output<br/>+ pyproject.toml]

    classDef root fill:#0f172a,stroke:#0f172a,color:#fff
    classDef l1 fill:#1e3a8a,stroke:#0f172a,color:#fff
    classDef l2 fill:#0f766e,stroke:#0f172a,color:#fff
    classDef gen fill:#7c2d12,stroke:#0f172a,color:#fff
    class ROOT,PYPROJ root
    class INFRA,DOCS,CUR,SCR,TS,I_DOCS,I_CFG,I_DOCKER,I_MODULES,D_FILES,C_FILES,S_DOCS,S_STAGES,T_FILES l1
    class PR,PR_CODE,PR_OTHER,PRC_LAYOUT l2
    class WIP,ARC,OUT gen
```

**Documentation in each directory:**

- **AGENTS.md** - Detailed directory-specific documentation
- **README.md** - Quick reference and navigation

**Note on src/ directory:**

- Root `src/` no longer exists (was empty shells)
- All code is in `infrastructure/` (generic) or `projects/{name}/src/` (project-specific)
- This separation enables reusability across projects

<a id="configuration-system"></a>

## ⚙️ Configuration System

### Configuration File (Recommended)

The system supports configuration through a YAML file, providing a centralized, version-controllable way to manage all paper metadata.

**Location**: `projects/{name}/manuscript/config.yaml` — resolved by
`infrastructure.core.project_paths.resolve_source_manuscript_dir`, which also
accepts a populated `docs/manuscript/` tree (conventional `manuscript/` with
real sources wins when both exist). See the function's docstring for the exact
precedence.
**Template**: `projects/{name}/manuscript/config.yaml.example`

**Example configuration**:

```yaml
paper:
  title: "Novel Optimization Framework"
  subtitle: ""  # Optional
  version: "1.0"

authors:
  - name: "Dr. Jane Smith"
    orcid: "0000-0000-0000-1234"
    email: "jane.smith@university.edu"
    affiliation: "University of Example"
    corresponding: true

publication:
  doi: "10.5281/zenodo.12345678"  # Optional
  journal: ""  # Optional
  volume: ""  # Optional
  pages: ""  # Optional

keywords:
  - "optimization"
  - "machine learning"

metadata:
  license: "Apache-2.0"
  language: "en"

# LLM Review Settings (optional)
llm:
  reviews:
    enabled: true
    types:
      - executive_summary  # Default: single review
      # Uncomment to enable additional reviews:
      # - quality_review
      # - methodology_review
      # - improvement_suggestions
  translations:
    enabled: true  # Set to false to disable translation generation
    languages:
      - zh  # Default: single translation (Chinese Simplified)
      # Uncomment to enable additional languages:
      # - hi  # Hindi
      # - ru  # Russian
```

**Benefits**:

- ✅ Version controllable (can be committed to git)
- ✅ Single file for all metadata
- ✅ Supports multiple authors with affiliations
- ✅ Structured format (YAML)
- ✅ Easy to edit and maintain

### Environment Variables (Alternative Method)

Environment variables are supported as an alternative configuration method and take precedence over config file values:

| Variable | Default | Description |
|----------|---------|-------------|
| `AUTHOR_NAME` | `"Project Author"` | Primary author name |
| `AUTHOR_ORCID` | `"0000-0000-0000-0000"` | Author ORCID identifier |
| `AUTHOR_EMAIL` | `"author@example.com"` | Author contact email |
| `DOI` | `""` | Digital Object Identifier (optional) |
| `PROJECT_TITLE` | `"Project Title"` | Project/research title |
| `LOG_LEVEL` | `1` | Logging verbosity (0=DEBUG, 1=INFO, 2=WARN, 3=ERROR) |

**Priority order**:

1. Environment variables (highest priority - override config file)
2. Config file (`projects/{name}/manuscript/config.yaml`)
3. Default values (lowest priority)

### Configuration Examples

#### Using Configuration File (Recommended)

```bash
# Edit projects/{name}/manuscript/config.yaml with your information
vim projects/{name}/manuscript/config.yaml

# Build with config file values
uv run python scripts/pipeline/stage_03_render.py --project {name}
```

#### Using Environment Variables

```bash
export AUTHOR_NAME="Dr. Jane Smith"
export PROJECT_TITLE="Novel Optimization Framework"
export AUTHOR_EMAIL="jane.smith@university.edu"
export AUTHOR_ORCID="0000-0000-0000-1234"
export DOI="10.5281/zenodo.12345678"  # Optional

uv run python scripts/pipeline/stage_03_render.py
```

#### Verbose Logging

```bash
export LOG_LEVEL=0  # Show all debug messages
uv run python scripts/pipeline/stage_03_render.py
```

### Runtime Configuration

Configuration is read at runtime by `scripts/pipeline/stage_03_render.py` and applied to:

- PDF metadata (title, author, date)
- LaTeX document properties
- Generated file headers
- Cross-reference systems
- Title page generation

<a id="rendering-pipeline"></a>

## 🚀 Rendering Pipeline

### Pipeline Execution

The template provides **three entry points** for pipeline execution:

#### Main Entry Point (Recommended)

```bash
# Routes to manuscript operations
./run.sh
```

#### Manuscript Operations

```bash
# Interactive menu with manuscript operations
./run.sh

# Non-interactive: default full pipeline (core+LLM). Opt-in tags stay out unless invoked directly. --core-only drops LLM and opt-in stages. See STAGE_SUMMARY.
./run.sh --pipeline
```

LLM review stages use the local Ollama workflow documented in
`infrastructure/llm/README.md`. Canonical smoke commands:

```bash
ollama serve
ollama pull gemma3:4b
uv run pytest tests/infra_tests/llm/ -m requires_ollama -v
```

### Secure Pipeline (`secure_run.sh`)

**Two steps:** (1) **Pipeline run** — the **same DAG** as the normal pipeline through Python
[`PipelineRunner`](infrastructure/orchestration/pipeline_runner.py) /
[`PipelineExecutor`](infrastructure/core/pipeline/executor.py) — **not** by shelling out to `./run.sh`; (2) **Steganography pass** — run
[`SteganographyProcessor`](infrastructure/steganography/) on the resulting PDFs.

**Pipeline run** is skipped when `--steganography-only`. When the pipeline run executes, **`--project <name>` is required** (single project per invocation). For steganography-only with no `--project`, PDFs for **all** discovered projects are processed.

For argv shaping into the `secure` subcommand from the same thin shell as `./run.sh`, use **`./run.sh --secure-run`** (see [`run.sh`](run.sh)). **`./secure_run.sh`** always execs `python -m infrastructure.orchestration secure` and does not replicate the full interactive main menu by itself.

**Steganography pass:** post-processes PDFs (companion `*_steganography.pdf`, `.hashes.json` manifest). Original PDFs stay untouched.

```bash
# Interactive path that forwards to the secure subcommand (same orchestration CLI as ./run.sh)
./run.sh --secure-run

# Full pipeline + steganography for one project
./secure_run.sh --project template_code_project

# Core DAG only (no LLM stages) + steganography
./secure_run.sh --project template_code_project --core-only

# Re-process existing PDFs only (omit --project to cover every discovered project)
./secure_run.sh --steganography-only --project template_code_project

# Multi-project: run pipelines separately, then harden all PDFs without re-running pipelines
./secure_run.sh --steganography-only
```

**Output files:**

```mermaid
flowchart LR
    PDF[projects/&lt;name&gt;/output/pdf]
    PDF --> A[&lt;name&gt;_combined.pdf<br/>Standard output · untouched]
    PDF --> B[&lt;name&gt;_combined_steganography.pdf<br/>Steganographically hardened copy]
    PDF --> C[&lt;name&gt;_combined.hashes.json<br/>SHA-256/SHA-512 integrity manifest]

    classDef d fill:#0f172a,stroke:#0f172a,color:#fff
    classDef f fill:#0f766e,stroke:#0f172a,color:#fff
    class PDF d
    class A,B,C f
```

**Steganographic techniques:** diagonal watermark overlays, QR + barcode strips, PDF
metadata/XMP injection, SHA-256/SHA-512 hash manifests, invisible text layers, optional
AES-256 password encryption.

**Configuration** (`infrastructure/config/secure_config.yaml`):

Controls all steganography settings. Any `steganography:` block in a project's
`manuscript/config.yaml` overrides these repo-level defaults. Key fields:

```yaml
steganography:
  overlays_enabled: true       # Diagonal watermark
  barcodes_enabled: true       # QR + Code128 strip
  metadata_enabled: true       # PDF metadata + XMP
  hashing_enabled: true        # SHA-256/512 manifest
  encryption_enabled: false    # AES-256 password (set pdf_password to enable)
  overlay_mode: "text"         # "text" | "qr" | "none"
  overlay_text: "CONFIDENTIAL"
  overlay_opacity: 0.08        # 0.02 subtle → 0.30 strong
  output_suffix: "_steganography"
```

**See also:** [`scripts/AGENTS.md`](scripts/AGENTS.md) · [`infrastructure/steganography/`](infrastructure/steganography/)

#### Entry Point Comparison

- **`./run.sh`**: Main entry point — interactive menu or pipeline run. Progress banners are generic `[i/N]` lines emitted from the resolved DAG (see [`PipelineRunner._banner`](infrastructure/orchestration/pipeline_runner.py)): the default full run is **10** stages and `--core-only` is **8**, per the generated `STAGE_SUMMARY` below (kept in sync with [`pipeline.yaml`](infrastructure/core/pipeline/pipeline.yaml)).
- **`./run.sh --pipeline`**: Non-interactive full DAG; optional LLM stages may skip if Ollama is unavailable.
- **`./run.sh --secure-run`**: Forwards to the `secure` orchestration subcommand (same Python CLI as bare `./run.sh`; use when you want argv shaping from the main shell).
- **`./secure_run.sh`**: Ensures steganography extras (`uv sync --group steganography`), then `python -m infrastructure.orchestration secure`. **`--project`** is required when running the pipeline phase (omit only for `--steganography-only` across all projects). See [Secure Pipeline](#secure-pipeline-secure_runsh) above.
- **`uv run python scripts/runner/execute_pipeline.py --project {name} --core-only`**: Core DAG only — LLM-tagged and opt-in stages excluded (counts in the generated `STAGE_SUMMARY` below).

### Pipeline Stages

**Full Pipeline Stages** — counts come from the generated `STAGE_SUMMARY` at the end of this file. `run.sh` shows `[0/N]` for clean and `[1/N]`–`[N/N]` for the numbered default-run stages (labels from [`STAGE_NAMES`](infrastructure/orchestration/menu.py)). `--core-only` drops LLM-tagged and opt-in stages.

- **[1/10] Clean Output Directories** - Clean working and final output directories (first default-run stage)
1. **Environment Setup** - Verify system requirements and dependencies
2. **Infrastructure Tests** - Run the focused `pipeline-smoke` infrastructure contract (may be skipped; full coverage gate is explicit)
3. **Project Tests** - Run project test suite (90% coverage minimum)
4. **Project Analysis** - Execute `projects/{name}/scripts/` analysis workflows
5. **PDF Rendering** - Historical stage label; render every enabled manuscript format from current inputs
6. **Output Validation** - Validate every enabled canonical format and blocking release evidence
7. **LLM Scientific Review** - AI-powered manuscript analysis (optional, requires Ollama)
8. **LLM Translations** - Multi-language technical abstract generation (optional, requires Ollama)
9. **Copy Outputs** - Copy, filter disabled renderer-owned formats, and validate final deliverables in root `output/`

**Opt-in long-horizon stages** (NOT in default core or `--core-only` runs — there is no `--tags` CLI flag; invoke the stage or runner script directly):

- **Ebook Generation** (`scripts/pipeline/stage_11_ebook.py`, tag `ebook`) — Generate EPUB, MOBI, and DOCX ebooks from the combined markdown manuscript. Gracefully skips (exit 2) when the combined markdown is absent. Invoke: `uv run python scripts/pipeline/stage_11_ebook.py --project <name>`.
- **docxplus Export** (`scripts/pipeline/stage_13_docxplus.py`, tag `docxplus`) — Export the project as a conforming `.docx`/`.docxplus` that carries its own source tree. Soft-fail: skips when the optional `docxplus` extra is not installed (`uv sync --extra docxplus`). Invoke: `uv run python scripts/pipeline/stage_13_docxplus.py --project <name>`.
- **Metadata Package** (`scripts/pipeline/stage_12_metadata.py`, tag `metadata`) — Generate ONIX 3.0 XML, metadata.json, and OPF skeleton from manuscript/config.yaml. Gracefully skips (exit 2) when config.yaml is absent. Invoke: `uv run python scripts/pipeline/stage_12_metadata.py --project <name>`.
- **Executable Bundle** (`scripts/runner/bundle_executable.py`, tag `bundle`) — Produce a container + lockfile + agent-runnable `manifest.json` for the project. Invoke the runner directly; it is not under `scripts/pipeline/`.
- **Archival Publication** (`scripts/runner/archive_publication.py`, tag `archival`) — Mirror the executable bundle to archival targets. Defaults to dry-run; pass `--commit` only with owner authorization. Invoke the runner directly; it is not under `scripts/pipeline/`.

**Infrastructure Tests Behavior:**

- **Single project pipeline mode**: Stage 2 runs `scripts/pipeline/stage_01_test.py --infra-only --infra-scope pipeline-smoke`, a focused real suite for DAG execution, advisory controls, evidence/profile/benchmark extension points, doc invariants, and tracked-artifact guards. This keeps project rebuilds fast without hiding the full repo gate.
- **Full infrastructure gate**: Run `uv run python scripts/pipeline/stage_01_test.py --infra-only --infra-scope full` (or the direct pytest command in the verification guide) for the coverage-bearing repository suite.
- **Multi-project mode** (`--all-projects`): Infrastructure tests run **once** for all projects at the start, then are **skipped** for individual project executions to avoid redundant testing. This is shown in logs as "Running infrastructure tests once for all projects..." followed by "Skipping stage: Infrastructure Tests" for each project.

**Multi-Project Executive Reporting** (`--all-projects` mode only):

- **Executive Reporting** - Cross-project metrics, summaries, and visual dashboards (generated after all projects, not as a numbered stage)

**Stage numbering:** see the generated `STAGE_SUMMARY` and `STAGE_TABLE` at the end of this file. YAML stage indices are distinct from script filename prefixes. Executable Bundle and Archival Publication use opt-in runner entry points under `scripts/runner/`.

### Manual Execution Options

**Individual Stage Execution:**

```bash
# Environment setup
uv run python scripts/pipeline/stage_00_setup.py --project {name}

# Test execution (combined infra + project)
uv run python scripts/pipeline/stage_01_test.py --project {name}

# Project analysis scripts
uv run python scripts/pipeline/stage_02_analysis.py --project {name}

# PDF rendering
uv run python scripts/pipeline/stage_03_render.py --project {name}

# Output validation
uv run python scripts/pipeline/stage_04_validate.py --project {name}

# Copy outputs
uv run python scripts/pipeline/stage_05_copy.py --project {name}

# LLM manuscript review (optional, requires Ollama)
uv run python scripts/pipeline/stage_06_llm_review.py --project {name}

# Generate executive report (multi-project only)
uv run python scripts/pipeline/stage_07_executive_report.py --project {name}
```

**Validation Tools:**

```bash
# Validate markdown files
uv run python -m infrastructure.validation.cli markdown projects/{name}/manuscript/

# Validate PDF outputs
uv run python -m infrastructure.validation.cli pdf output/{name}/pdf/{name}_combined.pdf
```

## Validation Systems

### PDF Validation

```bash
# Validate generated PDF for issues (per-project)
uv run python -m infrastructure.validation.cli pdf output/{name}/pdf/{name}_combined.pdf

# With verbose output
uv run python -m infrastructure.validation.cli pdf output/{name}/pdf/{name}_combined.pdf --verbose

# Specific PDF file
uv run python -m infrastructure.validation.cli pdf output/{name}/pdf/{name}_combined.pdf
```

**Validation Checks**:

- Unresolved references (`??`)
- Missing citations (`[?]`)
- LaTeX warnings and errors
- Document structure integrity
- Word count and content preview

### Markdown Validation

```bash
# Validate all markdown files
uv run python -m infrastructure.validation.cli markdown projects/{name}/manuscript/

# Strict mode (fail on any issues)
uv run python -m infrastructure.validation.cli markdown projects/{name}/manuscript/ --strict
```

**Validation Checks**:

- Image reference resolution
- Cross-reference integrity
- Equation label validation
- Link formatting
- Mathematical notation

### Test Coverage

See `docs/_generated/COUNTS.md` for current status from live test runs.

```bash
# Run via orchestrator
uv run python scripts/pipeline/stage_01_test.py --project {name}

# Manual with reports
uv run pytest tests/infra_tests/ --cov=infrastructure --cov-report=html
uv run pytest projects/{name}/tests/ --cov=projects/{name}/src --cov-report=html
```

**Requirements**:

- projects/{name}/src/ : 90% minimum (per-project standalone gate; exemplars meet it)
- combined-union public-project gate (`scripts/pipeline/stage_01_test.py --project-only --all-projects --public-projects`, `DEFAULT_FAIL_UNDER`) : 75% — deliberately lower than the per-project floor because per-project suites only cover their own `src/` while the union denominator spans the public exemplar source set. Local `--all-projects` without `--public-projects` still runs every discovered project, including rotating private symlinks. Per-project floors remain authoritative.
- infrastructure/ : 60% minimum

Tests should use real data and computation. The lexical no-mocks gate and the
semantic inventory are both enforced: dependency replacements must remain at
zero, while environment/path isolation is classified separately and permitted.

## Testing Framework

### Mock-framework prohibition and semantic-stand-in inventory

**CRITICAL REQUIREMENT**: Do not introduce `MagicMock`, `mocker.patch`,
`unittest.mock`, or another mocking framework. Prefer real data, local services,
subprocesses, and dependency injection. The lexical CI gate proves only that
prohibited framework imports/calls are absent; `--inventory` separately records
environment isolation and existing `monkeypatch.setattr`/`setitem` dependency
replacement. Negative control: `tests/infra_tests/validation/test_no_mock_enforcer.py` feeds the gate a known-wrong fixture containing a real `MagicMock()` call and asserts it is flagged, and the inventory ratchet fails when dependency-replacement debt grows (`test_inventory_ratchet_fails_only_when_debt_grows`).

```bash
uv run python scripts/audit/verify_no_mocks.py
uv run python scripts/audit/verify_no_mocks.py --inventory --max-dependency-replacements 0
```

This policy ensures:

- Tests validate actual behavior, not mocked behavior
- Integration points are truly tested
- Code is tested in realistic conditions
- No false confidence from mocked tests

### No-Mocks Implementation Patterns

**HTTP API Testing**: Use `pytest-httpserver` for local test servers

```python
# BEFORE (mocked)
with patch('requests.post') as mock_post:
    mock_post.return_value = MagicMock(status_code=200, json=lambda: {"result": "ok"})

# AFTER (HTTP)
def test_api_call(ollama_test_server):
    # ollama_test_server fixture provides HTTP server
    config = OllamaClientConfig(base_url=ollama_test_server.url_for("/"))
    client = LLMClient(config)
    response = client.query("test")  # HTTP request
    assert "response" in response.lower()
```

**CLI Testing**: Execute subprocess commands instead of mocking sys.argv

```python
# BEFORE (mocked)
with patch('sys.argv', ['cli.py', 'validate', 'file.pdf']):
    cli.main()

# AFTER (subprocess)
result = subprocess.run(
    ['python', '-m', 'infrastructure.validation.cli', 'validate', 'file.pdf'],
    capture_output=True, text=True
)
assert result.returncode == 0
```

**PDF Generation**: Create PDFs with reportlab instead of mocking PDF libraries

```python
# BEFORE (mocked)
with patch.dict('sys.modules', {'pdfplumber': mock_pdfplumber}):
    result = extract_text(pdf_file)

# AFTER (PDF)
from reportlab.pdfgen import canvas
c = canvas.Canvas(str(pdf_file))
c.drawString(100, 750, "Test content")
c.save()

result = extract_text(pdf_file)  # PDF processing
assert "Test content" in result
```

**File System Operations**: Use temp files and directories

```python
# BEFORE (mocked)
with patch('builtins.open') as mock_open:
    mock_open.return_value.__enter__.return_value.read.return_value = "content"

# AFTER (files)
def test_file_operation(tmp_path):
    test_file = tmp_path / "test.txt"
    test_file.write_text("content")
    result = read_file(test_file)  # File operation
    assert result == "content"
```

**External Tool Testing**: Keep default tests deterministic; mark opt-in tool/credential tests or fail with setup guidance

```python
# BEFORE (mocked subprocess)
with patch('subprocess.run') as mock_run:
    mock_run.return_value = MagicMock(returncode=0)

# AFTER (tool execution in an opt-in marker group)
@pytest.mark.requires_latex
def test_pandoc_conversion(tmp_path):
    pandoc = shutil.which('pandoc')
    assert pandoc, "Install pandoc before running requires_latex tests"
    md_file = tmp_path / "test.md"
    md_file.write_text("# Test")
    pdf_file = tmp_path / "test.pdf"

    result = subprocess.run([pandoc, str(md_file), '-o', str(pdf_file)])
    assert result.returncode == 0
    assert pdf_file.exists()
```

### Test Structure

Tests follow the **thin orchestrator pattern** principles:

- Import methods from `projects/{name}/src/` or `infrastructure/` modules
- Use data and computation
- Validate actual behavior; avoid adding semantic dependency replacements
- Ensure reproducible, deterministic results

### Test Categories

1. **Unit Tests** (`test_*.py`) - Individual function validation
2. **Integration Tests** - Script and pipeline integration
3. **Validation Tests** - PDF and markdown quality checks

### Running Tests

```bash
# All tests via orchestrator (recommended)
uv run python scripts/pipeline/stage_01_test.py

# Specific test file
uv run pytest projects/{name}/tests/test_example.py -v

# Infrastructure tests with coverage
uv run pytest tests/infra_tests/ --cov=infrastructure --cov-report=html

# Project tests with coverage
uv run pytest projects/{name}/tests/ --cov=projects/{name}/src --cov-report=html
```

<a id="output-formats"></a>

## 📤 Output Formats

### Generated Files Structure

```mermaid
flowchart TB
    OUT[output]
    OUT --> P[project<br/>Project-specific outputs]
    P --> PDF[pdf<br/>PDF documents]
    P --> TEX[tex<br/>LaTeX source files]
    P --> FIG[figures<br/>Generated figures]
    P --> DATA[data<br/>Generated datasets]
    P --> HTML[project_combined.html<br/>HTML version for IDE]

    PDF --> PDF_SECT[01_abstract.pdf · 02_introduction.pdf · ...]
    PDF --> PDF_COMB["{name}_combined.pdf<br/>Full manuscript"]

    TEX --> TEX_FILES[01_abstract.tex · project_combined.tex · ...]

    FIG --> FIG_FILES[example_figure.png · convergence_plot.png · ...]

    DATA --> DATA_FILES[example_data.csv · convergence_data.npz · ...]

    classDef d fill:#0f172a,stroke:#0f172a,color:#fff
    classDef f fill:#0f766e,stroke:#0f172a,color:#fff
    class OUT,P,PDF,TEX,FIG,DATA d
    class PDF_SECT,PDF_COMB,TEX_FILES,FIG_FILES,DATA_FILES,HTML f
```

### PDF Versions

1. **Standard PDF** (`{project}_combined.pdf`, e.g. `template_code_project_combined.pdf`)
   - Professional printing format
   - Optimized for LaTeX rendering
   - Cross-references and citations

2. **IDE-Friendly PDF** (`project_combined_ide_friendly.pdf`)
   - for text editor viewing
   - Better font rendering in IDEs
   - Simplified layout for screen reading

3. **HTML Version** (`project_combined.html`)
   - Web browser compatible
   - IDE integration
   - Interactive features (when available)

<a id="advanced-modules"></a>

## 🧪 **Advanced Modules**

The template includes reusable infrastructure modules for scientific development; see
[`docs/modules/modules-guide.md`](docs/modules/modules-guide.md) and
[`docs/_generated/COUNTS.md`](docs/_generated/COUNTS.md) for the
live module list and counts. Selected module examples:

### 🔒 Core Utilities (`infrastructure/core/`)

Enterprise-grade security and system monitoring.

**Key Features:**

- **Input Sanitization**: LLM prompt validation and threat detection
- **Security Monitoring**: Security event tracking and alerting
- **Rate Limiting**: Configurable request rate limiting with monitoring
- **Health Checks**: System health monitoring with component-level status
- **Security Headers**: HTTP security header implementation

**Usage:**

```python
from infrastructure.llm.core.sanitization import sanitize_llm_input
from infrastructure.core.security import get_security_validator
from infrastructure.core import SystemHealthChecker

# Validate LLM input with security checks
sanitized = sanitize_llm_input(user_prompt)

# Perform system health check
checker = SystemHealthChecker()
if checker.is_healthy():
    status = checker.get_health_status()
```

### 🔍 Validation System (`infrastructure/validation/`)

Comprehensive file integrity, manuscript structures, cross-references, and output validation.

**Key Features:**

- **File Integrity**: Hash-based verification of output files
- **Cross-Reference Validation**: LaTeX reference integrity checking
- **Data Consistency**: Format and structure validation
- **Academic Standards**: Compliance with writing standards
- **Build Artifact Verification**: Output validation

**Usage:**

```python
from infrastructure.validation.integrity import verify_output_integrity, generate_integrity_report

report = verify_output_integrity(output_dir)
print(generate_integrity_report(report))
```

### 🔬 Scientific Development (`infrastructure/scientific/`)

Scientific computing best practices and tools.

**Modular Structure:**

- `stability.py` - Numerical stability checking
- `benchmarking.py` - Performance benchmarking
- `confirmation.py` - Improvement-confirmation statistics

**Key Features:**

- **Numerical Stability**: Algorithm stability testing
- **Performance Benchmarking**: Execution time and memory analysis
- **Confirmation**: `confirm_improvement` verifies that measured improvements are real

**Usage:**

```python
from infrastructure.scientific import check_numerical_stability, benchmark_function

stability = check_numerical_stability(your_function, test_inputs)
benchmark = benchmark_function(your_function, test_inputs)
```

### 🤖 LLM Integration (`infrastructure/llm/`)

Local LLM assistance for research workflows.

**Key Features:**

- **Ollama Integration**: Local model support (privacy-first)
- **Template System**: Pre-built prompts for common research tasks
- **Context Management**: Multi-turn conversation handling
- **Streaming Support**: Response generation
- **Model Fallback**: Automatic fallback to alternative models
- **Token Counting**: Track usage and costs

**Research Templates:**

- Abstract summarization
- Code documentation
- Data interpretation
- Section drafting assistance
- Citation formatting
- Technical abstract translation (Chinese, Hindi, Russian)

**Usage:**

```python
from infrastructure.llm import LLMClient

client = LLMClient()
summary = client.apply_template("summarize_abstract", text=abstract)
response = client.query("What are the key findings?")
```

### 🎨 Rendering System (`infrastructure/rendering/`)

Multi-format output generation from single source.

**Key Features:**

- **PDF Rendering**: Professional LaTeX-based PDFs
- **Presentation Slides**: Beamer (PDF) and reveal.js (HTML) slides
- **Web Output**: Interactive HTML with MathJax
- **Scientific Posters**: Large-format poster generation
- **Format-Agnostic**: Single source, multiple outputs
- **Quality Validation**: Automated output checking

**Usage:**

```python
from infrastructure.rendering import RenderManager

manager = RenderManager()
pdf = manager.render_pdf("manuscript.tex")
slides = manager.render_slides("presentation.md", format="revealjs")
html = manager.render_web("manuscript.md")
all_outputs = manager.render_all("manuscript.md")
```

### 🚀 Publishing Module (`infrastructure/publishing/`)

Automated publishing to academic platforms.

**Module Structure:**

- `_metadata_extraction.py` - Publication metadata extraction (`extract_publication_metadata`) and DOI validation (`validate_doi`)
- `api.py` - Platform API clients (Zenodo, arXiv, GitHub)
- `citations.py` - Citation helpers (BibTeX CLI target plus APA/MLA library helpers)
- `metadata.py` - Publication metadata management
- `platforms.py` - Platform-specific integration logic
- `zenodo/`, `arxiv/`, `github/`, `huggingface/`, `osf/`, `pypi/`, `archival/`, `static_site/` - Per-platform subpackages

**Key Features:**

- **Zenodo Integration**: Upload with DOI minting
- **arXiv Preparation**: Submission package creation
- **GitHub Releases**: Automated release management
- **Metrics Tracking**: Download and citation tracking
- **Distribution Packages**: Publication bundles

**Usage:**

```python
from infrastructure.publishing import (
    extract_publication_metadata,
    publish_to_zenodo,
    create_github_release,
    prepare_arxiv_submission
)

# Extract metadata
metadata = extract_publication_metadata([Path("manuscript.md")])

# Publish to Zenodo
result = publish_to_zenodo(metadata, files, token)
print(result.doi)

# Create GitHub release
release = create_github_release(metadata, files, token)

# Prepare arXiv submission
package = prepare_arxiv_submission(metadata, files)
```

### **Module Integration**

All advanced modules follow the **thin orchestrator pattern**:

- **Business logic** in `infrastructure/` modules with test coverage
- **Orchestration** in separate utility scripts
- **Integration** with existing build pipeline
- **Testing** ensuring reliability
- **Documentation** for each module's functionality

**Additional opt-in modules:**

- **Scientific Connector Registry** (`infrastructure/search/connectors/`) — uniform `Connector` protocol over 8 science databases; `python -m infrastructure.search.connectors list-dbs` / `search` — see [`infrastructure/search/connectors/AGENTS.md`](infrastructure/search/connectors/AGENTS.md)
- **Provenance DAG** (`infrastructure/provenance/`) — content-addressed artifact tracking and lineage — see [`infrastructure/provenance/AGENTS.md`](infrastructure/provenance/AGENTS.md)
- **Research prompts** (`infrastructure/research/`) — reusable LLM prompt templates for research workflows — see [`infrastructure/research/AGENTS.md`](infrastructure/research/AGENTS.md)

**Testing Coverage:** measured per module in [`docs/development/coverage-gaps.md`](docs/development/coverage-gaps.md); suite locations under `tests/infra_tests/`.

- ✅ **Security** — `tests/infra_tests/core/`
- ✅ **Health Check** — `tests/infra_tests/core/`
- ✅ **Input Sanitization** — `tests/infra_tests/llm/`
- ✅ **Integrity** — `tests/infra_tests/validation/`
- ✅ **Publishing** — `tests/infra_tests/publishing/`
- ✅ **Scientific Dev** — `tests/infra_tests/scientific/`
- ✅ **Build Verifier** — `tests/infra_tests/validation/`
- ✅ **LLM Integration** — `tests/infra_tests/llm/` (measured coverage → [`docs/development/coverage-gaps.md`](docs/development/coverage-gaps.md))
- ✅ **Rendering System** — `tests/infra_tests/rendering/` (measured coverage → [`docs/development/coverage-gaps.md`](docs/development/coverage-gaps.md))
- ✅ **Reporting** — `tests/infra_tests/reporting/` (measured coverage → [`docs/development/coverage-gaps.md`](docs/development/coverage-gaps.md))

### Accessing Outputs

```bash
# Open combined PDF
open output/{name}/pdf/{name}_combined.pdf

# Open HTML version in browser
open output/{name}/{name}_combined.html

# List all generated files
ls -la output/{name}/

# Check PDF validation
uv run python -m infrastructure.validation.cli pdf output/{name}/pdf/{name}_combined.pdf
```

<a id="troubleshooting"></a>

## 🔧 Troubleshooting

### Quick Reference

- **General Troubleshooting**: [`docs/operational/troubleshooting/`](docs/operational/troubleshooting/)
- **Common Errors**: [`docs/operational/troubleshooting/common-errors.md`](docs/operational/troubleshooting/common-errors.md)
- **LLM Review Issues**: [`docs/operational/troubleshooting/llm-review.md`](docs/operational/troubleshooting/llm-review.md)
- **Checkpoint/Resume**: [`docs/operational/config/checkpoint-resume.md`](docs/operational/config/checkpoint-resume.md)
- **Performance Issues**: [`docs/operational/config/performance-optimization.md`](docs/operational/config/performance-optimization.md)
- **Headless / Cloud Deploy**: [`docs/CLOUD_DEPLOY.md`](docs/CLOUD_DEPLOY.md) ☁️

### Common Issues

#### Tests Failing

```bash
# Ensure coverage requirements met for both suites
uv run python scripts/pipeline/stage_01_test.py

# Or run individually with coverage reports
uv run pytest tests/infra_tests/ --cov=infrastructure --cov-fail-under=60
uv run pytest projects/{name}/tests/ --cov=projects/{name}/src --cov-fail-under=90
```

#### Project Fails In `--all-projects` But Passes Standalone

A multi-project run (`./run.sh --all-projects`) executes projects sequentially over a long
wall time; a project can fail there yet pass when re-run on its own. Triage:

```bash
# Re-run only the failed project (the multi-project summary prints this command)
./run.sh --project {name} --pipeline --core-only --skip-infra

# Or resume the whole run from the last good checkpoint
./run.sh --all-projects --pipeline --resume
```

- **`FileNotFoundError: .../projects/{name}/.venv/bin/python` at Project Tests** — the
  project's `.venv/` directory survived with a *dangling* `bin/python` symlink (its base
  interpreter was moved/removed, e.g. a relocated miniforge/uv install). The test runner now
  detects this via `resolve_test_python()` (`infrastructure/core/runtime/_python_env.py`):
  it only uses the project interpreter when that file actually exists and otherwise falls
  back to the workspace interpreter, so a stale `.venv/` no longer crashes the stage. To
  remove the stale env entirely: `rm -rf projects/{name}/.venv` (it is regenerated on demand;
  the analysis stage uses `uv run` which self-heals it).
- **Transient Project Tests / PDF Rendering failures under load** — flaky failures during a
  38-min multi-project sweep that do not reproduce standalone. Confirm health directly:
  a project's own gate (`projects/{name}/output/reports/test_results.json` →
  `total_failed: 0`) and render artifacts (`projects/{name}/output/pdf/*_combined.pdf`
  or `output/{name}/pdf/*_combined.pdf` present, with no `^! ` lines in adjacent
  `.log` files). If clean, re-run that single project.

#### `validate_docs_and_figures.py` exit 1 at Project Analysis (rotating project)

Some rotating/private projects ship a read-only self-validator. Its run-report,
session, and output-link checks often assert a **completed canonical run**
(`run_all.py` / later pipeline stages). Run as a plain analysis script it should
treat not-yet-generated artifacts as non-blocking warnings; pass `--strict` to
require the full canonical run as a hard gate. Without `--strict`, missing
artifacts stay warnings; with `--strict`, a run report that is absent or marks
the canonical run incomplete fails the validator instead of passing on warnings
(negative control: absent artifacts are expected to fail only the `--strict`
path). Keep project-specific artifact paths in the project docs, not in this
root manual.

#### Scripts Failing

```bash
# Run scripts individually to debug
uv run python scripts/pipeline/stage_02_analysis.py --project {name}

# Check import errors
uv run python -c "import importlib; importlib.import_module('projects.{name}.src')"
```

#### PDF Generation Issues

```bash
# Check LaTeX installation
which xelatex

# Validate LaTeX packages (pre-flight check)
uv run python -m infrastructure.rendering.latex_package_validator

# Validate markdown first
uv run python -m infrastructure.validation.cli markdown projects/{name}/manuscript/

# Check compilation logs
ls projects/{name}/output/pdf/*_compile.log output/{name}/pdf/*_compile.log
```

**Missing LaTeX Package Errors**:

If you see "File *.sty not found" during PDF rendering:

1. **Identify the missing package** from the error message
2. **Install via tlmgr** (BasicTeX package manager):

   ```bash
   sudo tlmgr update --self
   sudo tlmgr install multirow cleveref doi newunicodechar
   ```

3. **Verify installation**:

   ```bash
   /usr/local/texlive/2025basic/bin/universal-darwin/kpsewhich multirow.sty
   ```

4. **Run pre-flight validation**:

   ```bash
   uv run python -m infrastructure.rendering.latex_package_validator
   ```

**Common missing packages in BasicTeX**:

- `multirow`, `cleveref`, `doi`, `newunicodechar` - Require installation
- `bm`, `subcaption` - Already included (part of `tools` and `caption`)

**Alternative**: Install full MacTeX (~4 GB) instead of BasicTeX (~100 MB):

```bash
brew install --cask mactex
```

#### Missing Dependencies

```bash
# One-shot (macOS / Debian-Ubuntu): installs pandoc, a XeLaTeX TeX
# distribution, and missing LaTeX packages; idempotent, --check verifies only
bash scripts/shell/setup-system-deps.sh
# Install system dependencies
# Ubuntu/Debian:
sudo apt-get install -y pandoc texlive-xetex texlive-fonts-recommended fonts-dejavu

# macOS (BasicTeX - minimal):
brew install pandoc
brew install --cask basictex
sudo tlmgr update --self
sudo tlmgr install multirow cleveref doi newunicodechar

# macOS (MacTeX):
brew install pandoc
brew install --cask mactex
```

### Debug Mode

```bash
# Enable verbose logging
export LOG_LEVEL=0
uv run python scripts/pipeline/stage_03_render.py --project {name}

# Run with debug output
uv run python -m infrastructure.validation.cli pdf output/{name}/pdf/{name}_combined.pdf --verbose
```

### Log Files

Key log files for debugging:

- `projects/{name}/output/pdf/*_compile.log` - working LaTeX compilation logs
- `projects/{name}/output/pdf/_combined_manuscript.md` - combined markdown source when retained by the renderer
- `output/{name}/pdf/` - final copied PDF deliverables
- Test output from pytest runs

<a id="maintenance"></a>

## 🛠️ Maintenance

### System Updates

1. **Update Dependencies**

   ```bash
   # Update Python packages
   uv sync

   # Update system packages
   sudo apt-get update && sudo apt-get upgrade
   ```

2. **Version Control**

   ```bash
   # Check current status
   git status

   # Stage changes
   git add .

   # Commit with descriptive message
   git commit -m "feat: add validation feature"
   ```

3. **Backup Strategy**

   ```bash
   # Clean outputs before backup
   uv run python -c "from pathlib import Path; from infrastructure.core.files import clean_output_directories; clean_output_directories(Path('.'), '{name}')"

   # Backup source files only
   tar -czf project_backup.tar.gz projects/{name}/src/ projects/{name}/tests/ projects/{name}/scripts/ projects/{name}/manuscript/ docs/
   ```

### Adding Features

1. **Business Logic** → Add to `projects/{name}/src/`
2. **Tests** → Add to `projects/{name}/tests/`
3. **Scripts** → Add to `projects/{name}/scripts/` (use `projects/{name}/src/` methods)
4. **Documentation** → Update relevant `.md` files
5. **Validation** → Ensure coverage requirements met

### Performance Optimization

- **Parallel Testing**: Use `pytest-xdist` for faster test runs
- **Caching**: Enable pytest caching for repeated runs
- **Incremental Builds**: Only rebuild changed components
- **Performance Monitoring**: Automatic bottleneck detection in pipeline summary
- **Resource Tracking**: Memory and CPU usage reporting (when enabled)

See [`docs/operational/config/performance-optimization.md`](docs/operational/config/performance-optimization.md) for optimization guide.

### Checkpoint and Resume

The pipeline includes automatic checkpointing for resume capability:

```bash
# Resume from last checkpoint
uv run python scripts/runner/execute_pipeline.py --project {name} --core-only --resume
./run.sh --pipeline --resume

# Start fresh (clears checkpoint on success)
uv run python scripts/runner/execute_pipeline.py --project {name} --core-only
./run.sh --pipeline
```

**Features**:

- Automatic checkpoint after each successful stage
- Checkpoint validation before resume
- Graceful handling of corrupted checkpoints
- Preserves pipeline start time and stage durations

See [`docs/operational/config/checkpoint-resume.md`](docs/operational/config/checkpoint-resume.md) for documentation.

## 📚 References

### Internal Documentation

- [`README.md`](README.md) - Project overview and quick start
- [`docs/CLOUD_DEPLOY.md`](docs/CLOUD_DEPLOY.md) - **Headless / cloud server deployment** ☁️
- [`docs/RUN_GUIDE.md`](docs/RUN_GUIDE.md) - Full pipeline orchestration reference
- [`docs/core/how-to-use.md`](docs/core/how-to-use.md) - Usage guide
- [`docs/core/architecture.md`](docs/core/architecture.md) - System design details
- [`docs/core/workflow.md`](docs/core/workflow.md) - Development workflow
- [`projects/README.md`](projects/README.md) - Multi-project management guide
- [`docs/documentation-index.md`](docs/documentation-index.md) - Full documentation hub

### External Resources

- [Pandoc Manual](https://pandoc.org/MANUAL.html) - Document conversion
- [LaTeX Wikibook](https://en.wikibooks.org/wiki/LaTeX) - LaTeX documentation
- [Python Testing](https://docs.pytest.org/) - Testing framework

## 🎯 Best Practices

### Development Workflow

1. Write tests first (TDD)
2. Ensure coverage requirements met
3. Follow thin orchestrator pattern
4. Validate all outputs
5. Update documentation
6. Commit with clear messages

### Project Structure

- **Working outputs**: `projects/{name}/output/` (generated during pipeline)
- **Final deliverables**: `output/{name}/` (copied by stage 5)
- **Source code**: `projects/{name}/src/`
- **Tests**: `projects/{name}/tests/`
- **Scripts**: `projects/{name}/scripts/`
- **Manuscript**: `projects/{name}/manuscript/`

### Code Quality

- **Type Hints**: All public APIs must have type annotations
- **Documentation**: Clear docstrings for all functions
- **Error Handling**: Graceful failure with informative messages
- **Consistency**: Follow established patterns and conventions

### System Reliability

- **Deterministic Outputs**: All generation must be reproducible
- **Validation**: Check all aspects of output quality
- **Error Recovery**: Handle failures gracefully with clear messages
- **Performance Monitoring**: Track execution time and resource usage

---

## ✅ System Status: OPERATIONAL

**All systems confirmed functional with exemplar projects:**

- ✅ **Multi-project pipeline**: default **10**-stage DAG in `pipeline.yaml` (including LLM stages) + optional executive reporting for `--all-projects`
- ✅ **Test coverage excellence**: All active projects meet coverage requirements
- ✅ **Publication-quality outputs**: Professional PDFs, cross-referenced manuscripts, automated figures
- ✅ **Mathematical rigor**: Advanced equations, theorem proofs, convergence analysis
- ✅ **Testing**: Edge cases, performance benchmarks, type safety validation
- ✅ **Documentation**: AGENTS.md/README.md across all directories
- ✅ **Data testing**: Zero mocks, integration testing
- ✅ **Infrastructure robustness**: Fixed critical bugs, improved error handling

**Environment Management:**

- ✅ Matplotlib auto-configuration (headless operation via MPLBACKEND=Agg)
- ✅ Optional dependency handling (python-dotenv graceful fallback)
- ✅ Test failure tolerance (MAX_TEST_FAILURES environment variable)
- ✅ LaTeX path management (BasicTeX/MacTeX support)
- ✅ Docker containerization (`infrastructure/docker/Dockerfile` + `docker-compose.yml`)

**Modules (v2.1):**

- ✅ Security System (tests) - Input sanitization and monitoring
- ✅ Health Check System (tests) - System health monitoring
- ✅ Input Sanitization (tests) - LLM prompt validation
- ✅ LLM Integration — tests in `tests/infra_tests/llm/` (measured coverage → [`docs/development/coverage-gaps.md`](docs/development/coverage-gaps.md))
- ✅ Rendering System — tests in `tests/infra_tests/rendering/` (measured coverage → [`docs/development/coverage-gaps.md`](docs/development/coverage-gaps.md))
- ✅ Publishing API (integrated) - Zenodo, arXiv, GitHub automation
- ✅ Multi-project architecture (projects/{name}/ structure)

**Audit Status:**

- ✅ **High code coverage** across infrastructure and project gates (live % → [`docs/_generated/COUNTS.md`](docs/_generated/COUNTS.md), [`docs/development/coverage-gaps.md`](docs/development/coverage-gaps.md))
- ✅ Zero mock methods - all tests use data and HTTP calls
- ✅ All .cursorrules standards implemented
- ✅ compliance with thin orchestrator pattern
- ✅ Production-ready build pipeline (core pipeline + full pipeline with optional LLM stages)
- ✅ Reproducible outputs (deterministic with fixed seeds)
- ✅ Graceful degradation for optional features
- ✅ Multi-project support (projects/{name}/ structure)
- ✅ manuscript reference validation (all citations, figures, equations, sections resolved)
- ✅ HTTP testing with pytest-httpserver (no mocks for API calls)


<!-- BEGIN:STAGE_SUMMARY -->
The default [`pipeline.yaml`](infrastructure/core/pipeline/pipeline.yaml) declares **17 named stages** (indices 0–16). Default full runs execute **10** core+LLM stages; `--core-only` executes **8**. Opt-in tags (`archival`, `bundle`, `docxplus`, `ebook`, `metadata`, `provenance`, `science`) stay out of those default runs unless a stage is invoked directly. YAML stage indices do not match `stage_NN_*.py` prefixes.
<!-- END:STAGE_SUMMARY -->

<!-- BEGIN:STAGE_TABLE -->
<!-- This block is generated from [`infrastructure/core/pipeline/pipeline.yaml`](infrastructure/core/pipeline/pipeline.yaml) by `scripts/docgen/stage_table.py`. Do not hand-edit. Stage indices are **0-based positions in the YAML** and intentionally do **not** match the `scripts/pipeline/stage_NN_*.py` numeric prefixes (for example, stage 11, "Copy Outputs", runs `scripts/pipeline/stage_05_copy.py`). -->

| Stage | Script | Tags | Failure mode |
| ----- | ------ | ---- | ------------ |
| **0** Clean Output Directories | built-in `_run_clean_outputs` | `core`, `clean` | soft fail |
| **1** Environment Setup | `scripts/pipeline/stage_00_setup.py` | `core` | hard fail |
| **2** Infrastructure Tests | `scripts/pipeline/stage_01_test.py --infra-only --verbose --infra-scope pipeline-smoke` | `core`, `tests` | configurable tolerance |
| **3** Project Tests | `scripts/pipeline/stage_01_test.py --project-only --verbose` | `core`, `tests` | configurable test-failure tolerance; zero-test, project-local coverage, verifier-receipt/evidence, and internal runner failures hard fail |
| **4** Project Analysis | `scripts/pipeline/stage_02_analysis.py` | `core` | hard fail |
| **5** Connector Search | `scripts/pipeline/stage_08_connector_search.py` | `science` | skipped if not configured |
| **6** Provenance Record | `scripts/pipeline/stage_09_provenance_record.py --stage Connector Search` | `provenance` | skipped if not configured |
| **7** PDF Rendering | `scripts/pipeline/stage_03_render.py` | `core` | hard fail |
| **8** Output Validation | `scripts/pipeline/stage_04_validate.py` | `core` | enabled-format, enabled-PDF bookend, and artifact/provenance failures block; markdown, general output structure, and prose-quality checks remain advisory |
| **9** LLM Scientific Review | `scripts/pipeline/stage_06_llm_review.py --reviews-only` | `llm` | skipped if Ollama absent |
| **10** LLM Translations | `scripts/pipeline/stage_06_llm_review.py --translations-only` | `llm` | skipped if Ollama absent |
| **11** Copy Outputs | `scripts/pipeline/stage_05_copy.py` | `core` | soft fail |
| **12** Ebook Generation | `scripts/pipeline/stage_11_ebook.py` | `core`, `ebook` | soft fail |
| **13** docxplus Export | `scripts/pipeline/stage_13_docxplus.py` | `core`, `docxplus` | soft fail |
| **14** Metadata Package | `scripts/pipeline/stage_12_metadata.py` | `core`, `metadata` | soft fail |
| **15** Executable Bundle | `scripts/runner/bundle_executable.py` | `bundle` | soft fail |
| **16** Archival Publication | `scripts/runner/archive_publication.py` | `archival` | soft fail |
<!-- END:STAGE_TABLE -->

<!-- foam-orphan-nav:start (hand-maintained: links sub-docs so they are reachable; no generator refreshes or validates this block) -->

## Directory & sub-document map

Navigation links to in-tree documents (keeps them discoverable):

- [ISA — HumOS Template Package: Agentic Operability](ISA.md)
- [Public Template Exemplars Knowledge Base](projects/templates/AGENTS.md)
- [Public Template Design System](projects/templates/DESIGN.md)
- [Publishing Scripts](scripts/publish/README.md)

<!-- foam-orphan-nav:end -->

Discussion

Did this work in your project? Say what you used it for and what you changed. People and their agents can both post here.

Posts are public.Sign in to post

No one has posted yet. Be the first.