agentleFS
kekzlkekzl/imp/.claude/skills/building-and-testing/SKILL.md

building-and-testing

Skill83 starsChanged 2 days agoMITView on GitHub

Build, test suite, CI status, build/test failures - make build, test-gpu, verify-fast, GTEST_FILTER, Docker/CUDA toolchain, dependency bumps, "CI is red", "which gate failed", stale objects / segfault after a header edit, hooks, docker-entrypoint env vars, determinism, perplexity. Not perf (benchmark-cuda), not output quality (check-degeneration).

  • Reads credentials
  • Deletes or force-pushes
  • Commits and pushes

1 other project uses a version of this.

Adopted by

Most recently taken up by

The file

Files
SKILL.md

Build & Test - imp

Hard rules

#RuleDetail
1No CUDA toolkit on the hostEverything runs in Docker. Toolchain nvidia/cuda:13.5.0-devel-ubuntu26.04, GCC 15.2, C++23 for host AND device (CMAKE_CUDA23_STANDARD_COMPILE_OPTION shim at the top of CMakeLists.txt).
2build/ and build-dev/ are root-ownedmake dev-clean, or docker run --rm -v $PWD:/src -w /src ubuntu rm -rf build. Never sudo.
3No --mount=type=cache on the build dir in the Dockerfile03a2cc19: ninja reused stale objects. The ccache mount (id=imp-ccache) is content-addressed and stays; docker builder prune empties it, which only costs one cold build.
4models/ in the repo is a symlink farmCustom docker run mounts $HOME/models:/models; Makefile targets already do.
5Dependency pins live ONLY in cmake/imp-deps.cmakeCUTLASS, GTest, httplib, nlohmann/json, each a TAG plus the SHA the build fetches. make build injects both via scripts/dep_build_args.sh. check_dep_pins.sh runs offline in the deps gate (drift, missing SHA, unpinned uses:) and --online in Lint (tag must still resolve to the pinned commit).
6CI has no GPU runnerTest job skipped unless repo var HAS_GPU_RUNNER=true. GPU correctness and perf are gated LOCALLY, by hand or with IMP_HOOK_GPU=1 on the hooks: pre-commit (make build + GPU modules or the full make test-gpu), pre-push (make verify-fast, plus the paired A/B make verify-ab on kernel diffs). The hook prints the skipped command; run it before the PR.
6agit commit / git push are CPU jobsBoth hooks run make preflight only. With IMP_HOOK_GPU=1 they and make GPU targets refuse while another worktree holds scripts/gpu_lock.sh (scripts/require_free_gpu.sh, hook_gpu in scripts/hook_lib.sh), then check load. Holding the card across jobs: scripts/gpu_lock.sh acquire "<purpose>" in your worktree, release after, status shows the holder (lock /tmp/gpu-lock, shared with ~/.claude/skills/gpu-stats/gpu-lock.sh, keeper TTL 4 h; a live pid in the old /tmp/imp-gpu.lock also refuses). Still send the START/END handshake to other sessions. No --no-verify: the hooks touch no GPU by default. The skipped GPU tests are owed: run them and report the result in the PR (CI Test never runs: .github/workflows/ci.yml:999).
7Required check = Build, ruleset 14716423Static gates run as its FIRST step (scripts/ci_static_gates.sh, unfiltered) and block the merge since #1527. Rename the job without the ruleset and every PR sits at mergeStateStatus=BLOCKED.
8main's CI status is stale by defaultAuto-merge squashes with GITHUB_TOKEN, which starts no workflow run. Judge a fix by the PR run or the file; refresh with gh workflow run CI --ref main.

Command table

TaskCommandTime
Incremental build (inner loop)make dev2-14 s
Incremental build + CI unit lanemake dev-test (= ctest -L unit)~3 s
Full image (the gate; anything you measure or push)make build -> imp:test (main checkout) or imp:test-<dir>-<hash8> (worktree, scripts/image_tag.sh)~6 min cold; 10-40 s from a warm ccache (595 TUs, 2026-09-11); 0 s when the image already carries the tree (imp.tree label)
CPU unit binarymake test-unit (imp-tests-unit)<5 s
Full GPU suitemake test-gpu4-10 min
E2E on real modelsmake test-e2eSafeTensors only: Qwen3-8B NVFP4, Qwen3.8-27B NVFP4, Gemma-4-26B-A4B NVFP4 (also MOE_MODEL, the determinism suite) (paths in Makefile)
GDN restore group of test-e2e (a scripts/check-release.sh stage)make test-hybrid-restoreQwen3.5-4B-mxfp4; a skip fails the stage
Server e2e (the ONLY place handlers/batching run end to end)make test-serverincludes degen_suite.py and tests/test_server_tracing.py
Rerank / agents / external harnessesmake test-rerank, make test-agents, make test-agents-externalGPU + local model
NIAH retrieval gate, spec fidelitymake test-niah, make test-spec-fidelityspec-fidelity needs ~26 GB free
Vision goldensmake test-vision (IMP_VISION_GOLDEN_DUMP=1 regenerates)
Chat-template goldensmake chat-goldens (network, writes tests/refs/chat_template_goldens.h)
Pre-push gatemake verify-fast18 s without perf gate, 36 s with
Full / chunked / north-star gatesmake verify, make verify-chunked, make verify-north-star~5 min full
Kernel register/spill ratchet (CI kernels gate)make kernel-resources, re-pin make kernel-resources-updateneeds libimp.a
clang-formatmake format-check; never bare make formatrepo is not format-clean
clang-format, changed lines only (what CI checks)per file: ranges from git diff -U0 main -- f, then docker run --rm --user $(id -u):$(id -g) -v $PWD:/work -w /work $CLANG_FORMAT_IMG clang-format --style=file -i --lines=a:b ... f (--dry-run to check)image name: CLANG_FORMAT_IMG in Makefile
Host ASan/UBSan over test-core/test-textmake asanworks on WSL2
compute-sanitizermake sanitizedoes NOT work on WSL2 (WDDM)
Alloc-interpose censusmake check-alloc-interpose (build-interpose/)never benchmark it (reads ~3% low)
Server gcovmake coverageGPU + model
API mock contractpytest tests/api/ or tests/api/run_mock_tests.shno GPU; the mock lane also runs the nomodel tests
GPU free?make check-gpu (scripts/require_free_gpu.sh: utilisation AND memory)

Filtered GPU run (GTEST_FILTER is not threaded through make test-gpu):

docker run --rm --gpus all -v $HOME/models:/models \
  -e IMP_TEST_MODEL=/models/Qwen3-8B-Q8_0.gguf \
  imp:test imp-tests --gtest_filter="DegenerationTest.*"

DetEvalE2ETest.* matches 0 tests and prints PASSED; the working filter is *DetEvalE2ETest*.

Binaries

WhereWhat
imp:test imageimp-server, imp-cli, imp-bench, imp-quantize, imp-tests (full GPU), imp-tests-unit (generated wrapper = ctest -L unit: test-core + test-text + e2e unit filter), modules test-core test-text test-compute test-attention test-quant test-kv test-moe-gdn test-e2e
build-dev/ (host, from make dev)same binaries; run them inside imp:toolchain with -v $PWD:/src -w /src (that mount hides the image's own /src)

tests/test_ssm.cu -> test-moe-gdn. SamplingTest -> test-compute. A new CPU test belongs in test-core. Lane census: python3 tools/check_test_lanes.py --report (the Test lanes check fails on a test in no ctest entry or a unit-lane test demoted vs the base; a new GPU test in a gpu module needs no edit). The unit lane has no skips (guard_unit_skips in ctest -L unit fails on any): a test that needs a GPU or a model file goes to test-kv or to test-e2e outside _unit_e2e_filter; fixture files live in tests/fixtures/ (IMP_TEST_FIXTURES_DIR). A new tool binary needs a cp in the builder stage AND a COPY --from=builder line in the Dockerfile.

Gates and hooks

Gate groupRuns whereContent
filesizeBuild, hookscheck_filesize.py ([allow] pin = ceiling, +25 slack, shrink free), check_determinism_sites.py, check_dead_inline_accessors.py, check_log_fatal.py
lanesBuild, hooks, own check Test lanes (#1770)check_test_lanes.py --report (macros per lane, no-lane delta vs base); guard_unit_skips inside ctest -L unit fails on any runtime skip
entrypointBuild, hookstests/test_entrypoint.sh drives docker-entrypoint.sh against a stub (25 assertions)
allocBuild, hookscheck_alloc_sites.py + check_alloc_pairs.py (allowlist is two-way: removing a site needs the allowlist edit)
kernelsBuild only (needs artifact)make kernel-resources vs tools/kernel_resource_baseline.txt (REG >= 240 or non-zero local frame)
launchguardsBuild, hookscheck_launch_guards.py
docsBuild, hooks, Docs jobsync_docs.py --check, docs_lint.py
citationsBuild, hookscheck_doc_citations.py over all living docs (#1783)
hygieneBuild, Release hygiene job onlycheck-release.sh with SKIP_VERIFY=1 (also resolves relative links in docs)

Advisory jobs: Lint, clang-tidy, Mock API contract, Real API contract (model-less), Sanitizers, PTX fallback.

  • Hooks: make install-hooks copies scripts/pre-commit.hook and scripts/pre-push.hook into git rev-parse --git-path hooks (shared by all worktrees). Never edit .git/hooks/*: guard_precommit_filter in the CPU lane diffs them against the tracked files.
  • make build runs scripts/build_image.sh: skipped when the worktree image already carries the tree and build args (labels imp.tree = scripts/image_tag.sh tree, unstaged and untracked files included; imp.build; imp.worktree). IMP_FORCE_BUILD=1 builds anyway. Runners refuse an image whose imp.tree is not this tree (IMP_ALLOW_FOREIGN_IMAGE=1 overrides). Saves 3.5 min per hook stage on one tree.
  • Pre-commit = make preflight (~20 s); GPU step with IMP_HOOK_GPU=1 only: make build, then only the test modules the staged diff reaches (modules_for in scripts/pre-commit.hook: src/memory|runtime -> test-core test-kv test-e2e, src/compute -> test-compute test-attention test-quant test-kv test-moe-gdn, src/exec -> test-e2e test-moe-gdn test-compute test-attention, src/model|quant -> test-quant test-text test-e2e test-core, a tests/ file -> its module); the full make test-gpu (3108 tests in imp-tests --gtest_list_tests, 2026-09-28) on include/ src/core src/api CMakeLists cmake/ changes or with IMP_PRECOMMIT_FULL=1. .md .py .hook skip.
  • Pre-push = make preflight; GPU step with IMP_HOOK_GPU=1 only: require_free_gpu.sh, then make verify-fast; the perf gate runs only when the diff matches PERF_RE (src/{compute,exec,quant,runtime,model}/, any .cu/.cuh, cmake/, CMakeLists.txt, tests/perf_baseline*). On src/compute or src/exec diffs the paired A/B (make verify-ab, 2 %, ~5 min) replaces the single-arm perf gate; IMP_SKIP_AB=1 falls back to it. Skips: IMP_VERIFY_SKIP_PERF/_VRAM/_GRAPHS=1.
  • Hooks run make preflight: every ci_static_gates.sh group, hygiene included.
  • docs_lint.py writes docs/audit/docs-rewrite/STALE.md only with --write-stale; hooks and CI never touch it (#2194).

Traps

SymptomCauseFix
Segfault / cudaFree on garbage / MoEExecutorTest 5/5 red after a header or class-layout editstale .o despite ninjagrep -rl exec/executor.h src tools tests | xargs touch && make dev (healed 3/3)
Link error __cudaRegisterLinkedBinary after make devsame classtouch includers, rebuild
Gate result contradicts the treemake build COPYs the tree at start (#1531); verify-fast never builds (SKIP build line)rebuild after every edit or amend, then gate
Bench numbers move while a chain runsmake dev rebuilds build-dev/ under the running binarybench from copies (-v dir:/bin_arm)
NvFP4SmallMTest.BandwidthAboveStarvationFloor red in the full suitecontention in the run (401 vs 537.6 GB/s threshold; isolated 590-622)rerun isolated
NvFP4SmallMV2Test.BandwidthAboveStarvationFloor red in the pre-commit suitesame class: 39.68 % vs the 40 % floor, isolated 73-78 % (3/3, 2026-09-22)rerun isolated (docker run --rm --gpus all imp:test test-quant --gtest_filter=...); only with IMP_HOOK_GPU=1 does the hook run it
ci_static_gates.sh in imp:toolchain: FAIL registers + local frame vs the pin, docker: command not foundthe kernels group shells out to dockerrun make kernel-resources on the host (reads build-dev/libimp.a: make dev first)
test-core fails to link in Sanitizers / make asansources inside the IMP_BUILD_SERVER block (#1821)keep test-core's deps outside that block
--perplexity OOMs or pool floors on a head checkpointmtp_k=auto loads the MTP head (+0.79 GiB)--set speculative.mtp_k=0 in PPL harnesses
PPL differs 0.35% between runs of the SAME binarynon-deterministic forward--set runtime.deterministic=true on both arms (implies deterministic_gemm)
std::sort etc. "not declared"libstdc++ 15 dropped transitive <algorithm> (#906)add the include
nvcc arch errortarget is compute_120a/sm_120a + optional compute_120f PTX (IMP_SM120_FLAGS)never generic sm_120 (sm120-cuda-expert)
Compose/container ignores a config keyonly IMP_CONFIG / IMP_SET reach every key (#1823); the 19 legacy IMP_* names in docker-entrypoint.sh are frozen-e IMP_SET="key=v key2=v"; a bogus key fails at start (no such key)
Value looks like an unread env varengine reads only IMP_CONFIG IMP_DETERMINISTIC IMP_FMHA_FA2 IMP_WORKER_TIMING IMP_SPEC_TRACE IMP_PPL_DUMP; the entrypoint translates the restgrep docker-entrypoint.sh too
git worktree remove failsroot-owned build-dev/ inside itrm -rf via container, git worktree prune; a worktree also needs the models/ symlink dir or verify-fast fails in the hook
FAIL model directory not found: /src/models from make verify-fast in a git worktreemodels/ symlink farm exists only in the main checkout (scripts/verify.sh:192)IMP_VERIFY_MODELS=$HOME/models make verify-fast (or ln -s $HOME/models models)
A gitignored ./imp.conf in the repo rootdev builds with -w /src load it, images do not (#1784 fixed the build context)read the imp.conf loaded from line in every log

Determinism and quality

  • --set runtime.deterministic=true: temp=0 reproducibility incl. MoE routing atomics and top-k races; implies runtime.deterministic_gemm; promoted process-wide since #542; default OFF. Qwen3.6-35B stays non-deterministic at temp=0 (routing flips); never assert exact output there.
  • 7 DISABLED_ tests with reasons: 2 determinism boundaries (cross-context GDN, FMHA smem), the rest benches without assertions. Do not re-enable.
  • imp-cli --perplexity <file> is teacher-forced prefill (chunk-aware since #553); use tools/analysis/ppl_corpus_45k.txt (untracked, rebuild with tools/analysis/make_ppl_corpus.sh), not the 199-token ppl_corpus.txt.
  • imp-cli --json: one JSON document on stdout, logs on stderr (#1715).

PR / merge

Ship flow, auto-merge race, release cut: skill shipping-prs.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.