numerics-bisect
Skill77 starsChanged 8 days agoMITView on GitHub
Outputs differ between two paths that should agree - chunk sizes, prefill vs decode, prefix-cache resend, imp vs vLLM vs HF, two builds/kernels, "first token differs", "logprobs moved", "which layer diverges", near-tie. Not loops/garbage (check-degeneration), new arch (add-model-arch), perf (benchmark-cuda).
In this repository16 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- tests/CLAUDE.mdCLAUDE.md
- add-model-archSkill
- benchmark-cudaSkill
- building-and-testingSkill
- check-degenerationSkill
- codebase-auditSkill
- code-graphSkill
- docs-layersSkill
- docs-syncSkill
- find-stubsSkill
- quant-formatsSkill
- server-apiSkill
- shipping-prsSkill
- sm120-cuda-expertSkill
The file
Files
Numerics bisect - imp
Order is fixed: ids, logprobs, hidden states, reference. Each rung is cheaper than the next and kills a class of false leads.
Rungs
| # | Step | Tool | Decision |
| 1 | Compare token ids of both prompts first | imp-cli / server token ids vs the other path | ids differ = tokenizer or template bug (add-model-arch step 5), stop here |
| 2 | Logprob grid over the axis (chunk size, engine, build) | first-token logprobs per arm; chunk axis via --prefill-chunk-size <n> (runtime.prefill_chunk_size); raw logits via diagnostics.dump_logits_dir | a spread across the axis is the signal; greedy-text equality alone amplifies near-ties |
| 3 | Hidden-state dumps per layer | diagnostics.dump_hidden_dir + tools/analysis/layer_ab_diff.py (two imp runs) | first block with non-zero added divergence (rel@out - rel@in) owns the bug |
| 4 | Reference arbitration | HF transformers fp32 (GGUF: from_pretrained(..., gguf_file=...), dequantized) plus vLLM on the same SafeTensors checkpoint | the arm that matches both references is right |
| 5 | Judge near-tie tokens by multi-token NLL | imp-cli --perplexity tools/analysis/ppl_corpus_45k.txt (rebuild with tools/analysis/make_ppl_corpus.sh), --set runtime.deterministic=true both arms | never by one token: llama.cpp FA on vs off alone moved a near-tie token by 0.082 |
Dump traps
| Trap | Fix |
[DUMP_NPY] open failed | docker --user $(id -u):$(id -g) so the container can write the mounted dump dir |
| Dumps per stage, overwritten per forward | names carry layer, step and n, not the prefill offset: imp_L<layer>_step<step>_n<n>.bin (src/exec/executor_forward.cu), imp_step<step>_L<layer>_<tag>.npy per stage (src/exec/executor_debug.h); prefill is step 0, so chunks of equal n overwrite each other: copy the dir between forwards and key it by prefill offset for chunk studies |
| Default layer set | diagnostics.dump_hidden_dir=<path> dumps layers 0/5/15/29; =all dumps every layer into /tmp |
Known causes of chunk-dependent rows
- Kernel variant chosen by row count (sm120-cuda-expert numerics rules, #2152).
- MoE router GEMM on cuBLAS picked its algorithm by n; now
gemm_gate_fp32_rows(#2167). The NVFP4 act quantize is row-local. - hd=512 prefill attention (Gemma-4 global layers) on cuBLAS by n;
runtime.deterministicruns the fixed-order FMHA (#2167). - Resend check:
tools/analysis/prefix_resend/prefix_resend_probe.py(check-degeneration, from #2171).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.