add-model-arch
Skill83 starsChanged 2 days agoMITView on GitHub
New model architecture, porting a model family, or a model that loads but outputs wrong - "add support for <model>", loader detection, chat template, tokenizer parity, RoPE variant, "outputs garbage", "prompt-blind", "NaN logits", "does it fit in VRAM". Not kernel perf (sm120-cuda-expert), not quant formats (quant-formats).
In this repository16 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- tests/CLAUDE.mdCLAUDE.md
- benchmark-cudaSkill
- building-and-testingSkill
- check-degenerationSkill
- codebase-auditSkill
- code-graphSkill
- docs-layersSkill
- docs-syncSkill
- find-stubsSkill
- numerics-bisectSkill
- quant-formatsSkill
- server-apiSkill
- shipping-prsSkill
- sm120-cuda-expertSkill
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 12
Start with scripts/check_model.sh <hf-repo> [--nvfp4] [--bos]: download, HF BF16 PPL (CPU), imp PPL, /tokenize ids vs HF, reasoning split, …
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Adding a Model Architecture - imp
First: is it a new arch at all?
Diff config.json against a supported sibling before estimating. Qwen3.8 shipped with zero enum members (loads as QWEN35); the work was tokenizer parity, template goldens, KV dtype default and MTP head (#1750). HF reference values come from curl, never from memory.
Start with scripts/check_model.sh <hf-repo> [--nvfp4] [--bos]: download, HF BF16 PPL (CPU), imp PPL, /tokenize ids vs HF, reasoning split, json_object; one table, exit 1 on a FAIL row. A checkpoint whose rows are all ok needs only its MODELS.md row (MiniCPM5-2B, LFM2.5-24B-A2B).
Integration checklist (gpt-oss #572 is the reference PR)
| Step | Where | Notes |
| 1. Enum + registry | ModelArch in src/model/model_arch.h; parse_model_arch (GGUF general.architecture in gguf_loader.cpp, HF architectures/model_type in hf_config_loader.cpp; arch-only config.json keys: one file in src/model/hf_config/ + a kHfConfigHooks row in arch_registry.cpp), model_arch_name, apply_arch_defaults, sampling defaults in src/model/model.cpp | then ModelProfile (src/model/model_profile.h/.cpp, SSoT since #622/#623; AttnVariant { STANDARD, GEMMA4_SWA, GPTOSS_SWA, NOPE, MLA }). No new cfg.arch == X in hot paths. Add the arch to the KV-dtype lists kv_nvfp4_default_safe, kv_fp8_hint_default_safe, kv_fp8_no_hint_default_safe (evidence per family in model.cpp): a missing entry silently gets FP16 KV, which on a GDN hybrid gates context |
| 2. Loader | src/model/tensor_kind_matcher.cpp, weight_map.cpp; SafeTensors safetensors_loader.cpp; NVFP4 via llm_compressor_loader.cpp (compressed-tensors) or Modelopt (hf_quant_config.json) | the two NVFP4 layouts have RECIPROCAL tensor scales (quant-formats) |
| 3. Arch config | model_config.h + apply_arch_defaults | RoPE pair layout (NeoX vs GPT-J), YaRN/rope_freq_scale, SWA layer pattern, NoPE, sinks, softcap, norm placement, MoE router |
| 4. Chat template | family in src/model/chat_template_families.cpp (ChatTemplateFamily in chat_template.h), rendering in chat_template.cpp (+ jinja.cpp) | a new family needs a golden pin (make chat-goldens, tests/refs/chat_template_goldens.h, nine families since #1721, three more Jinja gaps fixed in #1701; Jinja fails SILENTLY). reasoning_effort must reach ChatTemplate::apply*/render_jinja + the server snapshot: identical prompt-token counts across efforts = not threaded (#1750: 67/67 before, 41/11/53 after) |
| 5. Tokenizer parity | template tests/test_tokenizer_qwen38.cpp (32/32 encode+decode vs HF), tests/test_qwen38_chat_template.cpp; harness tools/tokenizer_parity/run.sh (from #2159, branch fix/tokenizer-hf-parity until merged) | BERT-family GGUFs use SPM, not WordPiece. An HF reference built from a GGUF tokenizer can split special tokens (<think> -> 6 text ids instead of 151667): compare token ids before logits |
| 6. Kernels | only for genuinely new ops; check src/exec/ + src/compute/ first | new RoPE variants go into src/compute/rope_yarn.cuh (shared with MTP heads, #913) |
| 7. Verify | loads -> greedy coherent (check-degeneration) -> imp-cli --perplexity vs HF (within ~10-20%, often closer: gpt-oss 4.68 vs bf16 4.607, #663) -> decode/prefill sanity (benchmark-cuda) | PPL with runtime.deterministic=true, speculative.mtp_k=0, ppl_corpus_45k.txt (tools/analysis/make_ppl_corpus.sh) |
| 8. Docs | docs/MODELS.md row (+ docs/BENCHMARKS.md if hero); known gaps to docs/LIMITATIONS.md; perf baseline entry if gated |
A new checkpoint is UNTRUSTED INPUT: SafeTensors/tokenizer.json parsers are hardened (#1660, #1694); fuzz targets under fuzz/; no parsing shortcuts around the bounds checks.
Wrong-output fingerprints
| Symptom | Root-cause class | Case |
| Fluent but ignores the prompt ("prompt-blind") | RoPE pair layout: HF SafeTensors need rope_neox=true, GGUF pre-permutes Q/K | SafeTensors Llama/Mistral, #503 |
| Words fine, digits scrambled | position encoding (NoPE layer as RoPE or vice versa) | Nemotron-H rope_attn_disabled, #518 |
| Argmax always token 0 | NaN logits upstream (residual overflow, bad scale) | gpt-oss FP16 residual |
| Coherent to ~1k ctx, then garbage | YaRN/rope_freq_scale inverted or fused-rope path without YaRN | gpt-oss 1024x error, #572 |
| Wrong only with chunked prefill at long ctx | continuation-chunk path | #553 |
| Wrong language / valid-but-wrong tokens | weight upload / dequant layout (MoE: weight_upload.cpp expert promotion first) | Qwen3.6-35B NVFP4, #925 |
Garbage from token 0 (!!!) | silent VRAM-alloc failure in a decode fallback | MXFP4 GDN hybrids, #935 |
| Multimodal: describes a DIFFERENT picture | M-RoPE per-token (t,h,w) layout, src/model/mrope_positions.cpp | Qwen3-VL |
| Vision fluent but generic | tower loaded partly or embeddings never reach the sequence: tools/analysis/vision_sight_check.py | |
| Drift only at very long positions | YaRN float trap: __sinf/__cosf on an unreduced argument; long-ctx tests run ext_factor=0 and cannot see it | #1704 |
| Coherent-ish, drifts vs HF | MLA/YaRN rope_mscale on the wrong base (1.261x); pin the transformers oracle (4.44.2 for MLA) | #880 |
| Draft accept collapses, output correct | MTP head RoPE differs from the target (NeoX vs YaRN) | #913 |
| CLI fine, server broken | not arch: server-api | |
| Which block diverges | diagnostics.dump_hidden_dir + tools/analysis/layer_ab_diff.py (two imp runs, or imp vs an HF dump); dump_gdn_state_dir, dump_final_logits_dir | 0 non-finite GDN states over a 46579-token prefill on Qwen3.8 |
Known traps
rope_neox: GGUF converters pre-permute Q/K; HF SafeTensors do not.swa_layerswas Gemma-only hardcoded once; verify per-layer attention type on any interleaved-SWA arch.- The fused rope+KV-write kernel must apply the same YaRN scaling as the standalone path.
- Arch control tokens (Harmony channels) must not land on the banned list; the spec-verify argmax applies the mask since #1796.
- Gemma-4: per-layer
rope_freqsfor non-SWA layers,n_rot=hd. - GDN/hybrid state: BF16 storage + FP32 arithmetic is the default (
gdn.state_bf16, #1776/#1778); FP16 state NaNs at depth; the old "must be FP32" was a layout bug. - Multimodal checkpoints wrap the LM under
model.language_model.*(Qwen3.5-VL, #647): strip in the loader or every tensor "is missing". - Encoder/embedding archs are supported (nomic-bert #867, cosine 0.999 vs HF).
- VRAM arithmetic: card 32 607 MiB; CUDA primary context ~1680 MiB; library reserve is a per-model MEASURED value cached in
src/memory/library_reserve_cache.h(#1119;vram.library_reserve_cache, overridevram.library_reserve_mb); thekMeasuredLibraryReserveBytes~3900 MiB constant insrc/memory/plan.his the first-run fallback (measured 0 MiB on Qwen3-4B-IQ4_NL, 7460 on Qwen3-8B-Q8_0). An MTP head adds ~0.79 GiB whenspeculative.mtp_kauto takes it. Peaks per config:docs/internals/MEMORY.md. - Getting a checkpoint onto the NVFP4 path:
scripts/stage-model.sh(download +imp-quantizein one command; imp fetches nothing itself).
After it works
make verify-fast; a DegenerationTest-compatible probe prompt; think-channel archs through tools/analysis/degen_suite.py (think-leak); vision through vision_sight_check.py and make test-vision.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

