agentleFS
kekzlkekzl/imp/.claude/skills/add-model-arch/SKILL.md

add-model-arch

Skill83 starsChanged 2 days agoMITView on GitHub

New model architecture, porting a model family, or a model that loads but outputs wrong - "add support for <model>", loader detection, chat template, tokenizer parity, RoPE variant, "outputs garbage", "prompt-blind", "NaN logits", "does it fit in VRAM". Not kernel perf (sm120-cuda-expert), not quant formats (quant-formats).

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Start with scripts/check_model.sh <hf-repo> [--nvfp4] [--bos]: download, HF BF16 PPL (CPU), imp PPL, /tokenize ids vs HF, reasoning split, …

    line 12

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Adding a Model Architecture - imp

First: is it a new arch at all?

Diff config.json against a supported sibling before estimating. Qwen3.8 shipped with zero enum members (loads as QWEN35); the work was tokenizer parity, template goldens, KV dtype default and MTP head (#1750). HF reference values come from curl, never from memory.

Start with scripts/check_model.sh <hf-repo> [--nvfp4] [--bos]: download, HF BF16 PPL (CPU), imp PPL, /tokenize ids vs HF, reasoning split, json_object; one table, exit 1 on a FAIL row. A checkpoint whose rows are all ok needs only its MODELS.md row (MiniCPM5-2B, LFM2.5-24B-A2B).

Integration checklist (gpt-oss #572 is the reference PR)

StepWhereNotes
1. Enum + registryModelArch in src/model/model_arch.h; parse_model_arch (GGUF general.architecture in gguf_loader.cpp, HF architectures/model_type in hf_config_loader.cpp; arch-only config.json keys: one file in src/model/hf_config/ + a kHfConfigHooks row in arch_registry.cpp), model_arch_name, apply_arch_defaults, sampling defaults in src/model/model.cppthen ModelProfile (src/model/model_profile.h/.cpp, SSoT since #622/#623; AttnVariant { STANDARD, GEMMA4_SWA, GPTOSS_SWA, NOPE, MLA }). No new cfg.arch == X in hot paths. Add the arch to the KV-dtype lists kv_nvfp4_default_safe, kv_fp8_hint_default_safe, kv_fp8_no_hint_default_safe (evidence per family in model.cpp): a missing entry silently gets FP16 KV, which on a GDN hybrid gates context
2. Loadersrc/model/tensor_kind_matcher.cpp, weight_map.cpp; SafeTensors safetensors_loader.cpp; NVFP4 via llm_compressor_loader.cpp (compressed-tensors) or Modelopt (hf_quant_config.json)the two NVFP4 layouts have RECIPROCAL tensor scales (quant-formats)
3. Arch configmodel_config.h + apply_arch_defaultsRoPE pair layout (NeoX vs GPT-J), YaRN/rope_freq_scale, SWA layer pattern, NoPE, sinks, softcap, norm placement, MoE router
4. Chat templatefamily in src/model/chat_template_families.cpp (ChatTemplateFamily in chat_template.h), rendering in chat_template.cpp (+ jinja.cpp)a new family needs a golden pin (make chat-goldens, tests/refs/chat_template_goldens.h, nine families since #1721, three more Jinja gaps fixed in #1701; Jinja fails SILENTLY). reasoning_effort must reach ChatTemplate::apply*/render_jinja + the server snapshot: identical prompt-token counts across efforts = not threaded (#1750: 67/67 before, 41/11/53 after)
5. Tokenizer paritytemplate tests/test_tokenizer_qwen38.cpp (32/32 encode+decode vs HF), tests/test_qwen38_chat_template.cpp; harness tools/tokenizer_parity/run.sh (from #2159, branch fix/tokenizer-hf-parity until merged)BERT-family GGUFs use SPM, not WordPiece. An HF reference built from a GGUF tokenizer can split special tokens (<think> -> 6 text ids instead of 151667): compare token ids before logits
6. Kernelsonly for genuinely new ops; check src/exec/ + src/compute/ firstnew RoPE variants go into src/compute/rope_yarn.cuh (shared with MTP heads, #913)
7. Verifyloads -> greedy coherent (check-degeneration) -> imp-cli --perplexity vs HF (within ~10-20%, often closer: gpt-oss 4.68 vs bf16 4.607, #663) -> decode/prefill sanity (benchmark-cuda)PPL with runtime.deterministic=true, speculative.mtp_k=0, ppl_corpus_45k.txt (tools/analysis/make_ppl_corpus.sh)
8. Docsdocs/MODELS.md row (+ docs/BENCHMARKS.md if hero); known gaps to docs/LIMITATIONS.md; perf baseline entry if gated

A new checkpoint is UNTRUSTED INPUT: SafeTensors/tokenizer.json parsers are hardened (#1660, #1694); fuzz targets under fuzz/; no parsing shortcuts around the bounds checks.

Wrong-output fingerprints

SymptomRoot-cause classCase
Fluent but ignores the prompt ("prompt-blind")RoPE pair layout: HF SafeTensors need rope_neox=true, GGUF pre-permutes Q/KSafeTensors Llama/Mistral, #503
Words fine, digits scrambledposition encoding (NoPE layer as RoPE or vice versa)Nemotron-H rope_attn_disabled, #518
Argmax always token 0NaN logits upstream (residual overflow, bad scale)gpt-oss FP16 residual
Coherent to ~1k ctx, then garbageYaRN/rope_freq_scale inverted or fused-rope path without YaRNgpt-oss 1024x error, #572
Wrong only with chunked prefill at long ctxcontinuation-chunk path#553
Wrong language / valid-but-wrong tokensweight upload / dequant layout (MoE: weight_upload.cpp expert promotion first)Qwen3.6-35B NVFP4, #925
Garbage from token 0 (!!!)silent VRAM-alloc failure in a decode fallbackMXFP4 GDN hybrids, #935
Multimodal: describes a DIFFERENT pictureM-RoPE per-token (t,h,w) layout, src/model/mrope_positions.cppQwen3-VL
Vision fluent but generictower loaded partly or embeddings never reach the sequence: tools/analysis/vision_sight_check.py
Drift only at very long positionsYaRN float trap: __sinf/__cosf on an unreduced argument; long-ctx tests run ext_factor=0 and cannot see it#1704
Coherent-ish, drifts vs HFMLA/YaRN rope_mscale on the wrong base (1.261x); pin the transformers oracle (4.44.2 for MLA)#880
Draft accept collapses, output correctMTP head RoPE differs from the target (NeoX vs YaRN)#913
CLI fine, server brokennot arch: server-api
Which block divergesdiagnostics.dump_hidden_dir + tools/analysis/layer_ab_diff.py (two imp runs, or imp vs an HF dump); dump_gdn_state_dir, dump_final_logits_dir0 non-finite GDN states over a 46579-token prefill on Qwen3.8

Known traps

  • rope_neox: GGUF converters pre-permute Q/K; HF SafeTensors do not.
  • swa_layers was Gemma-only hardcoded once; verify per-layer attention type on any interleaved-SWA arch.
  • The fused rope+KV-write kernel must apply the same YaRN scaling as the standalone path.
  • Arch control tokens (Harmony channels) must not land on the banned list; the spec-verify argmax applies the mask since #1796.
  • Gemma-4: per-layer rope_freqs for non-SWA layers, n_rot=hd.
  • GDN/hybrid state: BF16 storage + FP32 arithmetic is the default (gdn.state_bf16, #1776/#1778); FP16 state NaNs at depth; the old "must be FP32" was a layout bug.
  • Multimodal checkpoints wrap the LM under model.language_model.* (Qwen3.5-VL, #647): strip in the loader or every tensor "is missing".
  • Encoder/embedding archs are supported (nomic-bert #867, cosine 0.999 vs HF).
  • VRAM arithmetic: card 32 607 MiB; CUDA primary context ~1680 MiB; library reserve is a per-model MEASURED value cached in src/memory/library_reserve_cache.h (#1119; vram.library_reserve_cache, override vram.library_reserve_mb); the kMeasuredLibraryReserveBytes ~3900 MiB constant in src/memory/plan.h is the first-run fallback (measured 0 MiB on Qwen3-4B-IQ4_NL, 7460 on Qwen3-8B-Q8_0). An MTP head adds ~0.79 GiB when speculative.mtp_k auto takes it. Peaks per config: docs/internals/MEMORY.md.
  • Getting a checkpoint onto the NVFP4 path: scripts/stage-model.sh (download + imp-quantize in one command; imp fetches nothing itself).

After it works

make verify-fast; a DegenerationTest-compatible probe prompt; think-channel archs through tools/analysis/degen_suite.py (think-leak); vision through vision_sight_check.py and make test-vision.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.