llama-patch-changes

Use this skill when editing the llama.cpp patch queue, refreshing patches from a llama.cpp checkout, updating the pinned upstream SHA, or changing build scripts that prepare or consume patched llama.cpp.

Boundaries

• Keep durable llama-side changes in the ordered queue under skippy/llama_cpp/patches: top-level core patches first, model_support/series second, and generated/series last.
• Keep the upstream pin in skippy/llama_cpp/upstream.txt.
• Do not add a submodule, vendor a llama checkout, or depend on the old Mesh-LLM llama.cpp fork.
• Do not treat edits in .deps/llama.cpp as durable until the patch queue has been regenerated and committed.
• Do not add llama-stage ABI/static in-process patches unless the task explicitly asks for that integration pass.
• Prefer small, reviewable llama commits with one functional boundary per patch. Keep patch numbers unique and contiguous within each queue lane.
• Put a new model family's implementation, conversion, templates, multimodal integration, runtime adaptations, and family tests in one focused patch in model_support/. Do not spread family-specific code through core patches.
• Keep generated graph-semantics edits in generated/; do not hand-maintain them in either the core or model-support lane.
• Do not append a terminal patch whose only purpose is to split, move, or clean up code introduced by earlier patches. Recreate the affected patches so they use the intended ownership boundaries from the outset.

Local Flow

Prepare the pinned upstream checkout and current patch queue:

scripts/prepare-llama.sh pinned

For actual llama-side editing, prefer a normal llama.cpp checkout or branch where commits can be named and inspected. Base the branch on upstream ggml-org/llama.cpp master, then carry the Mesh-LLM patch commits on top.

For a deliberate queue rewrite, reconstruct capability-owned core commits from the pinned upstream, add model-family support commits, then add the generated family shards. Verify the reconstructed head is tree-identical to the authoritative final checkout before regenerating each queue lane. Preserve the model_support/series and generated/series manifests explicitly rather than flattening their patches into the top-level queue.

repo_root="$(pwd)"
llama_checkout="${LLAMA_CHECKOUT:-$repo_root/.deps/llama.cpp}"
patch_backup="$(mktemp -d /tmp/mesh-llm-patches.XXXXXX)"
patch_root="$repo_root/skippy/llama_cpp/patches"
mkdir -p "$patch_backup/core"
mv "$patch_root"/*.patch "$patch_backup/core/"
mv "$patch_root/model_support" "$patch_backup/model_support"
mkdir -p "$patch_root/model_support"
git -C "$llama_checkout" format-patch \
  --start-number 1 \
  --output-directory "$patch_root" \
  "$(cat "$repo_root/skippy/llama_cpp/upstream.txt")..<core-head>"
git -C "$llama_checkout" format-patch \
  --start-number 1 \
  --output-directory "$patch_root/model_support" \
  "<core-head>..<model-support-head>"

Regenerate model_support/series from the sorted patch filenames. Leave the generated lane in place unless its generator inputs changed; if they did, regenerate it with the deterministic family-patch workflow rather than moving or formatting those commits by hand.

Keep the temporary backup until clean patch application and the required native build pass. Ordinary focused changes may append a patch without rebuilding unrelated functional boundaries.

Validation

Validate that patches apply in a clean checkout:

tmp_llama="$(mktemp -d /tmp/mesh-llm-llama.XXXXXX)"
trap 'rm -rf -- "$tmp_llama"' EXIT
LLAMA_WORKDIR="$tmp_llama" scripts/prepare-llama.sh pinned

For normal mesh-llm validation, use the repository build workflow:

just build

For Rust-only fallout from build-system or runtime call-site changes:

cargo fmt --all --check
cargo check -p mesh-llm

Run Cargo commands serially. This repo frequently hits Cargo lock conflicts when multiple Cargo commands run at once.

Model-load spot check (required for backend or model-switch changes)

A clean patch replay plus a green build does not prove the runtime works. Metal shaders in ggml-metal.metal are JIT-compiled on-device at first model open, so a broken shader builds green everywhere and only fails at load time. Machine-reconciled patches can also silently drop arch cases from switches in src/llama-model.cpp (for example llama_model_rope_type), which only fail when a model of that arch creates a context.

After any queue change that touches backend sources (.metal, CUDA, Vulkan), ggml.c/ggml-*.h kernel argument structs, or src/llama-model.cpp switch statements, load a small real model on your local backend and confirm one completion returns:

./target/debug/mesh-llm serve --model "Qwen/Qwen2.5-3B-Instruct-GGUF@main:q4_k_m" --log-format json
# wait for the model to appear, then:
curl -s http://127.0.0.1:9337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen2.5-3B-Instruct-GGUF:q4_k_m","messages":[{"role":"user","content":"Say OK"}],"max_tokens":5}'

On a Mac this exercises the Metal shader JIT path directly. Watch the JSON log for metal_library_init: error and model-open failure. If the change adds or touches support for a specific model family, spot-check a model of that family as well.

When regenerating the queue against a new upstream pin, also diff the arch case lists of reconciled switches against upstream and account for every deletion:

# example: rope-type switch parity
git -C .deps/llama.cpp show <upstream-pin>:src/llama-model.cpp \
  | awk '/llama_rope_type llama_model_rope_type/,/^}$/' \
  | grep -oE "LLM_ARCH_[A-Z0-9_]+" | sort > /tmp/upstream-cases.txt
awk '/llama_rope_type llama_model_rope_type/,/^}$/' .deps/llama.cpp/src/llama-model.cpp \
  | grep -oE "LLM_ARCH_[A-Z0-9_]+" | sort > /tmp/patched-cases.txt
comm -23 /tmp/upstream-cases.txt /tmp/patched-cases.txt  # must be empty or explained

Updating The Upstream Pin

Test the queue against current upstream without moving the pin:

scripts/prepare-llama.sh latest
just build
cargo test -p mesh-llm --lib

If the queue applies and validation passes, update the upstream pin:

cp skippy/llama_cpp/upstream.txt /tmp/old-llama-upstream.txt
git -C .deps/llama.cpp rev-parse "$(cat .deps/llama.cpp/.git/mesh-llm-upstream-sha)" > skippy/llama_cpp/upstream.txt

Commit the pin update with any patch refreshes.

Gotchas

• scripts/prepare-llama.sh configures local git identity for git am; keep that responsibility there for fresh CI checkouts.
• Patch files are mail-format artifacts and may intentionally contain whitespace that git diff --check reports. Do not hand-normalize patches in a way that changes or breaks git am.
• Build outputs live under .deps/llama.cpp/build; the root llama.cpp symlink is compatibility-only.
• Important backend flags include GGML_RPC=ON, BUILD_SHARED_LIBS=OFF, and LLAMA_OPENSSL=OFF; preserve CPU, Metal, CUDA, Vulkan, and ROCm behavior when touching build scripts.
• See mesh-llm/docs/LLAMA_CPP_FORK.md for the full patch-queue maintenance notes and mesh-llm/docs/LLAMA_STAGE_INTEGRATION_PLAN.md for deferred llama-stage integration.