Test Model — Validate and Benchmark LLM Models on xinfer

Phase 0: Gather Model List

Collect the models to test. The user provides one or both of:

Detecting models in a local folder

If the user provides a parent directory (not a single model), scan it to find testable models:

# List subdirectories that look like model folders
for d in /data/*/; do
  if [ -f "$d/config.json" ] || ls "$d"/*.gguf 2>/dev/null | head -1 >/dev/null; then
    echo "$d"
  fi
done

For each candidate directory, determine the model type by reading config.json:

import json, os, sys, glob

def detect_model(path):
    """Detect model type and quantization from a local directory."""
    config_path = os.path.join(path, "config.json")
    gguf_files = glob.glob(os.path.join(path, "*.gguf"))

    info = {"path": path, "name": os.path.basename(path.rstrip("/"))}

    if gguf_files:
        info["format"] = "gguf"
        info["gguf_file"] = os.path.basename(gguf_files[0])
        return info

    if not os.path.exists(config_path):
        return None

    cfg = json.load(open(config_path))
    arch = (cfg.get("architectures") or ["Unknown"])[0]

    supported = [
        "LlamaForCausalLM", "MistralForCausalLM", "Ministral3ForConditionalGeneration",
        "Qwen2ForCausalLM", "Qwen3ForCausalLM", "Qwen3MoeForCausalLM",
        "Qwen3_5ForCausalLM", "Qwen3_5MoeForCausalLM",
        "Qwen3_5ForConditionalGeneration", "Qwen3_5MoeForConditionalGeneration",
        "Qwen3NextForCausalLM",
        "Qwen3VLForConditionalGeneration",
        "Gemma3ForConditionalGeneration", "Gemma3ForCausalLM",
        "Gemma4ForCausalLM", "Gemma4ForConditionalGeneration",
        "Phi3ForCausalLM", "Phi4ForCausalLM",
        "Glm4ForCausalLM", "Glm4MoeForCausalLM",
    ]
    if arch not in supported:
        info["skip"] = f"Unsupported architecture: {arch}"
        return info

    info["arch"] = arch
    info["format"] = "safetensors"

    qcfg = cfg.get("quantization_config", {})
    qm = qcfg.get("quant_method", "")
    if qm in ("fp8", "modelopt", "compressed-tensors"):
        algo = qcfg.get("quant_algo", "")
        fmt = qcfg.get("format", "")
        if algo and ("nvfp4" in algo.lower() or "fp4" in algo.lower()):
            info["quant"] = "nvfp4"
        elif "nvfp4" in fmt.lower():
            info["quant"] = "nvfp4"
        elif "mxfp4" in fmt.lower():
            info["quant"] = "mxfp4"
        elif qm == "fp8":
            info["quant"] = "fp8"
        else:
            info["quant"] = qm
    elif qm in ("gptq", "awq"):
        info["quant"] = qm
    elif qm == "mxfp4":
        info["quant"] = "mxfp4"
    else:
        info["quant"] = "bf16"

    return info

Present the detected models to the user as a table and confirm before proceeding.

Phase 1: Estimate GPU Requirements and Detect Hardware

Detect available GPUs

nvidia-smi --query-gpu=index,name,memory.total,memory.free --format=csv,noheader,nounits

Parse the output to get gpu_id, name, total_mb, free_mb for each GPU.

Estimate model memory

Use these rough heuristics for memory estimation (single-GPU, including KV cache overhead):

Extract parameter count from the model name when possible (e.g. Qwen3.5-27B → 27B). For MoE models with A3B in the name, the weight memory uses total params but fits better than dense.

GPU assignment rules

1. If a model fits in one GPU's free memory, use --d <gpu_id> with the GPU that has the most free memory.
2. If a model needs 2 GPUs, use --d <id1>,<id2> with the two GPUs with the most free memory.
3. If a model exceeds all available GPU memory, report it as skipped and move to the next.
4. For models explicitly specified as multi-GPU by the user, respect that.

Phase 2: Build the Project

Build using build.sh:

cd <project_root>
./build.sh --install --features cuda,nccl,flashinfer,cutlass

Verify the build succeeds (exit code 0). The Error: Must provide model_id or weight_path message after build is expected — it means the binary compiled correctly.

If the build fails, check and fix compilation errors before proceeding.

Important, if you build on CUDA with cargo build, make sure always build xinfer binaries.

Phase 3: Create the Test Script

Create test_model.py in the project root with the following capabilities:

• Accept --port to specify the API server port
• Accept --wait for server readiness timeout
• Test both thinking=false and thinking=true modes
• Send a prompt with at least 1024 input tokens and request at least 2048 output tokens
• Measure end-to-end throughput (completion_tokens / total_time)
• Check output quality: detect excessive 3-gram repetition, too-short responses
• Report prompt tokens, completion tokens, time, throughput, and quality verdict
• Print a summary table at the end

The prompt should be a substantive multi-topic question (algorithms, data structures, etc.) padded with context tokens to reach the 1k+ input requirement. Use max_tokens: 2048 and temperature: 0.7. Set request timeout to 300s.

For thinking mode, add "extra_body": {"thinking": true} to the payload.

Quality checks:

• Response must be at least 100 characters
• 3-gram repetition: flag if any trigram appears more than max(10, 5% of total trigrams) times

Phase 4: Test Each Model

For each model, execute this sequence:

Step 1: Kill previous instances

pkill -9 -f 'xinfer' 2>/dev/null
sleep 3

Always wait 3 seconds after killing to ensure GPU memory is released.

Step 2: Start the server

Build the server command based on model type:

Run the server in the background with RUST_BACKTRACE=1 for debugging.

Step 3: Wait for server readiness

Poll GET /v1/models every 2-3 seconds until it returns HTTP 200, with a timeout of:

• Small models (< 10B): 120s
• Medium models (10-40B): 300s
• Large models (> 40B) or HF downloads: 600s

Step 4: Run the test script

python3 test_model.py --port 7000

Step 5: Handle failures

If the server fails to start or the test script returns errors:

1. Check server logs for panics or errors
2. Common issues and fixes:

3. Debug with unwrap: If the model crashes during inference, temporarily change guard.step() to guard.step().unwrap() in src/core/engine.rs to get a full stack trace. Revert after debugging.
4. If a model cannot be fixed, record the failure reason and continue to the next model.

Phase 5: Summarize Results

After all models are tested, produce a summary table:

## Test Results

| # | Model | Format | GPUs | thinking=false | thinking=true | Quality |
|---|-------|--------|------|----------------|---------------|---------|
| 1 | Qwen3.5-27B-FP8 | FP8 | 1 | 1342 in / 2048 out, 42.2 tok/s | 1342 in / 2048 out, 42.2 tok/s | OK |
| 2 | ... | ... | ... | ... | ... | ... |

### Notes
- Model X: SKIPPED — reason
- Model Y: FAILED — error description

Include for each model:

• Model name and quantization format
• Number of GPUs used
• Input/output token counts and throughput for both thinking modes
• Quality verdict (OK / ISSUES / FAILED / SKIPPED)

Quick Reference

Key files

Build features

Server flags