AI Tools & Model Post-Training Playbook

Covers two surfaces:

1. Local offline-first ML experiments in extensions/ai-tools/ (mechanistic interpretability, weight inspection, deterministic verifiers).
2. Reasoning & domain model post-training playbook (inference-time scaling, SFT distillation, and RLVR with GRPO/DAPO).

1. Local extension (extensions/ai-tools/)

• Bundle source: extensions/ai-tools/
• Installed: %APPDATA%\devin\extensions\ai-tools\ (Windows) or ~/.config/devin/extensions/ai-tools/ (POSIX)
• Full extension docs: USAGE.md inside that directory.

Conventions (all local tools)

• Self-contained <tool>.py with __main__ guard and JSON on stdout.
• --self-test runs fully offline asserts; exit 1 on failure.
• No network: real inputs come from explicit file paths or the local HF cache only.
• Deps pinned in requirements.txt, installed on demand.

Current tools

• abliterator.py: refusal-direction orthogonalization PoC (Arditi et al. 2024): difference-in-means direction, activation ablation, residual-weight orthogonalization. Dummy tensors by default; --weights F.safetensors or --model <cached> for real weights.
• prompt_compiler.py: deterministic prompt compilation against knowledge_bases/.

2. Reasoning & Domain Model Post-Training Playbook

Use this playbook when building, adapting, or distilling a custom reasoning or domain-specialist model (source architecture: Raschka Build a Reasoning Model From Scratch; validated against primary literature below).

Method selector (evidence-based decision tree)

Phase route & gates

Iron rules for model post-training

1. Verifier before trainer: never write a GRPO or distillation loop before the deterministic verifier (G1) is unit-tested on edge cases (formatting variations, symbolic equivalence, timeouts).
2. Never use intrinsic self-refinement without an external verifier: asking an LLM to critique and revise its own reasoning without an external verifier or dedicated RL training degrades or stalls accuracy (arXiv:2310.01798). Pair refinement loops with a deterministic check or verifier score.
3. Distill <7B, RLVR >=7B: do not spend GPU hours running pure RLVR from scratch on sub-3B models expecting emergent reasoning; distill from a strong teacher first (arXiv:2501.12948, arXiv:2501.19393).
4. Condition format rewards on correctness: if granting a reward bonus for <think>...</think> or \boxed{} tags, gate or scale it by answer correctness; otherwise the policy hacks the format reward with empty or repetitive reasoning.
5. Drop or decouple KL penalty for long-CoT RLVR (DAPO rules): when training long chain-of-thought from a non-reasoning base, the policy distribution must diverge from the reference model; omit the KL penalty (beta = 0) or keep it minimal, use token-level policy gradient loss, and skip prompts where all $G$ group samples have identical reward ($\sigma_R = 0$ yields zero gradient; arXiv:2503.14476).
6. Track 4 collapse signals every N steps: (a) policy entropy (sudden drop = mode collapse), (b) mean/max response length (length inflation / reward hacking), (c) zero-variance group ratio ($\sigma_R = 0$), and (d) held-out pass@1 vs training reward.

Primary references

• arXiv:2203.11171: Wang et al. 2022, Self-Consistency Improves Chain of Thought Reasoning in Language Models (ICLR 2023).
• arXiv:2310.01798: Huang et al. 2023, Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024).
• arXiv:2402.03300: Shao et al. 2024, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO).
• arXiv:2408.03314: Snell et al. 2024, Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (ICLR 2025).
• arXiv:2411.15124: Lambert et al. 2024, Tulu 3: Pushing Frontiers in Open Language Model Post-Training (RLVR & decontamination).
• arXiv:2501.12948: DeepSeek-AI 2025, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.
• arXiv:2501.19393: Muennighoff et al. 2025, s1: Simple test-time scaling (s1K curation & budget forcing).
• arXiv:2503.14476: Yu et al. 2025, DAPO: An Open-Source LLM Reinforcement Learning System at Scale.