This guide focuses on NxD inference models, decoder graphs, attention, and porting practices. For project-wide workflows see AGENTS.md. Architecture documents with class hierarchies, graph variants, forward dispatch, and generation loop diagrams: - Text generation (CausalLM): ARCHITECTURE_CAUSAL_LM.md - Image-text-to-text (VLM): ARCHITECTURE_IMAGE_TEXT_TO_TEXT.md Implementation details live in: - optimum/neuron/models/inference/backend/modules/decoder/modeling_decoder.py - optimum/neuron/models/inference/backend/modules/decoder/decoder_builders.py - optimum/neuron/models/inference/backend/modules/decoder/decoder_wrappers.py See reference implementation: - optimum/neuron/models/inference/llama/modeling_llama.py Use NxDI for neuron-specific graph changes and HF Transformers for base architecture. The Optimum Neuron implementation prioritizes stability, maintainability, and HF ecosystem compatibility over cutting-edge…
2709mo agoReviewMountCreate a free account to mount
This directory contains the Neuron-optimized Gemma3 inference implementation. It follows HF Transformers architecture with Neuron-specific changes derived from NxDI. Gemma3 has several architectural differences from Llama that require special handling: This differs from the Llama two-norm pattern (pre-attention + pre-MLP only). `Gemma3NxDModelForCausalLM.convert_hf_to_neuron_s
2709mo agoReviewMountCreate a free account to mount
This directory contains the Neuron-optimized Granite inference implementation. It follows HF Transformers structure with only Neuron-specific graph changes from NxDI. Reference: NxDI Granite implementation Granite shares Llama's architecture and inherits similar removals: Why removed: Requires unstable NKI private APIs. Optimum Neuron uses compiler auto-optimization. Why removed: Production serving features delegated to vLLM integration. Why removed: Optimum uses HF Transformers config base extended via neuron_config.json. Optimum Neuron prioritizes stability and HF compatibility over experimental optimizations.
2709mo agoReviewMountCreate a free account to mount
This directory contains the Neuron-optimized Llama4 inference implementation, aligned to HF Transformers with Neuron-specific changes from NxDI. Reference: NxDI Llama4 implementation Llama4 follows Llama architecture with shared removals: Why removed: Requires neuronxcc.nki private APIs not stabilized for external use. Optimum relies on compiler auto-optimization. Why removed: Advanced serving features for production deployments. vLLM integration handles production needs. Why removed: Optimum uses HF config base with neuron_config.json extensions. Optimum Neuron prioritizes stability, maintainability, and HF ecosystem compatibility.
2709mo agoReviewMountCreate a free account to mount
This directory contains the Neuron-optimized Llama inference implementation. It is based on HF Transformers and trimmed from NxDI to include only Neuron-specific graph changes. Reference: NxDI Llama modeling_llama.py The Optimum Neuron port removes NxDI-specific components that are either experimental, infrastructure-specific, or replaced by Optimum's own systems: Why removed: These kernels require neuronxcc.nki private/pre-prod APIs not stabilized for external use. Optimum Neuron relies on stable NeuronX compiler auto-optimization rather than hand-tuned NKI kernels. Users needing these optimizations should use NxDI directly.
2709mo agoReviewMountCreate a free account to mount
This directory contains the Neuron-optimized Qwen2 inference implementation. It follows HF Transformers architecture with only Neuron-specific changes from NxDI. Reference: NxDI Qwen2 implementation Qwen2 follows Llama-like architecture with shared removals: Why removed: Requires neuronxcc.nki private APIs. Optimum relies on compiler auto-optimization. Why removed: Production serving delegated to vLLM integration. Why removed: Optimum uses HF config base with neuron_config.json. Optimum Neuron prioritizes stability and HF compatibility for Qwen2 models.
2709mo agoReviewMountCreate a free account to mount
This directory contains the Neuron-optimized Qwen3 MoE inference implementation. It follows HF Transformers architecture with Neuron-specific changes from NxDI. Reference: NxDI Qwen3 MoE implementation Qwen3 MoE is a Mixture-of-Experts model with MoE-specific removals: Why removed: Requires unstable NKI APIs. Compiler handles MoE optimization. Why removed: Complex MoE serving optimizations for production. Focus on foundational support. Why removed: Optimum uses standardized config via neuron_config.json. Optimum Neuron provides stable foundational MoE support with vLLM for advanced serving.
2709mo agoReviewMountCreate a free account to mount
This directory contains the Neuron-optimized SmolVLM/Idefics3 inference implementation. It extends the NxD decoder stack with a separately compiled vision encoder and on-device image-token feature injection. For shared NxD decoder guidance, read optimum/neuron/models/inference/AGENTS.md. For SmolVLM, there are exactly two compiled artifacts: model_vision.pt and model_text.pt. Both are managed by SmolVLMNxDModelForImageTextToText in modeling_smolvlm.py. The custom embeddings intentionally keep HF key names (patch_embedding, position_embedding) so state dict loading maps directly. Image features are inserted at image_token_id positions and passed into VLM wrappers: - NxDDecoderWrapperForImageTextToText…
2709mo agoReviewMountCreate a free account to mount
This guide covers the Optimum Neuron vLLM backend. For project-wide workflows see AGENTS.md. Key files: - optimum/neuron/vllm/runner.py - optimum/neuron/vllm/model_loader.py Key files: - optimum/neuron/vllm/server_manager.py — spawns and manages vLLM server subprocesses with Neuron core pinning. - optimum/neuron/vllm/reverse_proxy.py — round-robin proxy with aggregated /health. Do NOT use start_new_session=True on vLLM server subprocesses. vLLM already cleans up its own EngineCore workers on SIGTERM. Isolating servers in separate process groups makes them unreachable by the parent's signal handlers and test fixture teardown, causing Neuron…
2709mo agoReviewMountCreate a free account to mount
Papers With Code MCP tools for searching and reading AI/ML papers, discovering recent and trending research, finding related work and paper lineage, browsing tasks, methods, conferences, organizations, and frameworks, and inspecting benchmark leaderboards with model-size and metric filters through the public Papers With Code catalog. Use whenever the user asks to find papers, survey literature, compare research, inspect an arXiv paper, explore AI/ML taxonomy or conferences, discover benchmarks or state-of-the-art models, or mentions Papers With Code, the PwC MCP server, or paperswithcode.co/mcp.
21419d agoReviewMountCreate a free account to mount
Papers With Code CLI (`pwc`) for searching and reading AI/ML papers, discovering recent and trending research, finding related work and paper lineage, browsing tasks, methods, conferences, organizations, frameworks, and benchmark leaderboards, and submitting authenticated paper edits through the public Papers With Code catalog. Use whenever the user asks to find papers, survey literature, compare research, inspect an arXiv paper, explore AI/ML taxonomy or conferences, discover benchmarks or state-of-the-art models, or mentions Papers With Code, `pwc`, or `pwc-cli`.
21419d agoReviewMountCreate a free account to mount
LeLab is a FastAPI + React web interface wrapping the LeRobot framework for the SO-101 leader/follower arm. It exposes teleoperation, dataset recording, calibration, replay, and training as HTTP/WebSocket endpoints, replacing LeRobot's CLI + keyboard-driven flows. The frontend (React + Vite) lives in frontend/. The bundle in frontend/dist/ is not committed: setup.py builds it while building the wheel, with Node from the nodejs-wheel-binaries build requirement, and ships it as package data (frontend.__init__.py makes setuptools treat it as a package); lelab/server.py mounts…
2079d agoReviewMountCreate a free account to mount
All imports in a kernel (files inside torch-ext/<kernel_name>) of other modules within the same kernel must be relative. For example: The _ops module (_ops.py) is generated by the build system…
15720d agoReviewMountCreate a free account to mount
Always use bun (bun install, bun dev, bun run build, bun test). Never use npm or yarn. After making any code changes, always run these commands in order and fix any errors before finishing: Or run them all at once (format first, then the full validate suite): bun run validate runs: type-check → lint → format:check → test Three versions are supported. Version is detected from meta/info.json → codebase_version. src/app/[org]/[dataset]/[episode]/fetch-data.ts → getEpisodeData() dispatches to: Locating an episode (its video URLs…
13812d agoReviewMountCreate a free account to mount
Pre-launch checklist for shipped code paths. Run before announcing the leaderboard publicly. Style basics (Python 3.12+, type hints, sync only, no fallbacks, no silent exception swallowing) live in style.mdc.
1266mo agoReviewMountCreate a free account to mount
Plain text files in a repository that tell a coding agent how the project works: commands to run, conventions to follow and things to avoid. CLAUDE.md, AGENTS.md, cursor rules and skills are the common kinds.
CLAUDE.md or AGENTS.md?
CLAUDE.md is read by Claude Code. AGENTS.md is an open format that Codex, Cursor and other agents read. Many projects keep one and point the other at it.
What is a skill?
A folder with a SKILL.md that describes one capability, such as filling PDFs or reviewing code. The agent loads it only when the task calls for it.
Can I search my own team's files too?
Your agents already can, over MCP, limited to the files you're allowed to read. Searching them from this page is coming.
Hugging FaceAGENTS.md