agentleFS
Sign inStart free
vllm-projectvllm-project/vllm-gaudi/AGENTS.md

vllm-gaudi

AGENTS.md68 starsChanged 9 months agoApache-2.0View on GitHub

This is vllm-gaudi — the vLLM Hardware Plugin for Intel® Gaudi® AI accelerators. It is a plugin package (not a fork) that integrates Intel Gaudi HPUs with the upstream vLLM inference engine via the vLLM plugin architecture. See the README — Getting Started for installation instructions (vLLM checkout, plugin install, Docker). The plugin follows vLLM's pluggable architecture. For Intel® Gaudi® hardware details (MME, TPC, profiling, etc.), see the Intel® Gaudi® Documentation. Key subsystems:

  • Installs packages
  • Where files are headingNew AGENTS.md files have more rules about keeping secrets out of the code, and fewer about code style.

    Of the AGENTS.md files we read, the 31,774 first seen in the last six months, against the older ones: the kind of rule that gained most share, and the kind that lost most. About files like this in general, not this one, and an estimate, since a kind is a pattern in the words.

  • In this repository2 other files it gives its agents.

The file

Show its rules and text

We split this AGENTS.md into the rules it gives an agent

16 rules. About 0 are common, 7 are distinctive, and 9 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈7

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 9

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Imports: Use absolute imports. Avoid star imports where possible (F405/F403 are currently suppressed)

    line 42
  2. Distinctive:

    Never commit directly to main: Always work on a feature branch.

    line 47
  3. Distinctive:

    If you have write access: create a branch in the repo (e.g., git checkout -b my-feature)

    line 48
  4. Distinctive:

    If you don't have write access: fork the repo and create a branch in your fork

    line 49
  5. Very specific:

    Create a Pull Request from your branch targeting the appropriate base branch.

    line 50
  6. Very specific:

    All PRs are squash-merged with PR number appended: Title (#NNN)

    line 51
  7. Very specific:

    Commit messages must include Signed-off-by: (enforced by pre-commit hook via DCO)

    line 52
  8. Very specific:

    No internal references: Never include internal ticket numbers (e.g., Jira IDs), links to internal sites (Jira, Confluence, etc.), or …

    line 63
  9. Distinctive:

    No raw performance numbers: Do not include absolute performance metrics or profiling data/screenshots.

    line 64
  10. Distinctive:

    PR titles must be descriptive: Use a clear summary of the change, not a ticket reference

    line 65
  11. Very specific:

    When adding a new operator in vllm_gaudi/ops/, add corresponding unit tests in tests/unit_tests/ops/

    line 72
  12. Very specific:

    When adding new model support, add model cards in tests/models/

    line 73
  13. Very specific:

    Document new env vars in docs/configuration/env_variables.md

    line 78
  14. Very specific:

    Create the op file in vllm_gaudi/ops/hpu_<name>.py

    line 83
  15. Very specific:

    Add unit tests in tests/unit_tests/ops/test_hpu_<name>.py

    line 85
  16. Distinctive:

    Add documentation if the feature is user-facing

    line 86

≈ “Common” is an estimate: at least 1 in 5 of the 40,246 AGENTS.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
AGENTS.md

Project Guidelines for vllm-gaudi

Project Overview

This is vllm-gaudi — the vLLM Hardware Plugin for Intel® Gaudi® AI accelerators. It is a plugin package (not a fork) that integrates Intel Gaudi HPUs with the upstream vLLM inference engine via the vLLM plugin architecture.

  • Package: vllm_gaudi (installed as vllm-gaudi)
  • License: Apache-2.0
  • Python: Supported versions are defined in pyproject.toml (requires-python)
  • Entry point: vllm_gaudi/__init__.py registers the HPU platform, ops, models, and utilities into vLLM's plugin system.

Getting Started

See the README — Getting Started for installation instructions (vLLM checkout, plugin install, Docker).

Architecture

The plugin follows vLLM's pluggable architecture. For Intel® Gaudi® hardware details (MME, TPC, profiling, etc.), see the Intel® Gaudi® Documentation.

Key subsystems:

DirectoryPurpose
vllm_gaudi/ops/HPU-specific operator implementations (attention, FP8, MoE, LoRA, etc.)
vllm_gaudi/models/Model-specific overrides and registrations for HPU
vllm_gaudi/attention/HPU attention backends and paged attention ops
vllm_gaudi/extension/Bucketing, profiling, quantization, runtime config, unified batching
vllm_gaudi/v1/V1 engine worker and model runner for HPU
vllm_gaudi/distributed/HPU communicator and KV transfer connectors (NIXL)
vllm_gaudi/lora/LoRA layer support for HPU-specific layers
docs/MkDocs-based documentation
tests/Unit tests (unit_tests/), full model tests (full_tests/), upstream compat tests (upstream_tests/)
calibration/FP8 calibration pipeline scripts
.cd/Docker build files and CI/CD configs

Code Style

  • Formatter: yapf (column limit 120) + ruff (line length 120)
  • Linter: ruff with pycodestyle (E), pyflakes (F), pyupgrade (UP), flake8-bugbear (B), flake8-simplify (SIM). Note: vllm_gaudi/extension/ is currently excluded from ruff
  • Pre-commit hooks: Install via pip install -r requirements-lint.txt && pre-commit install
  • SPDX headers: New Python files should include # SPDX-License-Identifier: Apache-2.0 as the first comment line
  • Imports: Use absolute imports. Avoid star imports where possible (F405/F403 are currently suppressed)
  • Line length: 120 characters max

Commit and PR Conventions

  • Never commit directly to main. Always work on a feature branch.
    • If you have write access: create a branch in the repo (e.g., git checkout -b my-feature)
    • If you don't have write access: fork the repo and create a branch in your fork
  • Create a Pull Request from your branch targeting the appropriate base branch. Default target is main unless a specific release branch is specified (e.g., releases/v0.16.0)
  • All PRs are squash-merged with PR number appended: Title (#NNN)
  • Commit messages must include Signed-off-by: (enforced by pre-commit hook via DCO)
  • Common title patterns:
    • Features: Feature name (#NNN) — e.g., RowParallel NIC chunking (#896)
    • Fixes: Fix description (#NNN) — e.g., Fix mamba cumsum padded calculations (#1009)
    • Upstream compatibility: [FIX_FOR_VLLM_CUSTOM=<hash>] Description (#NNN)
    • CI changes: [CI] Description (#NNN)
    • Reverts: Revert "Original title" (#NNN)
    • Topic tags: [warmup][multimodal] Description (#NNN)

Confidentiality and content rules

  • No internal references: Never include internal ticket numbers (e.g., Jira IDs), links to internal sites (Jira, Confluence, etc.), or internal file paths (e.g., /mnt/weka/...) in PR titles, descriptions, commit messages, or code comments
  • No raw performance numbers: Do not include absolute performance metrics or profiling data/screenshots. Relative improvements are acceptable (e.g., "~20% throughput gain for long contexts") but raw numbers, benchmarks, or profiler outputs must not appear in PRs
  • PR titles must be descriptive: Use a clear summary of the change, not a ticket reference

Testing

  • Unit tests: tests/unit_tests/ — test individual ops, workers, samplers
  • Full tests: tests/full_tests/ — end-to-end model generation tests using model cards
  • Run tests: Tests require Intel Gaudi hardware (HPU). Use pytest to run
  • When adding a new operator in vllm_gaudi/ops/, add corresponding unit tests in tests/unit_tests/ops/
  • When adding new model support, add model cards in tests/models/

Environment Variables

  • HPU-specific env vars are defined in vllm_gaudi/envs.py
  • Document new env vars in docs/configuration/env_variables.md
  • Feature flags belong in vllm_gaudi/extension/features.py

Adding New Ops

  1. Create the op file in vllm_gaudi/ops/hpu_<name>.py
  2. Register it in vllm_gaudi/__init__.py under register_ops()
  3. Add unit tests in tests/unit_tests/ops/test_hpu_<name>.py
  4. Add documentation if the feature is user-facing

Documentation

  • Uses MkDocs with navigation defined in docs/.nav.yml
  • Feature docs go in docs/features/
  • Configuration docs go in docs/configuration/
  • Developer guides go in docs/dev_guide/

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free