kserve-operator
Skill4 starsChanged 5 months agoMITView on GitHub
Create and manage KServe CRDs — InferenceService, ServingRuntime, InferenceGraph, LLMInferenceService, LocalModel, ClusterServingRuntime, and 22 CRDs under serving.kserve.io.
In this repository74 other files it gives its agents.
- AGENTS.mdAGENTS.md
- atlasSkill
- cert-managerSkill
- cilium-gatewaySkill
- cilium-networkSkill
- cnpgSkill
- dragonflySkill
- external-dnsSkill
- external-secretsSkill
- flaggerSkill
- fluxSkill
- gitea-apiSkill
- gitea-registrySkill
- gitea-runnerSkill
- giteaSkill
- gitea-teaSkill
- gitea-webhooksSkill
- harbor-apiSkill
- harbor-helmSkill
- harborSkill
- harbor-terraformSkill
- helm-chartSkill
- helm-opsSkill
- helmSkill
- higress-helmSkill
- higress-operatorSkill
- higressSkill
- kserve-helmSkill
- kserveSkill
- kubeflow-pipelinesSkill
- kubeflowSkill
- kubeflow-trainerSkill
- kubeflow-training-operatorSkill
- mariadb-helmSkill
- mariadb-operatorSkill
- mariadbSkill
- netbird-cliSkill
- netbird-operatorSkill
- netbird-serverSkill
- netbirdSkill
- netbird-terraformSkill
- nextcloud-apiSkill
- nextcloud-helmSkill
- nextcloudSkill
- nvidia-device-pluginSkill
- rook-ceph-operatorSkill
- rook-cephSkill
- rook-ceph-toolboxSkill
- sealed-secretsSkill
- stakater-reloaderSkill
- tailscaleSkill
- tailscale-talosSkill
- talosconfigSkill
- talosctlSkill
- talosSkill
- talos-terraformSkill
- tekton-operatorSkill
- tekton-pacSkill
- tekton-pipelinesSkill
- tektonSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 591
Default storage container template applied across namespaces.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
KServe — Operator CRDs & Configuration
Repository: github.com/kserve/kserve
Latest release: v0.18.0 (Apr 29, 2026) — v0.19.0-rc0 available
CNCF: Incubating (Jan 2025)
License: Apache 2.0
Stars: 10k+
Architecture
RawPredictOp → Preprocess → Predict → [Explain] → Postprocess → ExplainOp
(Transformer) (Predictor) (Transformer)
InferenceService (CRD) ──► KServe Controller ──► Deployment + Service + HPA + Istio/Knative
│
Storage: S3/GCS/Azure/HF/PVC/HTTP/HDFS
│
Modelcar (OCI) or PVC or storage init containerDeployment Modes
| Mode | Networking | Use Case |
| Standard | Istio VirtualService | Enterprise, stable, full traffic control |
| Knative | Knative + Istio (Kourier optional) | Auto-scaling to zero, serverless |
| Raw | Direct Service + Ingress/Gateway | Simple, lightweight, no service mesh |
Annotations Control
| Annotation | Effect |
serving.kserve.io/deploymentMode | Serverless (Knative, default), RawDeployment, ModelMesh |
serving.tempaltes.kserve.io/default | Template rendering default |
sidecar.istio.io/inject | true to inject Istio sidecar |
CRDs (22 total — primary 9 covered here)
All under serving.kserve.io:
| CRD | API Version | Scope | Short Name | Description |
| InferenceService | v1beta1 / v1alpha1 | Namespaced | isvc | Core serving unit |
| ServingRuntime | v1alpha1 | Namespaced | sr | Runtime template (ModelMesh) |
| ClusterServingRuntime | v1alpha1 | Cluster | csr | Cluster-scoped runtime template |
| InferenceGraph | v1alpha1 | Namespaced | ig | Router graph (ensemble/switch/splitter) |
| TrainedModel | v1alpha1 | Namespaced | tm | Model-scoped config for ServingRuntime |
| LLMInferenceService | v1alpha1 | Namespaced | — | LLM-specific (disaggregated prefill/decode) |
| LocalModelCache | v1alpha1 | Namespaced | lm | Node-level model cache for LocalModelNode |
| LocalModelNode | v1alpha1 | Cluster | lmn | Node DaemonSet for model loading |
| ClusterStorageContainer | v1alpha1 | Cluster | csc | Default storage container template |
InferenceService — serving.kserve.io/v1beta1
The primary CRD for model serving.
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: sklearn-iris
spec:
predictor:
# Built-in predictor
sklearn:
storageUri: s3://models/iris/model.joblib
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
readinessProbe:
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
livenessProbe:
periodSeconds: 5
failureThreshold: 3
timeoutSeconds: 1
# ServiceOrchestration (multi-model composition)
# serviceOrchestrationSpec:
# pipelining: router
# Node selection
# nodeSelector:
# node-type: gpu
# Service account for storage access
serviceAccountName: kserve-sa
# Min/max replicas
minReplicas: 1
maxReplicas: 5
# Scale target (Knative mode)
scaleTarget: 1
scaleMetric: concurrency
containerConcurrency: 5
# Canary rollout
canaryTrafficPercent: 10
canary:
sklearn:
storageUri: s3://models/iris-v2/model.joblib
# Transformer (optional, pre/post-processing)
transformer:
container:
image: myrepo/transformer:latest
env:
- name: MODEL_NAME
value: iris
resources:
requests:
cpu: 100m
memory: 128Mi
# Explainer (optional)
explainer:
alibi:
storageUri: s3://models/iris-explainer/
resources:
requests:
cpu: 100m
memory: 256Mi
# Storage via modelcar (OCI)
# storage:
# modelcar:
# enabled: true
# pullAlways: trueSpec Fields
| Field | Type | Required | Description |
predictor | PredictorSpec | ✅ | Model predictor configuration |
predictor.minReplicas | int | ❌ | Min pods |
predictor.maxReplicas | int | ❌ | Max pods |
predictor.scaleTarget | int | ❌ | Scale to zero target |
predictor.scaleMetric | string | ❌ | concurrency, rps, cpu, memory |
predictor.containerConcurrency | int | ❌ | Max concurrent requests (Knative) |
predictor.canaryTrafficPercent | int | ❌ | Canary traffic percentage |
predictor.canary | object | ❌ | Canary predictor spec (same structure) |
predictor.serviceAccountName | string | ❌ | KSA for storage access |
predictor.readinessProbe | Probe | ❌ | Custom readiness probe |
predictor.livenessProbe | Probe | ❌ | Custom liveness probe |
predictor.annotations | map | ❌ | Pod annotations |
predictor.nodeSelector | map | ❌ | Node selector |
predictor.affinity | Affinity | ❌ | Pod affinity |
predictor.tolerations | []Toleration | ❌ | Node tolerations |
predictor.topologySpreadConstraints | []TSC | ❌ | Topology spread |
predictor.sidecarInject | string | ❌ | istio-proxy, envoy, etc. |
predictor.serviceOrchestrationSpec | object | ❌ | Multi-model routing |
predictor.storage | StorageSpec | ❌ | Storage config |
transformer | ComponentSpec | ❌ | Pre/post-processing |
explainer | ComponentSpec | ❌ | Model explainability |
storage.modelcar.enabled | bool | ❌ | Modelcar OCI mode |
storage.modelcar.pullAlways | bool | ❌ | Always pull model image |
Built-in Predictors
| Predictor | Spec Field | Image (default) | Protocol |
| SKLearn | sklearn | kserve/sklearnserver | REST/gRPC (v2) |
| TensorFlow | tensorflow | kserve/tfserving | REST/gRPC (TFServing) |
| PyTorch | pytorch | kserve/torchserve | REST/gRPC (TorchServe) |
| Triton | triton | nvcr.io/nvidia/tritonserver | REST/gRPC (KServe v2) |
| ONNX | onnx | kserve/onnxserver | REST/gRPC (v2) |
| XGBoost | xgboost | kserve/xgbserver | REST/gRPC (v2) |
| LightGBM | lightgbm | kserve/lgbserver | REST/gRPC (v2) |
| PMML | pmml | kserve/pmmlserver | REST (PMML) |
| Paddle | paddle | kserve/paddleserver | REST/gRPC |
| HuggingFace | huggingface | kserve/huggingfaceserver | REST (TGI/transformers) |
# SKLearn
predictor:
sklearn:
storageUri: s3://models/iris/
protocolVersion: v2
# Triton
predictor:
triton:
storageUri: s3://models/triton-repo/
runtimeVersion: 24.12
protocolVersion: v2 # KServe v2 protocol or grpc-v2
# HuggingFace
predictor:
huggingface:
storageUri: s3://models/llama/
resources:
limits:
nvidia.com/gpu: 1
env:
- name: HF_HUB_DISABLE_TELEMETRY
value: "true"Custom Predictor (Container)
predictor:
containers:
- name: kserve-container
image: myrepo/custom-model:latest
ports:
- containerPort: 8080
protocol: TCP
env:
- name: MODEL_DIR
value: /models
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
nvidia.com/gpu: 1Multi-Model Composition
predictor:
serviceOrchestrationSpec:
pipelining: router # OR tf (TensorFlow DAG)
sklearn:
storageUri: s3://models/step1/
pytorch:
storageUri: s3://models/step2/Examples
S3 model with canary rollout:
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: mymodel
spec:
predictor:
minReplicas: 2
canaryTrafficPercent: 10
sklearn:
storageUri: s3://models/v1/
canary:
sklearn:
storageUri: s3://models/v2/GPU serving with token auth:
predictor:
minReplicas: 1
maxReplicas: 3
pytorch:
storageUri: s3://models/resnet/
resources:
limits:
nvidia.com/gpu: 1ServingRuntime / ClusterServingRuntime — v1alpha1
Used with ModelMesh deployment mode. Defines runtime templates for model serving.
apiVersion: serving.kserve.io/v1alpha1
kind: ServingRuntime
metadata:
name: sklearn-runtime
spec:
# Supported model formats
supportedModelFormats:
- name: sklearn
version: "1"
autoSelect: true
- name: sklearn
version: "2"
# Runtime container
containers:
- name: kserve-container
image: kserve/sklearnserver:v0.18.0
args:
- --model-dir=/models
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: "1"
memory: 2Gi
# Node-level multi-model pooling (ModelMesh)
multiModel: true
modelSize: Medium # Small | Medium | Large | XLarge
replicas: 2
# Pool-scoped (ModelMesh)
pooled: true
# Serverless scaling behavior
scaleToZero:
enabled: false
gracePeriod: 300
# Protocol
protocolVersions:
- v2 # KServe v2 protocol
# Built-in adapter (REST/gRPC inference adapter)
# builtInAdapter: {} # optionalClusterServingRuntime
Same spec but cluster-scoped. Available across all namespaces.
apiVersion: serving.kserve.io/v1alpha1
kind: ClusterServingRuntime
metadata:
name: triton-runtime
spec:
supportedModelFormats:
- name: triton
autoSelect: true
containers:
- name: kserve-container
image: nvcr.io/nvidia/tritonserver:24.12-py3
args:
- tritonserver
- --model-repository=/models
multiModel: true
protocolVersions:
- grpc-v2
- v2Spec Fields
| Field | Type | Description |
supportedModelFormats[] | []ModelFormat | Model formats this runtime supports |
supportedModelFormats[].name | string | Format name |
supportedModelFormats[].version | string | Format version |
supportedModelFormats[].autoSelect | bool | Auto-select for this format |
containers[] | []Container | Runtime containers |
multiModel | bool | Serve multiple models per pod |
modelSize | string | Small, Medium, Large, XLarge |
replicas | int | Min replicas |
pooled | bool | Pool-scoped runtime |
scaleToZero.enabled | bool | Allow scale to zero |
scaleToZero.gracePeriod | int | Grace period before scale to zero |
protocolVersions[] | []string | Inference protocol versions |
builtInAdapter | object | Built-in adapter config |
disabled | bool | Disable this runtime |
InferenceGraph — v1alpha1
Router graph for multi-model routing.
apiVersion: serving.kserve.io/v1alpha1
kind: InferenceGraph
metadata:
name: model-router
spec:
# Route nodes
nodes:
# Entry point
root:
routerType: Sequence # Sequence | Switch | Ensemble | Splitter
routes:
- serviceUrl: predictor1.default.svc.cluster.local
weight: 80
- serviceUrl: predictor2.default.svc.cluster.local
weight: 20
# Conditional routing
classifier:
routerType: Switch
condition: "input.category == 'a'"
routes:
- serviceUrl: model-a.default.svc.cluster.local
data: "$request"
- serviceUrl: model-b.default.svc.cluster.local
data: "$request"
# Ensemble (merge results)
ensemble:
routerType: Ensemble
routes:
- serviceUrl: model-a.default.svc
- serviceUrl: model-b.default.svc
# Splitter (fan-out)
splitter:
routerType: Splitter
routes:
- serviceUrl: shard-0.default.svc
- serviceUrl: shard-1.default.svc
# Entry point router
entryPoint: root
# Auth
auth:
enabled: true
token: my-auth-token
# Autoscaling
autoscaler:
initialScale: 1
maxScale: 10
scaleDownDelay: 300Router Types
| Type | Behavior |
Sequence | Step-by-step pipeline |
Switch | Conditional routing (first match) |
Ensemble | Fan-out, merge results |
Splitter | Fan-out by weight |
Spec Fields
| Field | Type | Description |
nodes | map[string]Node | Router node definitions |
entryPoint | string | Entry node name |
auth.enabled | bool | Enable auth on graph |
auth.token | string | Auth token |
auth.tokenSecretKeyRef | object | Token from secret |
autoscaler | Autoscaler | Autoscaling config |
Node fields:
| Field | Type | Description |
routerType | string | Sequence, Switch, Ensemble, Splitter |
routes[] | []Route | Route definitions |
routes[].serviceUrl | string | InferenceService URL |
routes[].weight | int | Traffic weight |
routes[].data | string | Request data template |
condition | string | Condition expression (Switch only) |
LLMInferenceService — v1alpha1
LLM-optimized InferenceService for disaggregated prefill/decode, vLLM integration.
apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
name: llama-llm
spec:
# LLM model
modelProvider: vllm
modelName: llama-3-70b
modelStorage: s3://models/llama-3-70b/
# Disaggregated serving (separate prefill/decode)
disaggregated: true
prefill:
replicas: 2
resources:
limits:
nvidia.com/gpu: 4
env:
- name: MAX_BATCH_SIZE
value: "256"
decode:
replicas: 4
resources:
limits:
nvidia.com/gpu: 1
# vLLM specific
vllm:
maxModelLen: 8192
tensorParallelSize: 4
gpuMemoryUtilization: 0.9
# Autoscaling
autoscaler:
initialScale: 1
maxScale: 10
downscaleDelay: 300
# Storage
storage:
modelcar:
enabled: trueSpec Fields
| Field | Type | Description |
modelProvider | string | vllm, triton, tgi, transformers |
modelName | string | Model identifier |
modelStorage | string | Model URI (s3://..., pvc://..., hf://...) |
disaggregated | bool | Separate prefill/decode pods |
prefill.replicas | int | Prefill pod count |
prefill.resources | ResourceRequirements | Prefill GPU resources |
decode.replicas | int | Decode pod count |
decode.resources | ResourceRequirements | Decode GPU resources |
vllm.maxModelLen | int | Max sequence length |
vllm.tensorParallelSize | int | TP degree |
vllm.gpuMemoryUtilization | float | GPU mem utilization (0-1) |
autoscaler | Autoscaler | Scaling config |
storage | StorageSpec | Model storage config |
Storage Configuration
Storage URI Formats
| Scheme | Format | Credentials |
| S3 | s3://bucket/key | SecretRef or IRSA (IAM roles for service accounts) |
| GCS | gs://bucket/key | Workload Identity or SecretRef |
| Azure | az://container/key | SecretRef (account key) |
| HuggingFace | hf://model-id | HF_TOKEN env var |
| PVC | pvc://claim/path | PVC must exist |
| HTTP | http(s)://url | No auth needed |
| HDFS | hdfs://nn:port/path | SecretRef |
Storage Credentials via Secret
apiVersion: v1
kind: Secret
metadata:
name: storage-creds
annotations:
serving.kserve.io/s3-endpoint: s3.us-east-1.amazonaws.com
serving.kserve.io/s3-usehttps: "1"
serving.kserve.io/s3-region: us-east-1
serving.kserve.io/s3-verify-ssl: "1"
type: Opaque
stringData:
AWS_ACCESS_KEY_ID: AKIA...
AWS_SECRET_ACCESS_KEY: ...Reference via service account:
apiVersion: v1
kind: ServiceAccount
metadata:
name: kserve-sa
secrets:
- name: storage-creds
---
spec:
predictor:
serviceAccountName: kserve-saIRSA (IAM Roles for Service Accounts) — AWS
apiVersion: v1
kind: ServiceAccount
metadata:
name: kserve-sa
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/kserve-s3-accessClusterStorageContainer — v1alpha1
Default storage container template applied across namespaces.
apiVersion: serving.kserve.io/v1alpha1
kind: ClusterStorageContainer
metadata:
name: s3-storage
spec:
container:
name: storage-initializer
image: kserve/storage-initializer:v0.18.0
env:
- name: AWS_ENDPOINT_URL
value: s3.us-east-1.amazonaws.com
- name: AWS_REGION
value: us-east-1
supportedUriFormats:
- prefix: s3://TrainedModel — v1alpha1
Model-scoped configuration within a ServingRuntime.
apiVersion: serving.kserve.io/v1alpha1
kind: TrainedModel
metadata:
name: iris-v2
namespace: default
spec:
inferenceService: sklearn-iris
model:
storageUri: s3://models/iris-v2/
framework: sklearn
memory: 256Mi
scaleTarget: 1LocalModelCache / LocalModelNode — v1alpha1
Node-level model caching for faster start times.
apiVersion: serving.kserve.io/v1alpha1
kind: LocalModelCache
metadata:
name: llm-cache
spec:
modelSize: 70Gi
nodeGroup: gpu-nodes
storage:
local:
nodeStateRoot: /mnt/local-models
persistentVolumeClaim:
name: model-cache-pvc
size: 200Gi
---
apiVersion: serving.kserve.io/v1alpha1
kind: LocalModelNode
metadata:
name: gpu-node-1
spec:
cacheConfigRef: llm-cache
nodeName: gpu-worker-1
nodeAgentImage: kserve/local-model-node-agent:v0.18.0Modelcar OCI Mode
Model stored as an OCI image, loaded into the serving pod without init container.
predictor:
storage:
modelcar:
enabled: true
pullAlways: true
containers:
- name: kserve-container
image: myrepo/my-model:latest # model AND runtime combinedCommon Patterns
Minimal inference with S3:
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: quick-start
spec:
predictor:
sklearn:
storageUri: s3://kserve-examples/sklearn/irisMulti-node Triton with GPU:
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: triton-gpu
spec:
predictor:
triton:
storageUri: s3://models/triton-repo/
runtimeVersion: 24.12
protocolVersion: grpc-v2
resources:
limits:
nvidia.com/gpu: 2
minReplicas: 2
maxReplicas: 8Canary rollout with traffic splitting:
spec:
predictor:
sklearn:
storageUri: s3://models/v1/
canaryTrafficPercent: 10
canary:
sklearn:
storageUri: s3://models/v2/InferenceGraph with conditional routing:
apiVersion: serving.kserve.io/v1alpha1
kind: InferenceGraph
metadata:
name: smart-router
spec:
nodes:
root:
routerType: Switch
condition: "input.model == 'large'"
routes:
- serviceUrl: gpu-cluster.default.svc
default:
routerType: Sequence
routes:
- serviceUrl: cpu-cluster.default.svc
entryPoint: rootLLMInferenceService with vLLM:
apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
name: llama-service
spec:
modelProvider: vllm
modelStorage: s3://models/llama-3-8b/
disaggregated: false
vllm:
maxModelLen: 4096
tensorParallelSize: 1
gpuMemoryUtilization: 0.9
autoscaler:
initialScale: 1Common Mistakes
- Storage endpoint annotation mismatch — S3 storage requires
serving.kserve.io/s3-endpointannotation on the secret, not an env var in the predictor. - Missing service account — Without a service account with storage credentials, predictions fail with "model not found" at the
/modelsmount point. - Multi-model vs single-model — ModelMesh (serving.kserve.io/deploymentMode=ModelMesh) uses ServingRuntime for multi-model. Standard mode uses InferenceService per model.
- Protocol version confusion —
v1= KServe v1 protocol (legacy, sklearn/tensorflow).v2= KServe v2 protocol (Triton, MLServer, most built-in).grpc-v2= gRPC KServe v2. - Knative scaling to zero — Default is
minReplicas: 0in Knative mode. SetminReplicas: 1to avoid cold starts. - Canary traffic percent integer — Must be 0–100 integer. Fractional values not supported.
- Storage URI trailing slash — Some storage backends are sensitive to trailing slashes in
storageUri. - InferenceService vs LLMInferenceService — LLMInferenceService is a separate CRD, not a field on InferenceService. Use the v1alpha1 API version.
- Disaggregated prefill/decode — Requires multi-node GPU topology. Prefill uses more GPU memory per token, decode needs lower latency.
- Modelcar with non-OCI image —
storage.modelcar.enabled: trueexpects the predictor image to embed the model. Standard init container mode is default.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

