agentleFS
Aidas-devAidas-dev/k8s-agent-skills/skills/kserve-operator/SKILL.md

kserve-operator

Skill4 starsChanged 5 months agoMITView on GitHub

Create and manage KServe CRDs — InferenceService, ServingRuntime, InferenceGraph, LLMInferenceService, LocalModel, ClusterServingRuntime, and 22 CRDs under serving.kserve.io.

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Default storage container template applied across namespaces.

    line 591

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

KServe — Operator CRDs & Configuration

Repository: github.com/kserve/kserve
Latest release: v0.18.0 (Apr 29, 2026) — v0.19.0-rc0 available
CNCF: Incubating (Jan 2025)
License: Apache 2.0
Stars: 10k+

Architecture

RawPredictOp → Preprocess → Predict → [Explain] → Postprocess → ExplainOp
                (Transformer)  (Predictor)           (Transformer)

InferenceService (CRD) ──► KServe Controller ──► Deployment + Service + HPA + Istio/Knative
                                │
                          Storage: S3/GCS/Azure/HF/PVC/HTTP/HDFS
                                │
                     Modelcar (OCI) or PVC or storage init container

Deployment Modes

ModeNetworkingUse Case
StandardIstio VirtualServiceEnterprise, stable, full traffic control
KnativeKnative + Istio (Kourier optional)Auto-scaling to zero, serverless
RawDirect Service + Ingress/GatewaySimple, lightweight, no service mesh

Annotations Control

AnnotationEffect
serving.kserve.io/deploymentModeServerless (Knative, default), RawDeployment, ModelMesh
serving.tempaltes.kserve.io/defaultTemplate rendering default
sidecar.istio.io/injecttrue to inject Istio sidecar

CRDs (22 total — primary 9 covered here)

All under serving.kserve.io:

CRDAPI VersionScopeShort NameDescription
InferenceServicev1beta1 / v1alpha1NamespacedisvcCore serving unit
ServingRuntimev1alpha1NamespacedsrRuntime template (ModelMesh)
ClusterServingRuntimev1alpha1ClustercsrCluster-scoped runtime template
InferenceGraphv1alpha1NamespacedigRouter graph (ensemble/switch/splitter)
TrainedModelv1alpha1NamespacedtmModel-scoped config for ServingRuntime
LLMInferenceServicev1alpha1Namespaced—LLM-specific (disaggregated prefill/decode)
LocalModelCachev1alpha1NamespacedlmNode-level model cache for LocalModelNode
LocalModelNodev1alpha1ClusterlmnNode DaemonSet for model loading
ClusterStorageContainerv1alpha1ClustercscDefault storage container template

InferenceService — serving.kserve.io/v1beta1

The primary CRD for model serving.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
spec:
  predictor:
    # Built-in predictor
    sklearn:
      storageUri: s3://models/iris/model.joblib
      resources:
        requests:
          cpu: 100m
          memory: 256Mi
        limits:
          cpu: "1"
          memory: 1Gi
      readinessProbe:
        periodSeconds: 5
        successThreshold: 1
        timeoutSeconds: 1
      livenessProbe:
        periodSeconds: 5
        failureThreshold: 3
        timeoutSeconds: 1

    # ServiceOrchestration (multi-model composition)
    # serviceOrchestrationSpec:
    #   pipelining: router

    # Node selection
    # nodeSelector:
    #   node-type: gpu

    # Service account for storage access
    serviceAccountName: kserve-sa

    # Min/max replicas
    minReplicas: 1
    maxReplicas: 5

    # Scale target (Knative mode)
    scaleTarget: 1
    scaleMetric: concurrency
    containerConcurrency: 5

    # Canary rollout
    canaryTrafficPercent: 10
    canary:
      sklearn:
        storageUri: s3://models/iris-v2/model.joblib

  # Transformer (optional, pre/post-processing)
  transformer:
    container:
      image: myrepo/transformer:latest
      env:
        - name: MODEL_NAME
          value: iris
      resources:
        requests:
          cpu: 100m
          memory: 128Mi

  # Explainer (optional)
  explainer:
    alibi:
      storageUri: s3://models/iris-explainer/
      resources:
        requests:
          cpu: 100m
          memory: 256Mi

  # Storage via modelcar (OCI)
  # storage:
  #   modelcar:
  #     enabled: true
  #     pullAlways: true

Spec Fields

FieldTypeRequiredDescription
predictorPredictorSpec✅Model predictor configuration
predictor.minReplicasint❌Min pods
predictor.maxReplicasint❌Max pods
predictor.scaleTargetint❌Scale to zero target
predictor.scaleMetricstring❌concurrency, rps, cpu, memory
predictor.containerConcurrencyint❌Max concurrent requests (Knative)
predictor.canaryTrafficPercentint❌Canary traffic percentage
predictor.canaryobject❌Canary predictor spec (same structure)
predictor.serviceAccountNamestring❌KSA for storage access
predictor.readinessProbeProbe❌Custom readiness probe
predictor.livenessProbeProbe❌Custom liveness probe
predictor.annotationsmap❌Pod annotations
predictor.nodeSelectormap❌Node selector
predictor.affinityAffinity❌Pod affinity
predictor.tolerations[]Toleration❌Node tolerations
predictor.topologySpreadConstraints[]TSC❌Topology spread
predictor.sidecarInjectstring❌istio-proxy, envoy, etc.
predictor.serviceOrchestrationSpecobject❌Multi-model routing
predictor.storageStorageSpec❌Storage config
transformerComponentSpec❌Pre/post-processing
explainerComponentSpec❌Model explainability
storage.modelcar.enabledbool❌Modelcar OCI mode
storage.modelcar.pullAlwaysbool❌Always pull model image

Built-in Predictors

PredictorSpec FieldImage (default)Protocol
SKLearnsklearnkserve/sklearnserverREST/gRPC (v2)
TensorFlowtensorflowkserve/tfservingREST/gRPC (TFServing)
PyTorchpytorchkserve/torchserveREST/gRPC (TorchServe)
Tritontritonnvcr.io/nvidia/tritonserverREST/gRPC (KServe v2)
ONNXonnxkserve/onnxserverREST/gRPC (v2)
XGBoostxgboostkserve/xgbserverREST/gRPC (v2)
LightGBMlightgbmkserve/lgbserverREST/gRPC (v2)
PMMLpmmlkserve/pmmlserverREST (PMML)
Paddlepaddlekserve/paddleserverREST/gRPC
HuggingFacehuggingfacekserve/huggingfaceserverREST (TGI/transformers)
# SKLearn
predictor:
  sklearn:
    storageUri: s3://models/iris/
    protocolVersion: v2

# Triton
predictor:
  triton:
    storageUri: s3://models/triton-repo/
    runtimeVersion: 24.12
    protocolVersion: v2  # KServe v2 protocol or grpc-v2

# HuggingFace
predictor:
  huggingface:
    storageUri: s3://models/llama/
    resources:
      limits:
        nvidia.com/gpu: 1
    env:
      - name: HF_HUB_DISABLE_TELEMETRY
        value: "true"

Custom Predictor (Container)

predictor:
  containers:
    - name: kserve-container
      image: myrepo/custom-model:latest
      ports:
        - containerPort: 8080
          protocol: TCP
      env:
        - name: MODEL_DIR
          value: /models
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          nvidia.com/gpu: 1

Multi-Model Composition

predictor:
  serviceOrchestrationSpec:
    pipelining: router  # OR tf (TensorFlow DAG)
  sklearn:
    storageUri: s3://models/step1/
  pytorch:
    storageUri: s3://models/step2/

Examples

S3 model with canary rollout:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: mymodel
spec:
  predictor:
    minReplicas: 2
    canaryTrafficPercent: 10
    sklearn:
      storageUri: s3://models/v1/
    canary:
      sklearn:
        storageUri: s3://models/v2/

GPU serving with token auth:

predictor:
  minReplicas: 1
  maxReplicas: 3
  pytorch:
    storageUri: s3://models/resnet/
    resources:
      limits:
        nvidia.com/gpu: 1

ServingRuntime / ClusterServingRuntime — v1alpha1

Used with ModelMesh deployment mode. Defines runtime templates for model serving.

apiVersion: serving.kserve.io/v1alpha1
kind: ServingRuntime
metadata:
  name: sklearn-runtime
spec:
  # Supported model formats
  supportedModelFormats:
    - name: sklearn
      version: "1"
      autoSelect: true
    - name: sklearn
      version: "2"

  # Runtime container
  containers:
    - name: kserve-container
      image: kserve/sklearnserver:v0.18.0
      args:
        - --model-dir=/models
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          cpu: "1"
          memory: 2Gi

  # Node-level multi-model pooling (ModelMesh)
  multiModel: true
  modelSize: Medium  # Small | Medium | Large | XLarge
  replicas: 2

  # Pool-scoped (ModelMesh)
  pooled: true

  # Serverless scaling behavior
  scaleToZero:
    enabled: false
    gracePeriod: 300

  # Protocol
  protocolVersions:
    - v2  # KServe v2 protocol

  # Built-in adapter (REST/gRPC inference adapter)
  # builtInAdapter: {}  # optional

ClusterServingRuntime

Same spec but cluster-scoped. Available across all namespaces.

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterServingRuntime
metadata:
  name: triton-runtime
spec:
  supportedModelFormats:
    - name: triton
      autoSelect: true
  containers:
    - name: kserve-container
      image: nvcr.io/nvidia/tritonserver:24.12-py3
      args:
        - tritonserver
        - --model-repository=/models
  multiModel: true
  protocolVersions:
    - grpc-v2
    - v2

Spec Fields

FieldTypeDescription
supportedModelFormats[][]ModelFormatModel formats this runtime supports
supportedModelFormats[].namestringFormat name
supportedModelFormats[].versionstringFormat version
supportedModelFormats[].autoSelectboolAuto-select for this format
containers[][]ContainerRuntime containers
multiModelboolServe multiple models per pod
modelSizestringSmall, Medium, Large, XLarge
replicasintMin replicas
pooledboolPool-scoped runtime
scaleToZero.enabledboolAllow scale to zero
scaleToZero.gracePeriodintGrace period before scale to zero
protocolVersions[][]stringInference protocol versions
builtInAdapterobjectBuilt-in adapter config
disabledboolDisable this runtime

InferenceGraph — v1alpha1

Router graph for multi-model routing.

apiVersion: serving.kserve.io/v1alpha1
kind: InferenceGraph
metadata:
  name: model-router
spec:
  # Route nodes
  nodes:
    # Entry point
    root:
      routerType: Sequence  # Sequence | Switch | Ensemble | Splitter
      routes:
        - serviceUrl: predictor1.default.svc.cluster.local
          weight: 80
        - serviceUrl: predictor2.default.svc.cluster.local
          weight: 20

    # Conditional routing
    classifier:
      routerType: Switch
      condition: "input.category == 'a'"
      routes:
        - serviceUrl: model-a.default.svc.cluster.local
          data: "$request"
        - serviceUrl: model-b.default.svc.cluster.local
          data: "$request"

    # Ensemble (merge results)
    ensemble:
      routerType: Ensemble
      routes:
        - serviceUrl: model-a.default.svc
        - serviceUrl: model-b.default.svc

    # Splitter (fan-out)
    splitter:
      routerType: Splitter
      routes:
        - serviceUrl: shard-0.default.svc
        - serviceUrl: shard-1.default.svc

  # Entry point router
  entryPoint: root

  # Auth
  auth:
    enabled: true
    token: my-auth-token

  # Autoscaling
  autoscaler:
    initialScale: 1
    maxScale: 10
    scaleDownDelay: 300

Router Types

TypeBehavior
SequenceStep-by-step pipeline
SwitchConditional routing (first match)
EnsembleFan-out, merge results
SplitterFan-out by weight

Spec Fields

FieldTypeDescription
nodesmap[string]NodeRouter node definitions
entryPointstringEntry node name
auth.enabledboolEnable auth on graph
auth.tokenstringAuth token
auth.tokenSecretKeyRefobjectToken from secret
autoscalerAutoscalerAutoscaling config

Node fields:

FieldTypeDescription
routerTypestringSequence, Switch, Ensemble, Splitter
routes[][]RouteRoute definitions
routes[].serviceUrlstringInferenceService URL
routes[].weightintTraffic weight
routes[].datastringRequest data template
conditionstringCondition expression (Switch only)

LLMInferenceService — v1alpha1

LLM-optimized InferenceService for disaggregated prefill/decode, vLLM integration.

apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: llama-llm
spec:
  # LLM model
  modelProvider: vllm
  modelName: llama-3-70b
  modelStorage: s3://models/llama-3-70b/

  # Disaggregated serving (separate prefill/decode)
  disaggregated: true
  prefill:
    replicas: 2
    resources:
      limits:
        nvidia.com/gpu: 4
    env:
      - name: MAX_BATCH_SIZE
        value: "256"
  decode:
    replicas: 4
    resources:
      limits:
        nvidia.com/gpu: 1

  # vLLM specific
  vllm:
    maxModelLen: 8192
    tensorParallelSize: 4
    gpuMemoryUtilization: 0.9

  # Autoscaling
  autoscaler:
    initialScale: 1
    maxScale: 10
    downscaleDelay: 300

  # Storage
  storage:
    modelcar:
      enabled: true

Spec Fields

FieldTypeDescription
modelProviderstringvllm, triton, tgi, transformers
modelNamestringModel identifier
modelStoragestringModel URI (s3://..., pvc://..., hf://...)
disaggregatedboolSeparate prefill/decode pods
prefill.replicasintPrefill pod count
prefill.resourcesResourceRequirementsPrefill GPU resources
decode.replicasintDecode pod count
decode.resourcesResourceRequirementsDecode GPU resources
vllm.maxModelLenintMax sequence length
vllm.tensorParallelSizeintTP degree
vllm.gpuMemoryUtilizationfloatGPU mem utilization (0-1)
autoscalerAutoscalerScaling config
storageStorageSpecModel storage config

Storage Configuration

Storage URI Formats

SchemeFormatCredentials
S3s3://bucket/keySecretRef or IRSA (IAM roles for service accounts)
GCSgs://bucket/keyWorkload Identity or SecretRef
Azureaz://container/keySecretRef (account key)
HuggingFacehf://model-idHF_TOKEN env var
PVCpvc://claim/pathPVC must exist
HTTPhttp(s)://urlNo auth needed
HDFShdfs://nn:port/pathSecretRef

Storage Credentials via Secret

apiVersion: v1
kind: Secret
metadata:
  name: storage-creds
  annotations:
    serving.kserve.io/s3-endpoint: s3.us-east-1.amazonaws.com
    serving.kserve.io/s3-usehttps: "1"
    serving.kserve.io/s3-region: us-east-1
    serving.kserve.io/s3-verify-ssl: "1"
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: AKIA...
  AWS_SECRET_ACCESS_KEY: ...

Reference via service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kserve-sa
secrets:
  - name: storage-creds
---
spec:
  predictor:
    serviceAccountName: kserve-sa

IRSA (IAM Roles for Service Accounts) — AWS

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kserve-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/kserve-s3-access

ClusterStorageContainer — v1alpha1

Default storage container template applied across namespaces.

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterStorageContainer
metadata:
  name: s3-storage
spec:
  container:
    name: storage-initializer
    image: kserve/storage-initializer:v0.18.0
    env:
      - name: AWS_ENDPOINT_URL
        value: s3.us-east-1.amazonaws.com
      - name: AWS_REGION
        value: us-east-1
  supportedUriFormats:
    - prefix: s3://

TrainedModel — v1alpha1

Model-scoped configuration within a ServingRuntime.

apiVersion: serving.kserve.io/v1alpha1
kind: TrainedModel
metadata:
  name: iris-v2
  namespace: default
spec:
  inferenceService: sklearn-iris
  model:
    storageUri: s3://models/iris-v2/
    framework: sklearn
    memory: 256Mi
  scaleTarget: 1

LocalModelCache / LocalModelNode — v1alpha1

Node-level model caching for faster start times.

apiVersion: serving.kserve.io/v1alpha1
kind: LocalModelCache
metadata:
  name: llm-cache
spec:
  modelSize: 70Gi
  nodeGroup: gpu-nodes
  storage:
    local:
      nodeStateRoot: /mnt/local-models
    persistentVolumeClaim:
      name: model-cache-pvc
      size: 200Gi
---
apiVersion: serving.kserve.io/v1alpha1
kind: LocalModelNode
metadata:
  name: gpu-node-1
spec:
  cacheConfigRef: llm-cache
  nodeName: gpu-worker-1
  nodeAgentImage: kserve/local-model-node-agent:v0.18.0

Modelcar OCI Mode

Model stored as an OCI image, loaded into the serving pod without init container.

predictor:
  storage:
    modelcar:
      enabled: true
      pullAlways: true
  containers:
    - name: kserve-container
      image: myrepo/my-model:latest  # model AND runtime combined

Common Patterns

Minimal inference with S3:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: quick-start
spec:
  predictor:
    sklearn:
      storageUri: s3://kserve-examples/sklearn/iris

Multi-node Triton with GPU:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: triton-gpu
spec:
  predictor:
    triton:
      storageUri: s3://models/triton-repo/
      runtimeVersion: 24.12
      protocolVersion: grpc-v2
      resources:
        limits:
          nvidia.com/gpu: 2
    minReplicas: 2
    maxReplicas: 8

Canary rollout with traffic splitting:

spec:
  predictor:
    sklearn:
      storageUri: s3://models/v1/
    canaryTrafficPercent: 10
    canary:
      sklearn:
        storageUri: s3://models/v2/

InferenceGraph with conditional routing:

apiVersion: serving.kserve.io/v1alpha1
kind: InferenceGraph
metadata:
  name: smart-router
spec:
  nodes:
    root:
      routerType: Switch
      condition: "input.model == 'large'"
      routes:
        - serviceUrl: gpu-cluster.default.svc
    default:
      routerType: Sequence
      routes:
        - serviceUrl: cpu-cluster.default.svc
  entryPoint: root

LLMInferenceService with vLLM:

apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: llama-service
spec:
  modelProvider: vllm
  modelStorage: s3://models/llama-3-8b/
  disaggregated: false
  vllm:
    maxModelLen: 4096
    tensorParallelSize: 1
    gpuMemoryUtilization: 0.9
  autoscaler:
    initialScale: 1

Common Mistakes

  • Storage endpoint annotation mismatch — S3 storage requires serving.kserve.io/s3-endpoint annotation on the secret, not an env var in the predictor.
  • Missing service account — Without a service account with storage credentials, predictions fail with "model not found" at the /models mount point.
  • Multi-model vs single-model — ModelMesh (serving.kserve.io/deploymentMode=ModelMesh) uses ServingRuntime for multi-model. Standard mode uses InferenceService per model.
  • Protocol version confusion — v1 = KServe v1 protocol (legacy, sklearn/tensorflow). v2 = KServe v2 protocol (Triton, MLServer, most built-in). grpc-v2 = gRPC KServe v2.
  • Knative scaling to zero — Default is minReplicas: 0 in Knative mode. Set minReplicas: 1 to avoid cold starts.
  • Canary traffic percent integer — Must be 0–100 integer. Fractional values not supported.
  • Storage URI trailing slash — Some storage backends are sensitive to trailing slashes in storageUri.
  • InferenceService vs LLMInferenceService — LLMInferenceService is a separate CRD, not a field on InferenceService. Use the v1alpha1 API version.
  • Disaggregated prefill/decode — Requires multi-node GPU topology. Prefill uses more GPU memory per token, decode needs lower latency.
  • Modelcar with non-OCI image — storage.modelcar.enabled: true expects the predictor image to embed the model. Standard init container mode is default.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.