Walkthrough

This page walks one order end to end and says why each part is shaped the way it is. Everything it describes holds from v0.0.1 of the module onward.

It is not the place to learn today’s behaviour in short form. For that:

• User guide — order inference, read the status, find out why an order waits;
• Administration guide — enable the module, author classes, grant access, watch, disable;
• Examples — the manifests, without the narrative.

The ai-inference controller deploys the order workload itself: a StatefulSet with the request runtime, a Service, exposure, a device claim and the rest of the order objects. No delivery package is involved.

When the module is enabled, Helm creates cluster-scoped default-llm (External, Token) with the standard module: ai-inference label, replica bounds and an accelerator policy that permits a shared whole device. Use it directly: the order asks for no accelerator, and the platform assigns the device, the count and the share of it.

1. InferenceServiceClass

Bundled class (module)

After module install the cluster already has:

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: default-llm
spec:
  modelPolicy:
    allowedEndpointTypes:
      - Chat
      - Embeddings
      - Rerank
  exposurePolicy:
    type: External
    authentication: Token
    https:
      mode: CertManager
      certManager:
        clusterIssuerName: selfsigned

Expected: status.phase: Ready.

allowedEndpointTypes is what lets an order name only a class and a model: an order names no contract of its own, and the platform picks one from this list. The bundled class permits every contract the module supports — Chat, Embeddings and Rerank — and the ORDER of the list matters, because the platform takes the first element of the effective set.

The effective set is this list narrowed by whatever the source of the model says the model can serve. On the ai-models catalog path that narrowing happens: an embedding model is served as an embedding model on this very class. On the direct Hugging Face path there are no such facts, so the whole list stands and the FIRST contract wins — Chat. An order of an embedding model on that path is served as chat, and nothing refuses it. If that is your case, declare your own InferenceServiceClass whose list names the single contract you need.

Additional administrator class (optional)

spec.acceleratorPolicy.allowedDeviceClasses is optional — it sets the allow-list of DeviceClass names for orders on that class. An order may narrow this allow-list with its own spec.resources.accelerator.deviceClasses (section 2 below); an empty intersection is refused.

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: llm-chat-shared
spec:
  acceleratorPolicy:
    allowedDeviceClasses:
      - nvidia-h100-mps-mig
    allowedSharingModes:
      - Shared
    allowedPlacementTypes:
      - WholeDevice
    maxAcceleratorCount: 1
    minSharePercent: 25
    maxSharePercent: 100
  exposurePolicy:
    type: External
    authentication: Token

Expected: status.phase: Ready, conditions Validated=True, Ready=True.

2. InferenceService (namespace user)

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: support-llm
  namespace: support
spec:
  inferenceServiceClassName: default-llm
  model:
    ref:
      name: Qwen/Qwen2.5-32B-Instruct-AWQ
    src: HuggingFace

An order is a class reference, a model, and an optional narrowing of the device class list. Its spec has no other fields.

The class reference is itself optional: an order that omits spec.inferenceServiceClassName entirely skips the class-derived checks below — no allow-list intersection, no status.model.endpointType — and gets exposure and admission built into the platform (the values the bundled default-llm class carries on the day this phase shipped, not a live reference to that object). Every other policy already behaves the same with no class as with a class that leaves the corresponding block unset.

An order may narrow the allow-list of the class with its own spec.resources.accelerator.deviceClasses — the multi-element class list from the example above lets an order pick one:

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: support-llm
  namespace: support
spec:
  inferenceServiceClassName: default-llm
  model:
    ref:
      name: Qwen/Qwen2.5-32B-Instruct-AWQ
    src: HuggingFace
  resources:
    accelerator:
      deviceClasses:
        - nvidia-h100-mps-mig

An empty intersection with the class’s acceleratorPolicy.allowedDeviceClasses is refused with reason ClaimDeviceMissing; a partial intersection is accepted, and the intersection — not the order’s own list — is what travels to the launch plan.

The inference API contract is picked by the platform out of the allow-list of the class intersected with the facts of the model: the only allowed contract, or the first of several. The chosen value shows up in status.model.endpointType — the one window the owner of an order has on that choice, because the class is not theirs to read. An order that needs a different contract needs a different class.

The name of the runtime and every launch parameter come from the recipe of the launch plan, compiled against the hardware the plan chose. MM catalog keys (limitMmPerPrompt and similar) live there too. A different value is a different recipe or a different class — an administrator’s decision.

A class must declare a non-empty allow-list of API contracts. A class without one is refused on write: the contract is picked from that list, an order names none, and a class without it would leave its orders with no contract at all.

An order that still carries a runtime block is rejected by a client asking for strict field validation, and silently pruned by one that does not.

Before persisting an object, the API server validates the model source union. It rejects: the model.ref.namespace field on HuggingFace and ClusterModel references; ClusterLocal classes with https; and certificate branches that do not match https.mode.

3. Controller behavior

1. Validation: the class exists, is Ready and declares its allow-list of API contracts; model.src=HuggingFace.
2. Build the order workload objects and apply them in namespace support; every object is named after the order.
3. Compute settings.domain (for example support-llm.services.company.com) and publish status.endpoint.
4. When authentication: Token, create Secret {name}-auth and set status.authSecretName.
5. HTTP probe against the Inference API; set phase: Ready only after success.
6. After Ready, periodic healthcheck; sustained API errors → phase: Failed, reason ServiceUnhealthy (bootstrap before first Ready stays Pending, reason HealthCheckFailed).

4. Verify the result

Reading the status of a ready order is described once, in the User guide; the sections below say only what each scenario adds to that status.

5. Hugging Face token (optional)

For gated models, create a Secret in the order namespace and reference it in spec.model.authSecretRef:

spec:
  model:
    ref:
      name: meta-llama/Llama-3.1-8B
    src: HuggingFace
    authSecretRef:
      name: hf-token
      key: token   # optional; the default key is "token"

The controller validates the Secret along with the order and mounts it into the model artifact fetcher of the order workload as HF_TOKEN.

RBAC

Intentionally denied to humans: inferenceserviceclasses/planner, */status, Secrets, bind / escalate / impersonate. PrivilegedUser / Admin / ClusterAdmin deltas are empty (no extra actions).

Templates: templates/user-authz-cluster-roles.yaml, templates/rbacv2/**.

Class policies

Example embeddings-dedicated class with admission/scaling/update and authentication: None:

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: embeddings-dedicated
spec:
  admissionPolicy:
    allowedNamespaces: ["*"]
  exposurePolicy:
    type: ClusterLocal
    authentication: None
  scalingPolicy:
    allowedPriorityClassNames:
      - demo-inference-high
    minReplicas: 1
    maxReplicas: 4
  updatePolicy:
    strategy: RollingUpdate

An order under that class names no replica bounds, so the bounds of the class above apply to it:

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: search-embeddings
  namespace: ml
spec:
  inferenceServiceClassName: embeddings-dedicated
  model:
    ref:
      name: intfloat/multilingual-e5-large
    src: HuggingFace

Expected when Ready: status.constraints.minReplicas and status.constraints.maxReplicas, an HPA with the replica bounds and a StatefulSet with its update strategy. With maxReplicas above one the condition ScalingHealthy reports what the HPA sees — true when healthy, false when growth is held. status.endpoint.type mirrors exposurePolicy.type (ClusterLocal / External). Which inference API contracts a class admits is decided by allowedEndpointTypes on the class, required and non-empty; an order names no contract itself. Refusal reason of this validation: NamespaceNotAllowed.

Integration with ai-models

Requires module ai-models with its internal lookup API when catalog integration is enabled. Public ModuleConfig uses catalog.mode (Enabled by default, or None to disable catalog clients). Internal transport defaults to aiInference.aiModels.catalogTransport: Rest: controller and planner call GET /api/internal/v1/models/lookup by model.ref. When Rest is unavailable, they immediately read Model / ClusterModel CRs in the same reconcile. Explicit rollback: internal catalogTransport: Kubernetes (CR-only). The distribution and catalog-import APIs are not used for model-facts lookup.

ModuleConfig (public catalog mode)

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: ai-inference
spec:
  settings:
    catalog:
      mode: Enabled                 # None — disable catalog clients / RBAC

Internal catalog transport (not ModuleConfig)

In openapi/values.yaml (module internal values, not public ModuleConfig):

aiInference:
  aiModels:
    catalogTransport: Rest         # default; Kubernetes — explicit rollback
    catalogLookup:                 # used when catalogTransport is Rest
      baseURL: https://ai-models-controller.d8-ai-models.svc.cluster.local:8080
      path: /api/internal/v1/models/lookup
      timeoutSeconds: 10
      notReadyRequeueSeconds: 30

InferenceServiceClass with modelPolicy

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: llm-chat-shared
spec:
  acceleratorPolicy:
    allowedDeviceClasses:
      - nvidia-h100-mps-mig
    allowedSharingModes:
      - Shared
    allowedPlacementTypes:
      - Partition
    maxAcceleratorCount: 1
  admissionPolicy:
    allowedNamespaces: ["*"]
  modelPolicy:
    allowedEndpointTypes:
      - Chat
    maxParameterCount: 70B
  exposurePolicy:
    type: External
    authentication: Token
  scalingPolicy:
    allowedPriorityClassNames:
      - demo-inference-normal
    minReplicas: 1
    maxReplicas: 1
  updatePolicy:
    strategy: RollingUpdate

InferenceService with catalog model.ref

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: support-llm
  namespace: support
spec:
  inferenceServiceClassName: llm-chat-shared
  model:
    ref:
      kind: ClusterModel
      name: qwen2-5-32b-instruct-awq
    src: ai-models

For a namespace Model, set kind: Model, name, and optional namespace in ref; the built workload does not carry the model namespace.

Controller behavior

1. Validation of the class and its policies.
2. Catalog fact lookup by model.ref (default — over HTTP; when Rest is unavailable — immediate CR read; with Kubernetes — CR status only).
3. Local catalog matrix (modelPolicy against the facts): on success ModelResolved=True; otherwise ModelResolved=False with reason (ModelNotFound, ModelNotReady, …).
4. The workload of the order is not built until ModelResolved=True.
5. On success the platform builds the objects of the order itself; the API contract it selected is published in status.model.endpointType and the scope of the model in status.constraints.modelScope.
6. API health probe and phase: Ready.

Verify status

When Ready, expect:

• status.model.endpointType: Chat — the API contract the platform picked from the class allow-list
• status.constraints.modelScope: Cluster — from catalog facts
• status.endpoint.type: External — mirrors exposurePolicy.type, not the API type
• condition ModelResolved: True

kubectl get inferenceservice support-llm -n support -o jsonpath='{.status.model.endpointType}{"\n"}{.status.endpoint.type}{"\n"}'
kubectl get inferenceservice support-llm -n support -o jsonpath='{.status.constraints.modelScope}{"\n"}'

Resource planner

Builds on the catalog section above (model.ref, ModelResolved). Adds launch planning, condition Planned and status.resolved. Requires module gpu for the GPU inventory and the in-cluster resource planner Deployment shipped with ai-inference.

Compatibility: bundled default-llm and classes without acceleratorPolicy: the controller does not call the planner and does not publish Planned / status.resolved.

Prerequisites

• Ready ClusterModel / Model, from the catalog section above.
• Module gpu with PhysicalGPU inventory visible to the planner.
• Planner Service reachable at the module default (aiInference.planner.baseURL in chart values; see openapi/values.yaml).

InferenceServiceClass with acceleratorPolicy

Extend the llm-chat-shared class from the catalog section above with acceleratorPolicy (administrator manifest):

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: llm-chat-shared
spec:
  acceleratorPolicy:
    allowedDeviceClasses:
      - nvidia-h100-mps-mig
  admissionPolicy:
    allowedNamespaces: ["*"]
  modelPolicy:
    allowedEndpointTypes:
      - Chat
    maxParameterCount: 70B
  acceleratorPolicy:
    allowedSharingModes: [Shared]
    allowedPlacementTypes: [Partition, WholeDevice]
    maxAcceleratorCount: 1
  exposurePolicy:
    type: External
    authentication: Token
  scalingPolicy:
    allowedPriorityClassNames:
      - demo-inference-normal
    minReplicas: 1
    maxReplicas: 4
  updatePolicy:
    strategy: RollingUpdate

Expected: status.phase: Ready.

InferenceService with accelerator and launchStrategy

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: support-llm
  namespace: support
spec:
  inferenceServiceClassName: llm-chat-shared
  model:
    ref:
      kind: ClusterModel
      name: qwen2-5-32b-instruct-awq
    src: ai-models
  launchStrategy: Throughput

launchStrategy selects the compiled recipe branch (Latency, Throughput, or Balance; default Latency when omitted). The branches genuinely differ on the vast majority of hardware — a wider context horizon or a throughput-oriented balance is reachable through this field, not by changing the recipe or class as a whole. The count of devices is not an order field — the launch plan assigns it within acceleratorPolicy.maxAcceleratorCount of the class.

The share of a device is ASSIGNED by the launch plan, and the order names none. The plan takes the smallest whole percent that admits the memory the model needs on the chosen device, clamps it into minSharePercent/maxSharePercent of the class, and falls back to a whole device when the class permits no sharing, when the chosen placement is not a whole device in a sharing mode, or when either memory figure is unknown. The assigned value is published in status.resolved.sharePercent, so the owner of an order sees how much of the card is theirs. A share below 100 also requires the selected DRA device to publish allowMultipleAllocations with a sharePercent request policy. For a share of 1..99 the controller sends capacity.requests.memory from the model requirement of the launch plan, written into the claim as gpu.deckhouse.io/memory. Model memory and the MPS compute share are accounted independently.

Controller behavior

1. Validation and the catalog matrix → ModelResolved=True.
2. The accelerator policy of the class bounds the outcome of planning; the order has nothing of its own to check against it.
3. POST /api/v1/launch-plan — the planner returns placement + runtime recipe; on success:
   • status.resolved — the placement the order runs on: deviceClass, acceleratorProductName, acceleratorMemoryGiB, placementMode, sharingMode, sharePercent; everything else the planner computed travels in the launch plan of the order and is not published in the status;
   • condition Planned=True, reason LaunchPlanCalculated.
4. On planner refusal — Planned=False with stable reason (NoCapacity, QuantizationMismatch, …); the workload objects are not created or updated until planning succeeds.
5. After Planned=True the recipe from the plan is applied into the arguments of the runtime container; then the health probe → phase: Ready, as in the sections above.
6. When maxReplicas > 1 — HPA observation and the background preemption loop may adjust donor maxReplicas; the bundled default-llm class is out of scope.

Verify status

While reconciling, expect Planned before the workload of the order exists. When Ready:

status:
  phase: Ready
  resolved:
    deviceClass: nvidia-hopper-s3-shared
    acceleratorProductName: NVIDIA H100 80GB HBM3
    acceleratorMemoryGiB: 24
    placementMode: Partition
    sharingMode: Shared
    replanCount: 0
  conditions:
    - type: ModelResolved
      status: "True"
    - type: Planned
      status: "True"
      reason: LaunchPlanCalculated
    - type: Ready
      status: "True"

kubectl get inferenceservice support-llm -n support -o jsonpath='{.status.resolved}{"\n"}{.status.conditions[?(@.type=="Planned")]}{"\n"}'
kubectl get statefulset support-llm -n support -o jsonpath='{.spec.template.spec.containers[0].command}{"\n"}'

Planner refusal examples: Planned=False, reason: NoCapacity or QuantizationMismatch.

Planner runtime fallback

Builds on the resource planner section above. When there is no entry in runtime-recipes.compiled.yaml for (family, model.name) and runtime fallback is enabled, the planner synthesizes a generic vLLM recipe from the images/catalogs/ tables (recipe-taxonomy.yaml and platform-hardware-families.yaml), estimates vramMinimumGiB, and returns recipeSource: fallback. Compiled presets and recipes always win over the fallback path.

Compatibility: with aiInference.planner.runtimeFallback.enabled: false (chart values / module internal config), behavior matches the resource planner path above — no synthetic recipe. Classes without acceleratorPolicy are unchanged.

Module settings

In openapi/values.yaml (not public ModuleConfig):

aiInference:
  planner:
    runtimeFallback:
      enabled: true   # default true

The controller forwards the flag to the planner as runtimeFallback.enabled. To disable:

aiInference:
  planner:
    runtimeFallback:
      enabled: false

Minimum metadata for fallback

The controller does not parse quantization from the name — only the planner on the fallback branch. For modelPolicy.maxParameterCount, the controller judges whatever the path of the order resolved: the catalog fact on the ai-models path, the order value on the direct one, and the same model.name parsing when that first rung is empty.

Model and ClusterModel lookup facts do not return tensor geometry today. The planner gets geometry from a shipped recipe preset when one exists; otherwise the fallback estimate is geometry-blind. Publishing geometry from ai-models is not available today, and this module does not assume future catalog fields.

Example: model without a compiled recipe

ClusterModel named qwen2-5-32b-instruct-awq whose lookup facts omit parameterCount and quantization. The order also omits spec.model.parameterCount, so size and quantization come from model.name:

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: support-llm-fallback
  namespace: support
spec:
  inferenceServiceClassName: llm-chat-shared
  model:
    ref:
      kind: ClusterModel
      name: qwen2-5-32b-instruct-awq
    src: ai-models

Expected once the GPU inventory is populated:

status:
  resolved:
    deviceClass: nvidia-hopper-s3-shared
    acceleratorProductName: NVIDIA H100 80GB HBM3
    acceleratorMemoryGiB: 24
    placementMode: Partition
    sharingMode: Shared
  conditions:
    - type: Planned
      status: "True"
      reason: LaunchPlanCalculated

Without a recognizable size (no catalog, no spec.model.parameterCount, name without 32b/360m token) — Planned=False, reason NoCompatibleRuntimeAvailable.

Verify status

kubectl get inferenceservice support-llm-fallback -n support \
  -o jsonpath='{.status.resolved.deviceClass}{"\n"}{.status.resolved.acceleratorProductName}{"\n"}'
kubectl get statefulset support-llm-fallback -n support \
  -o jsonpath='{.spec.template.spec.containers[0].command}{"\n"}'