User guide

This guide describes how to create and modify virtualization module resources in a project or cluster namespace.

Quick start on creating a virtual machine

This section walks through a minimal scenario: you create an Ubuntu 24.04 image, a disk from that image, and a virtual machine (VM), connect to it over the console, and then delete the created resources.

Images

An image holds the contents of a disk that you use to create virtual machine disks. A VirtualImage is created in a project and available only in the project or namespace where it was created.

A virtual machine accesses an attached image in read-only mode.

An image appears in a project in three steps:

1. You create a VirtualImage resource and specify a data source in it.
2. Deckhouse Platform (DP) downloads the image from that source to the storage, which is either DVCR or a PVC, depending on the selected type.
3. The downloaded image becomes available for creating disks.

Sources and storage options

The image source can be an HTTP server hosting the image file, a container image registry, or a file on your computer that you upload from the command line. You can also create an image from another image, from a virtual machine disk, or from a disk snapshot.

Image types, supported file formats, and compression algorithms are described in Image types and formats in the admin guide.

The downloaded image is stored in one of two ways, set by the .spec.storage parameter:

• ContainerRegistry: The default option, the image is stored in DVCR.
• PersistentVolumeClaim: The image is stored in a PVC. This option is preferable if the storage can clone PVCs quickly, because disks are created from such an image faster.

The PHASE column in the d8 k get vi output shows the progress of image creation; for its values, see the .status.phase field. To follow the creation in real time, add the -w flag. If an image stays not ready for a long time, check the .status.conditions block and the d8 k describe vi output for the reason.

Until an image reaches the Ready phase, you can change its .spec block, and the download restarts after each change. For a ready image, the .spec block can no longer be changed. For all image parameters, see VirtualImage.

Creating an image from an HTTP server

The simplest way to create an image is to provide a link to a file hosted on an HTTP server.

Verifying the integrity of a downloaded image

The checksum block makes DP verify what it downloaded from the HTTP server. The image reaches the Ready phase only if the downloaded file matches every specified checksum, otherwise the resource ends up in the Failed phase:

d8 k apply -f - <<EOF
apiVersion: virtualization.deckhouse.io/v1alpha2
kind: VirtualImage
metadata:
  name: ubuntu-24-04
spec:
  storage: ContainerRegistry
  dataSource:
    type: HTTP
    http:
      url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
      checksum:
        sha256: 78be890d71dde316c412da2ce8332ba47b9ce7a29d573801d2777e01aa20b9b5
EOF

Take the checksum from the mirror that publishes the image and put it in the field of the matching algorithm:

The speeds are an order of magnitude, not a promise. They were measured on an x86-64 CPU with the SHA instruction set, and a CPU without it computes SHA-1 and SHA-256 several times slower. What holds on any CPU is the distance between the rows. SHA-1 and SHA-256 are computed by dedicated instructions, MD5 and SHA-512 by hand-written assembly, and all four hash data faster than it arrives over the network, so their cost stays invisible against the download itself.

The Streebog algorithms have no hardware support anywhere and are about two orders of magnitude slower. For a 10 GiB image, that’s around ten minutes of hashing alone, and image creation becomes CPU-bound rather than network-bound. Use them only when a GOST checksum is actually required. Both lengths cost the same, because GOST R 34.11-2012 uses one compression function for 256 and 512 bits, and the shorter variant differs only in the initial value.

Checksums are computed in a single pass over the downloaded data, so specifying several checksums at once costs the sum of their times. Empty fields cost nothing.

The same block is available for the Upload source in the dataSource.upload.checksum parameter and works the same way. The uploaded data is verified against every specified checksum, and on a mismatch the resource stays in the Failed phase.

Storing an image in a PVC

To create disks from an image faster, store it in a PVC. DP can then clone the volume instead of unpacking the image again.

Creating an image from a container image registry

DP can pull an image from an external container image registry, but the disk file must be located in the container image under the /disk path. The following steps show how to prepare such a container image and create a project image from it.

Uploading an image from the command line

If the image file is on your computer, upload it directly. DP creates a temporary upload endpoint for this and waits for the data.

Creating an image from a disk

You can create an image from a disk if the disk isn’t attached to any virtual machine, or if the machine it’s attached to is powered off.

d8 k apply -f - <<EOF
apiVersion: virtualization.deckhouse.io/v1alpha2
kind: VirtualImage
metadata:
  name: linux-vm-root
spec:
  storage: ContainerRegistry
  dataSource:
    type: ObjectRef
    objectRef:
      kind: VirtualDisk
      name: linux-vm-root
EOF

Creating an image from a disk snapshot

You can create an image from a disk snapshot if the snapshot is in the Ready phase.

d8 k apply -f - <<EOF
apiVersion: virtualization.deckhouse.io/v1alpha2
kind: VirtualImage
metadata:
  name: linux-vm-root
spec:
  storage: ContainerRegistry
  dataSource:
    type: ObjectRef
    objectRef:
      kind: VirtualDiskSnapshot
      name: linux-vm-root-snapshot
EOF

Image properties are convenient to review in the web interface, in Virtualization → Images:

• The list shows the image name, status, size, format in the Type column, and visibility scope in the Availability column, and the filters narrow it down by status, image, and type.
• On the image page, the Information tab collects the creation parameters in the Storage and Source blocks, and the State block shows the average download speed, format, unpacked size, size in storage, creation time, the CD-ROM flag, and the image path in DVCR.
• The Meta and YAML tabs show labels with annotations and the full resource specification.

Disks

A disk stores virtual machine (VM) data, including the operating system and application files. A disk is described by the VirtualDisk resource, and its specification consists of two blocks:

• persistentVolumeClaim: Storage parameters, that is, the StorageClass and the size.
• dataSource: The data source, which can be an image, another disk, or a snapshot.

Without the dataSource block, an empty disk is created, and then you have to specify at least the size in persistentVolumeClaim. If a source is set, you can omit the persistentVolumeClaim block, and DP takes the size from the source and picks the storage class based on it too. When no class can be picked, DP uses the cluster-wide default StorageClass or the class set for disks in the module settings.

The PHASE column in the d8 k get vd output shows the progress of disk creation; for its values, see the .status.phase field. If a disk stays not ready for a long time, the .status.conditions block tells you the reason.

Until a disk reaches the Ready phase, you can change any field of the .spec block, and the creation restarts after the change. For a ready disk, only the size and the storage class remain editable, in the .spec.persistentVolumeClaim.size and .spec.persistentVolumeClaim.storageClassName parameters.

How storage affects a disk

Disk behavior depends on the storage behind the selected StorageClass. The differences show up in two properties.

The volume type determines the format in which DP creates the disk. On file system volumes (FileSystem, for example NFS), the disk is created in the qcow2 format, and on block devices (Block, for example iSCSI or Ceph RBD), data is written directly. Some storage types support both.

The volume binding mode determines when the disk is created:

• Immediate: The disk is created right away, independently of virtual machines, and you can attach it to a machine on any cluster node.

  

• WaitForFirstConsumer: The disk is created only after it’s attached to a virtual machine, and it’s placed on the node where that machine starts.

  

DP determines the remaining parameters, including the disk format, on its own from the capabilities of the selected StorageClass.

To view the available storage types, run the following command:

d8 k get storageclass

Example output:

NAME                   PROVISIONER                           RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
rv-thin-r1 (default)   replicated.csi.storage.deckhouse.io   Delete          Immediate              true                   48d
rv-thin-r2             replicated.csi.storage.deckhouse.io   Delete          Immediate              true                   48d
nfs-4-1-wffc           nfs.csi.k8s.io                        Delete          WaitForFirstConsumer   true                   30d

In the web interface, the same list is available on the System tab, in Storage → Storage classes.

Creating an empty disk

An empty disk is what you need to install an operating system on it or to store data separately from the system disk.

Creating a disk from an image

You can fill a disk with data from an image created earlier, either a project VirtualImage or a cluster ClusterVirtualImage.

Specifying the disk size is optional. If you don’t set it, DP creates the disk exactly at the unpacked size of the image, and if you do set it, the size must be no smaller than the unpacked one.

Uploading a disk from the command line

If the image file is on your computer, upload it straight into a disk. DP creates a temporary upload endpoint for this and waits for the data.

Disk properties are convenient to review in the web interface, in Virtualization → Disks:

• The list shows the disk name, status, size, storage class in the Class column, the virtual machine that uses the disk in the Used by column, and the resource age.
• On the disk page, the Configuration tab shows the data source, size, storage class, and the list of VMs in the Used in row.
• The Diagnostics tab shows the PVC name, its phase, size, storage class, PV name, and age, as well as the duration of the disk creation stages in the Diagnostics summary block.

Changing the disk size

You can grow a disk even while it’s attached to a running virtual machine (VM). You can’t shrink a disk.

Migrating disks to other storage

In commercial DP editions, you can move a disk to another storage by changing its storage class. The move works both for disks defined in the VM specification and for disks attached as a separate resource.

To move a disk, specify the new storage class in the .spec.persistentVolumeClaim.storageClassName parameter:

d8 k patch vd disk --type=merge --patch '{"spec":{"persistentVolumeClaim":{"storageClassName":"new-storage-class-name"}}}'

# You can achieve the same result by editing the resource.
d8 k edit vd disk

After that, a live migration of the VM starts, during which the disk moves to the new storage.

If you need to move several disks of the same machine, change the storage class one disk at a time:

d8 k patch vd disk1 --type=merge --patch '{"spec":{"persistentVolumeClaim":{"storageClassName":"new-storage-class-name"}}}'
d8 k patch vd disk2 --type=merge --patch '{"spec":{"persistentVolumeClaim":{"storageClassName":"new-storage-class-name"}}}'

DP retries a failed migration with a growing delay. The first attempt runs immediately, the next ones after 5 and 10 seconds, then the delay doubles and from the seventh attempt stays at 300 seconds. To cancel the migration, restore the previous storage class in the specification.

Exporting a disk or a snapshot

Export writes the contents of a disk or its snapshot to a file so that you can move the data outside the cluster. For this feature to work, the storage-volume-data-manager module has to be enabled, and the command line requires the d8 utility version 0.20.7 or later.

To import a downloaded disk back into the cluster, upload it as an image or as a disk.

d8 data export download -n <NAMESPACE> vd/<VD_NAME> -o file.img

d8 data export download -n <NAMESPACE> vds/<VD_SNAPSHOT_NAME> -o file.img

Virtual machines

To create a virtual machine (VM), use the VirtualMachine resource. Its parameters let you configure:

• the virtual machine class;
• the resources required for the virtual machine to run (CPU, memory, disks, and images);
• the placement rules for the virtual machine on cluster nodes;
• the bootloader settings and optimal parameters for the guest OS;
• the virtual machine startup policy and the policy for applying changes;
• the initial configuration scripts (cloud-init);
• the list of block devices.

For a full description of virtual machine configuration parameters, see the configuration reference.

Creating a virtual machine

The following steps show how to start an Ubuntu 24.04 virtual machine on the disk you created earlier. The cloud-init script installs the qemu-guest-agent agent and the nginx service, and creates the cloud user with the cloud password.

Virtual machine life cycle

From creation to deletion, a virtual machine goes through several phases. The current one is shown by the .status.phase field, and the details of what’s happening to the machine are in the .status.conditions block.

The conditions in the .status.conditions block answer the question of why the machine is in its current phase. To view the ones that have a message, run the following command:

d8 k get vm <VM_NAME> -o json | jq '.status.conditions[] | select(.message != "")'

Where <VM_NAME> is the virtual machine name.

Diagnostics by phase

While a VM is in the Pending phase, it waits for its dependent resources to become ready, that is, disks, images, the VM class, and the secret with the initial configuration script. A delay in this phase means that one of the resources isn’t ready or that the namespace or project quotas are exhausted. The conditions ending in Ready show what exactly blocks the startup:

d8 k get vm <VM_NAME> -o json | jq '.status.conditions[] | select(.type | test(".*Ready"))'

In the Starting phase, the dependent resources are ready and DP starts the VM on one of the nodes. If the startup drags on, there’s no suitable node, or the suitable nodes lack CPU or memory. The Running condition reports the reason:

d8 k get vm <VM_NAME> -o json | jq '.status.conditions[] | select(.type=="Running")'

In the Migrating phase, the machine moves to another node by live migration. The migration doesn’t start or gets interrupted if the CPU instruction sets on the nodes are incompatible, the kernel versions differ, no node matches the placement rules, or the suitable nodes lack resources. The Migrating condition together with the .status.migrationState block shows the migration progress:

d8 k get vm <VM_NAME> -o json | jq '.status | {condition: .conditions[] | select(.type=="Migrating"), migrationState}'

The Terminating phase is irreversible, all resources associated with the VM are released, but the resources themselves aren’t deleted.

Conditions of a running VM

For a running machine, a few conditions are worth watching:

• AgentReady with the True status means that qemu-guest-agent is running in the guest system, and then the .status.guestOSInfo block contains information about the guest OS.
• FirmwareUpToDate with the False status means that it’s time to update the VM firmware.
• ConfigurationApplied with the False status means that the specified configuration hasn’t been applied to the running machine yet.
• AwaitingRestartToApplyConfiguration with the True status means that some of the changes apply only after a restart, and you have to perform it manually.
• SizingPolicyMatched with the False status means that the machine resources don’t match the sizing policy of its class. Until you bring the parameters in line with the policy, you can’t save configuration changes.
• Migratable shows whether the machine can be moved by live migration. The condition is computed only for a running VM, and a powered-off one doesn’t have it. The False status with the VirtualMachineNoMigrationTarget reason means that the machine itself is suitable for migration, but there’s no suitable node in the cluster. The True status with the VirtualMachineWaitingForMigrationTarget reason means that suitable nodes exist, but none of them can accept the machine right now, and this state resolves on its own.

Eviction from a node

The EvictionRequired condition appears when the node with your VM is put into maintenance mode. If the node was only made unschedulable with the d8 k cordon command but maintenance hasn’t started, the condition doesn’t appear.

The condition message tells you what will happen to the machine, namely a live migration without stopping the guest OS, a restart by DP with the cluster administrator’s permission, or waiting until the machine is restarted. A machine that can be moved by live migration isn’t restarted just to free the node. Until the eviction starts, the condition is only a warning, because maintenance can be canceled.

After a restart, the machine starts on another suitable node. If there’s no such node, it stays in the Pending phase, and the Running condition shows the reason from the scheduler. A node in maintenance mode doesn’t accept new machines, so a VM pinned to it by placement rules or using a device passed through from it starts only after the node returns to service.

Viewing the state in the web interface

The web interface shows the phase of a machine, its resources, and current problems on the machine page.

1. Go to the Projects tab and select the project you need.
2. Go to Virtualization → Virtual machines.
3. Select the VM you need from the list and click its name.

The page header shows the current VM phase, its IP address, class, resource configuration, the number of attached disks and images, the node it’s placed on, whether the guest OS agent is running, and the time since startup. The page itself is split into the Configuration, Monitoring, Operations, Events, VNC, TTY, Network policies, Snapshots, Diagnostics, Meta, and YAML tabs.

On the Configuration tab of a running VM, usage charts appear next to the CPU cores and Memory size fields. In the Disks block, each device shows its boot order number, name, size, status, attachment method, storage class, and current disk load. In the Networks block, the network name, its status, and the IP and MAC addresses are shown, and the main cluster network is labeled Main.

Configuring CPU and coreFraction

Two parameters set the CPU resources of a machine. The .spec.cpu.cores parameter sets the number of virtual cores, and .spec.cpu.coreFraction sets the guaranteed share of the power of each of them.

spec:
  cpu:
    cores: 2
    coreFraction: 20%

In this example, the machine gets two virtual cores and a guaranteed 20% of the power of each, that is, 0.4 cores in total, regardless of the node load. When the node has free resources, the machine can take both cores in full. Such a reserve lets you keep more machines on a node than it has physical cores, without losing stability under load.

If coreFraction isn’t set, each virtual core gets 100% of a physical one.

The guaranteed share is taken into account when selecting a node, so a machine doesn’t start where the node can’t provide the guarantees to all machines placed on it. The figure shows two machines with one core each, the first with coreFraction: 20% and the second with coreFraction: 80%.

Automatic coreFraction (Auto)

DP can pick the CPU time share on its own, following how much the machine consumes.

Instead of a fixed percentage, you can set coreFraction: Auto. Then DP picks the fraction, raising it when the machine lacks CPU and lowering it when the machine is idle. The number of cores and the amount of memory stay unchanged, and the new fraction applies without a restart.

spec:
  cpu:
    cores: 4
    coreFraction: Auto

The selection works as follows:

• The initial core fraction is 10% or the nearest value allowed by the sizing policy.
• The fraction changes in steps. If the class sizing policy defines a coreFractions list, its values become the steps, otherwise 5%, 10%, 15%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, and 99% are used.
• The 100% value is never selected automatically, because at that value the CPU requests equal the limits, and such a machine can’t be changed without a restart. If 100% is listed in coreFractions, it simply isn’t used, and the ceiling becomes the next value down, while without a sizing policy the ceiling is 99%.
• For the same reason, the sizing policy has to leave at least two values to choose from. A policy that allows only 50%, or 50% and 100%, would pin the machine at 50% forever, so such a combination is rejected and you have to set the fraction explicitly.
• The recommended value is published in the .status.recommendedResources.cpu.coreFraction field, and the applied one in .status.resources.cpu.coreFraction. Every change is reported by the CoreFractionScaling event.
• If the node doesn’t have room for the new requests, the machine moves to another node and keeps running.

The CoreFractionAutoscaling condition shows whether the selection is running. While it works, the condition has the True status, and for the first few minutes, until statistics accumulate, the reason is WaitingForRecommendation, and then CoreFractionAutoscalingEnabled.

If the selection becomes unavailable, the condition switches to False, and the reason explains why:

• CoreFractionAutoscalingDisabled: Vertical autoscaling is disabled.
• InPlaceResizeDisabled: In-place resource changes are disabled.
• SizingPolicyHasNoSteps: The sizing policy is narrowed down to a single value.

The machine keeps running with its current fraction, but stops following the load.

To opt out of automatic selection, set an explicit percentage. Switching between 100% and Auto in either direction requires a machine restart, because it changes the QoS class, and the other transitions apply on the fly.

Sizing policy

An administrator can restrict the resource combinations available to machines of a certain class by setting a sizing policy in the .spec.sizingPolicies parameter of the VirtualMachineClass resource. If there’s no policy, resources are set freely.

The policy splits the number of cores into ranges and sets the allowed amount of memory and the allowed coreFraction values for each of them:

spec:
  sizingPolicies:
    - cores:
        min: 1
        max: 4
      memory:
        min: 1Gi
        max: 8Gi
      coreFractions: [5, 10, 20, 50, 100]
    - cores:
        min: 5
        max: 8
      memory:
        min: 5Gi
        max: 16Gi
      coreFractions: [20, 50, 100]

With such a policy, a machine with two cores falls into the first range, gets from 1 to 8 GiB of memory and one of the values 5%, 10%, 20%, 50%, or 100%. A machine with six cores falls into the second range, where 5 to 16 GiB of memory is available and the core fraction is 20%, 50%, or 100%.

The policy also limits oversubscription. For example, the minimum value coreFraction: 20% guarantees each machine one fifth of a core, which means oversubscription doesn’t exceed 5 to 1.

If the machine configuration doesn’t match the policy, the SizingPolicyMatched condition with the False status appears in the status. Such a machine keeps running, but you can’t save changes to its configuration until the resources are brought in line with the policy. The same happens when an administrator changes the policy of a class whose machines are already running.

Besides the bounds, a range can set a grid step in the cores.step and memory.step parameters, and memory bounds per single core in the memory.perCore block.

A request that violates the policy is rejected with a message that names the parameter and the allowed values. Every message ends with the hint check the sizing policy of the VirtualMachineClass or contact the administrator for more information, which is omitted below.

For the supercpu class with the policy above, the messages look like this:

• the number of cores is outside all ranges, cores: 10: does not match any sizing policy of VirtualMachineClass "supercpu": its 10 CPU core(s) fall outside the allowed ranges (1-4, 5-8); set the number of cores (spec.cpu.cores) accordingly;
• an unsupported core fraction, cores: 2 and coreFraction: 30%: the CPU core fraction "30%" is not allowed; set the core fraction (spec.cpu.coreFraction) to one of: 5%, 10%, 20%, 50%, 100%;
• memory outside the range, cores: 2 and size: 16Gi: the memory size (16Gi) is out of the range allowed by the sizing policy; set the memory size (spec.memory.size) between 1Gi and 8Gi.

If a range defines a step or per-core memory bounds, four more messages become possible:

• cores off the step grid: the number of CPU cores (7) does not match the sizing policy step; set the number of cores (spec.cpu.cores) to 6 or 8;
• memory off the step grid: the memory size (1536Mi) does not match the sizing policy step; set the memory size (spec.memory.size) to 1Gi or 2Gi;
• memory per core outside the range: the memory size (18Gi) is not allowed for 6 CPU core(s); set the memory size (spec.memory.size) between 6Gi and 12Gi, or change the number of cores (spec.cpu.cores) (the sizing policy allows between 1Gi and 2Gi of memory per core);
• memory per core off the step grid: the memory size (2560Mi) does not match the per-core sizing policy step for 2 CPU core(s); set the memory size (spec.memory.size) to 2Gi or 4Gi, or change the number of cores (spec.cpu.cores).

When there are several violations, all the reasons are listed in one message under the does not match the sizing policy of VirtualMachineClass "supercpu" for several reasons: heading.

CPU topologies

The topology determines how the CPU cores of a machine are distributed across sockets, and compatibility with applications sensitive to the CPU configuration depends on it. You set only the total number of cores in the .spec.cpu.cores parameter, and DP calculates the number of sockets itself:

spec:
  cpu:
    cores: 20

The more cores there are, the more sockets they’re split across, and the larger the step with which you can change their number. The total number of cores has to be a multiple of the number of sockets, otherwise the request is rejected.

For example, 20 cores give two sockets of 10 cores, and 80 cores give eight sockets of 10. The maximum for one machine is 248 cores.

DP publishes the calculated topology in the status:

status:
  resources:
    cpu:
      topology:
        coresPerSocket: 10
        sockets: 2

The memory overhead depends on the actually active cores and amounts to 8 MiB per logical core, that is, per the product of the number of sockets, cores per socket, and threads per core.

Configuring the OS type and bootloader

The osType parameter defines the operating system type and applies the optimal set of virtual devices and parameters for a virtual machine (VM) to work correctly.

Supported values:

• Generic (default): For Linux and other operating systems. The standard virtual device configuration is used.
• Windows: For Microsoft Windows operating systems. Automatically enables Hyper-V features, a TPM device, and other settings optimized for Windows.
• Legacy: For operating systems without built-in AHCI and virtio drivers: Windows XP, Windows 2000, Windows Server 2003, DOS-era systems, and Linux with a kernel older than 2.6.19. Such a VM gets the i440fx chipset, and with enableParavirtualization: false it also gets the IDE bus for disks and CD-ROM and the RTL8139 network adapter, whose drivers these operating systems have.

The set of virtual devices the guest OS sees:

USB device passthrough works for Legacy, but the UHCI controller is limited to the USB 1.1 speed of 12 Mbps, so fast storage in such a VM hits the bus limit.

Choose Legacy when the guest operating system can’t work with an AHCI controller, not just because it’s old. For Linux with kernel 2.6.19 and newer, osType: Generic with enableParavirtualization: false is a fit, because there’s no four-device limit there and attaching disks on the fly through VirtualMachineBlockDeviceAttachment remains available.

For Legacy, the following isn’t available:

• Changing the number of CPU cores and the amount of memory on a running VM, because these guest operating systems don’t bring them into service. The change is accepted, the VM shows it in .status.restartAwaitingChanges along with the AwaitingRestartToApplyConfiguration condition, and it applies after a restart.
• Changing the contents of .spec.blockDeviceRefs on a running VM, because these disks stay static in both paravirtualization modes, so add a block device before the VM starts.
• The EFI and EFIWithSecureBoot bootloaders, as well as initialization (cloud-init and Sysprep), because these guest operating systems don’t support them.
• Guest OS information in the VM status and file system information.

You can install the QEMU guest agent in such an OS from an archived virtio-win release, and the VM does show AgentReady, but its version is too old for DP. The VM gets the AgentVersionNotSupported condition, guest OS information isn’t collected, and there’s nothing to update the agent to.

A snapshot with requiredConsistency: true also doesn’t complete successfully, but for a different reason. DP requests a file system freeze, and the agent replies that the command is disabled in its build with the guest-fsfreeze-status has been disabled for this instance message. On Windows, the freeze goes through the VSS provider, which isn’t in this build. The snapshot waits in the InProgress phase for about ten minutes and moves to Failed, so set requiredConsistency: false for such VMs.

With enableParavirtualization: false, one more limitation applies. There can be no more than four block devices in total, because the IDE bus provides two channels with two devices each.

Example configuration for a Windows XP virtual machine:

spec:
  osType: Legacy
  bootloader: BIOS
  enableParavirtualization: false
  # other parameters...

The bootloader parameter defines the bootloader type of the virtual machine:

• BIOS (default): Uses the legacy BIOS.
• EFI: Uses the Unified Extensible Firmware Interface (UEFI/EFI).
  • EFIWithSecureBoot: Uses UEFI/EFI with Secure Boot support.

Example configuration for a Windows virtual machine:

spec:
  osType: Windows
  bootloader: EFI
  # other parameters...

Example configuration for a Linux virtual machine (you can omit the default values):

spec:
  osType: Generic
  bootloader: BIOS
  # other parameters...

The enableParavirtualization parameter controls the use of the virtio bus for attaching the virtual devices of the VM. A change to this parameter takes effect only after the VM restarts.

• true (default): The virtio bus is used for disks, network interfaces, and other devices, which gives better performance. You can change the contents of .spec.blockDeviceRefs on a running VM without a restart, adding and removing devices, if the disk is available on the node where the VM runs.
• false: Standard device emulation is used (SATA for disks, e1000 for network interfaces; IDE and RTL8139 for the Legacy OS type), which may be required for compatibility with older operating systems that lack VirtIO drivers. Changes to .spec.blockDeviceRefs on a running VM (adding and removing disks and images, including ISO) take effect after the VM restarts. To attach and detach disks without a restart, use the VirtualMachineBlockDeviceAttachment (vmbda) resource, without changing the list in the VM specification.

For the Legacy OS type, the default value true isn’t a fit, because these operating systems have no built-in virtio drivers, so for a VM you’re going to install from the original media, set enableParavirtualization: false, otherwise the installer reports that it found no hard drives. A warning is issued when creating and modifying such a VM.

Keep enableParavirtualization: true for a VM with the Legacy OS type only when the virtio drivers are already installed in the guest OS. Then the VM keeps the i440fx chipset and the BIOS bootloader, but gets disks on virtio-blk, the virtio-net adapter, and loses the four-device limit. The CD-ROM stays on the IDE bus, because virtio-blk has no drive.

To switch a system that’s already installed:

1. Install the virtio drivers in the guest OS. For Windows XP, 2000, and Server 2003, take an archived virtio-win release, because the current releases no longer contain drivers for these systems, and install viostor, the virtio-blk driver. The package has no virtio-scsi driver for these operating systems.
2. Power off the VM.
3. Set enableParavirtualization: true.
4. Start the VM.

The order of the steps matters. The disk controller driver has to appear in the guest OS before the switch, not after, because the VM boots from that very controller. If the VM doesn’t boot after the switch, set enableParavirtualization: false back and restart it. The disks return to the IDE bus and the guest OS boots as before.

Example configuration with paravirtualization disabled:

spec:
  enableParavirtualization: false
  # other parameters...

VM initialization scripts

Initialization scripts are designed for the initial configuration of a virtual machine when it starts.

The following initialization scripts are supported:

• Cloud-Init.
• Sysprep.

Cloud-Init

Cloud-Init is a tool for automatically configuring virtual machines at first boot. It performs a wide range of configuration tasks without manual intervention.

Key Cloud-Init capabilities:

• creating users, setting passwords, adding SSH keys for access;
• automatically installing the required software at first boot;
• running arbitrary commands and scripts to configure the system;
• automatically starting and enabling system services (for example, qemu-guest-agent).

Here are the typical scenarios.

1. Adding an SSH key for a preinstalled user that may already be present in a cloud image (for example, the ubuntu user in official Ubuntu images). The name of such a user depends on the image. Check it in the documentation for your distribution.

   #cloud-config
   ssh_authorized_keys:
     - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD... your-public-key ...

2. Creating a user with a password and an SSH key:

   #cloud-config
   users:
     - name: cloud
       passwd: <PASSWORD_HASH>
       lock_passwd: false
       sudo: ALL=(ALL) NOPASSWD:ALL
       shell: /bin/bash
       ssh-authorized-keys:
         - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD... your-public-key ...
   ssh_pwauth: True

   Where <PASSWORD_HASH> is the password hash in quotes, generated with the mkpasswd --method=SHA-512 --rounds=4096 command.

3. Installing packages and services:

   #cloud-config
   package_update: true
   packages:
     - nginx
     - qemu-guest-agent
   runcmd:
     - systemctl daemon-reload
     - systemctl enable --now nginx.service
     - systemctl enable --now qemu-guest-agent.service

The following example shows how to pass the script to a virtual machine:

spec:
  provisioning:
    type: UserData
    userData: |
      #cloud-config
      package_update: true
      ...

apiVersion: v1
kind: Secret
metadata:
  name: cloud-init-example
data:
  userData: <base64 data>
type: provisioning.virtualization.deckhouse.io/cloud-init

spec:
  provisioning:
    type: UserDataRef
    userDataRef:
      kind: Secret
      name: cloud-init-example

Sysprep

To configure virtual machines running Windows with Sysprep, only the Secret resource option is supported.

Here is an example of a Secret resource with a Sysprep script:

apiVersion: v1
kind: Secret
metadata:
  name: sysprep-example
data:
  unattend.xml: <base64 data>
type: provisioning.virtualization.deckhouse.io/sysprep

A fragment of the virtual machine configuration that uses a Sysprep initialization script in a Secret resource:

spec:
  provisioning:
    type: SysprepRef
    sysprepRef:
      kind: Secret
      name: sysprep-example

Guest OS agent

Install QEMU Guest Agent in the guest system so that DP can interact with the operating system inside the VM. The agent is needed for three things:

• it makes consistent disk and VM snapshots possible;
• it reports information about the running system, and that information lands in the .status.guestOSInfo block;
• it shows that the operating system has actually booted, rather than just the virtual machine having started.

DP works with qemu-guest-agent version 5.2.0 and later. To check the installed version, run the following command:

qemu-guest-agent --version

Guest system information looks like this:

status:
  guestOSInfo:
    id: fedora
    kernelRelease: 6.11.4-301.fc41.x86_64
    kernelVersion: "#1 SMP PREEMPT_DYNAMIC Sun Oct 20 15:02:33 UTC 2024"
    machine: x86_64
    name: Fedora Linux
    prettyName: Fedora Linux 41 (Cloud Edition)
    version: 41 (Cloud Edition)
    versionId: "41"

The AGENT column shows whether the agent is running:

d8 k get vm -o wide

Example output:

NAME     PHASE     UPTIME   CORES   COREFRACTION   MEMORY   NEED RESTART   AGENT   MIGRATABLE   NODE           IPADDRESS    AGE
fedora   Running   5d21h    6       5%             8000Mi   False          True    True         virtlab-pt-1   10.66.10.1   5d21h

Install the agent with the command for your distribution and start the service:

# Debian and derivatives.
sudo apt install qemu-guest-agent

# CentOS and derivatives.
sudo yum install qemu-guest-agent

sudo systemctl enable --now qemu-guest-agent

For Linux, it’s convenient to automate the installation with an initialization script:

#cloud-config
package_update: true
packages:
  - qemu-guest-agent
runcmd:
  - systemctl enable --now qemu-guest-agent.service

The agent doesn’t need any configuration after installation. If your snapshots need application data consistency, put the preparation scripts in the /etc/qemu-ga/hooks.d/ directory on Debian and Ubuntu, or /etc/qemu/fsfreeze-hook.d/ on RHEL, CentOS, and Fedora. The scripts have to be executable, and the agent runs them before the file system freeze and after the thaw, so you don’t have to stop the application services.

Connecting to a virtual machine

You can connect to a virtual machine (VM) in four ways. The first is a remote management protocol such as SSH, which you configure in the guest OS yourself. The second is the serial console. The third is VNC. The fourth is SPICE, if it is enabled for the machine.

d8 v console linux-vm

Successfully connected to linux-vm console. The escape sequence is ^]
linux-vm login: cloud
Password: cloud

d8 v vnc linux-vm

d8 v spice linux-vm

d8 v ssh cloud@linux-vm

The serial console and VNC are exclusive, only one user works in them, and a new connection disconnects whoever is already working with the machine.

Before connecting, d8 v console and d8 v vnc report who took the stream and since when, and offer a choice:

The serial console of linux-vm is in use:
  user       serviceaccount default/alice
  connected  12 minutes ago (14:32), from the d8 v command line

Connect and disconnect them? [y] yes  [N] no  [w] wait until free:

The w answer means waiting until the other user disconnects and connecting automatically. Pressing Enter cancels the connection, because the safe option is selected by default. The --force flag connects without asking, and it’s also what you need for a non-interactive run in a script.

SPICE

SPICE is a second remote display protocol that, unlike VNC, brings the sound of the guest system, redirects USB devices from your computer into it, and shares a clipboard with it. It works alongside VNC, so open VNC sessions and the web interface keep working.

SPICE is disabled by default. To turn it on, set the .spec.spice.enabled parameter and restart the machine:

spec:
  spice:
    enabled: true

Together with SPICE, the machine gets a virtio-gpu video adapter. If the guest system has no driver for it, as in Windows 7 and Windows XP, set another adapter model with the virtualization.deckhouse.io/video annotation, using the vga, bochs, or ramfb value.

A shared clipboard, a resize to the client window, and a local cursor are added by the SPICE guest agent. Install it in the guest system, on Linux it’s the spice-vdagent package, on Windows it’s spice-guest-tools.

Connecting requires the remote-viewer client from the virt-viewer package, and the d8 v spice command opens it for you. If you don’t have such a client, run the proxy alone and connect with your own client to the port the command prints:

d8 v spice linux-vm --proxy-only

Startup policy and VM state management

The startup policy determines how DP maintains the machine state. It’s set by the .spec.runPolicy parameter:

• AlwaysOnUnlessStoppedManually: The default option. The machine always runs, and you can stop it only manually.
• AlwaysOn: The machine always runs, and even after a shutdown from the guest OS DP starts it again.
• Manual: You manage the machine state yourself.
• AlwaysOff: The machine is always off, and you can’t start it.

You can manage the machine state in two ways, by creating a VirtualMachineOperation resource or by using the d8 utility. The resource describes the action declaratively, and the utility creates the same resource for you.

The following example shows how to perform an operation on a virtual machine:

d8 v restart linux-vm

d8 k create -f - <<EOF
apiVersion: virtualization.deckhouse.io/v1alpha2
kind: VirtualMachineOperation
metadata:
  generateName: restart-linux-vm-
spec:
  virtualMachineName: linux-vm
  # Type of the operation to perform.
  type: Restart
EOF

d8 k get virtualmachineoperation

# Short form of the command.
d8 k get vmop

Only one operation runs at a time for a single machine. A new operation either supersedes the active one or fails, and which of the two happens depends on the pair of types:

A superseded operation moves to the Superseded phase. An operation that can’t supersede the active one moves to the Failed phase, so you have to create it again after the active one finishes. Restore and clone operations don’t supersede other operations.

Changing the VM configuration

You can change the configuration of a virtual machine (VM) at any time after creation. On a powered-off machine, the changes apply right away, and on a running one it depends on what exactly you changed.

The following example shows how to change the configuration of a virtual machine:

spec:
  disruptions:
    restartApprovalMode: Automatic

Changing the number of cores without a restart

You can change the number of cores of a running machine without rebooting it, if the change is applicable through live migration. Within the current CPU topology, you can both add and remove cores.

The feature is disabled by default. To enable it, an administrator adds HotplugCPUWithLiveMigration to the .spec.settings.featureGates parameter:

kind: ModuleConfig
metadata:
  name: virtualization
spec:
  settings:
    featureGates:
      - HotplugCPUWithLiveMigration

In the web interface, the same toggle is called Change CPU without reboot and is located in the Experimental features block on the System tab, in Deckhouse → Modules → virtualization → Configuration. Only a DP administrator has the rights for this.

When the feature is enabled and the new .spec.cpu.cores value stays within the current topology, DP applies the change by live migration. If the change requires a different topology, the machine has to be rebooted. The topology calculation rules are described in CPU topologies.

d8 k patch vm linux-vm --type merge -p '{"spec":{"cpu":{"cores":4}}}'

The guest OS doesn’t always bring new cores into service on its own, especially after a live migration. In Linux, a core is brought online through sysfs:

echo 1 > /sys/devices/system/cpu/cpu1/online

To make this happen automatically, add a udev rule:

cat <<'EOF' > /etc/udev/rules.d/99-hotplug-cpu.rules
SUBSYSTEM=="cpu",ACTION=="add",RUN+="/bin/sh -c '[ ! -e /sys$devpath/online ] || echo 1 > /sys$devpath/online'"
EOF

The cores brought into service appear in the output of nproc, cat /proc/cpuinfo, and top.

When you reduce the number of cores within the current topology, the distribution of cores across sockets is preserved.

Changing the amount of memory without a restart

You can increase the amount of memory of a running machine without rebooting it. Reducing it requires a restart.

The feature is disabled by default. To enable it, an administrator adds HotplugMemoryWithLiveMigration to the .spec.settings.featureGates parameter:

kind: ModuleConfig
metadata:
  name: virtualization
spec:
  settings:
    featureGates:
      - HotplugMemoryWithLiveMigration

In the web interface, the toggle is called Change memory without reboot and is located in the same place, in the Experimental features block of the module settings.

When the feature is enabled, the new .spec.memory.size value is greater than the current one, and the machine allows migration, DP applies the change by live migration. A restart is required if the memory is reduced, if the original size is less than 1 GiB, or if the machine can’t be migrated. Without a restart, memory grows up to 256 GiB, the ceiling built into the machine configuration at first start.

d8 k patch vm linux-vm --type merge -p '{"spec":{"memory":{"size":"4Gi"}}}'

As with cores, the guest OS may not bring new memory blocks into service on its own. In Linux, a block is brought online through sysfs, and the device name is visible in the lsmem output or in the /sys/bus/memory/devices/ directory:

echo 1 > /sys/bus/memory/devices/memoryXXX/online

To make this happen automatically, add a udev rule:

cat <<'EOF' > /etc/udev/rules.d/99-hotplug-memory.rules
SUBSYSTEM=="memory",ACTION=="add",DEVPATH=="/devices/system/memory/memory[0-9]*", TEST=="state", ATTR{state}!="online", ATTR{state}="online"
EOF

Placing VMs on nodes

Four mechanisms control where exactly a virtual machine (VM) starts:

• .spec.nodeSelector: The simplest way, it selects nodes with the required labels.
• .spec.affinity.nodeAffinity: Sets the preferred nodes for placement.
• .spec.affinity.virtualMachineAndPodAffinity: Places the machine next to other machines and workloads.
• .spec.affinity.virtualMachineAndPodAntiAffinity: On the contrary, spreads them across different nodes.

Conditions can be hard or soft. A hard requiredDuringSchedulingIgnoredDuringExecution condition is mandatory, and the machine doesn’t start if there’s no suitable node. A soft preferredDuringSchedulingIgnoredDuringExecution condition is taken into account by the scheduler where possible.

All rules, including .spec.nodeSelector from the VM class, apply together. If at least one hard condition can’t be met, the machine stays in the Pending phase. So set consistent rules, prefer combinations of labels over single hard restrictions, and keep spare nodes for critical workloads. Also consider the startup order: if one machine has to end up next to another, the second one has to start first. If the nodes you need have taints, add the matching tolerations to the machine.

To set the placement rules in the web interface:

1. Go to the Projects tab and select the project you need.
2. Go to Virtualization → Virtual machines.
3. Select the VM you need from the list and click its name.
4. On the Configuration tab, scroll down to the VM placement toggle.
5. Select the input mode. In the Basic setup mode, the rules are set with the Co-location and Separate placement toggles, and in the Editing mode, the placement block is set manually as YAML.
6. Enable the toggle you need and fill in the fields. The Select rule mode field offers Required (requiredDuringSchedulingIgnoredDuringExecution) and Preferred (preferredDuringSchedulingIgnoredDuringExecution), and the Placement rule field offers placement relative to other VMs or relative to node labels.
7. Click the Save button that appears.

Tolerance to node restrictions

Tolerations let a VM start on nodes with restrictions (taints) that otherwise block scheduling. This is useful when you need to run VMs on special nodes (for example, test nodes) or nodes with certain characteristics.

Here is an example of using tolerations to allow a start on nodes with the node.deckhouse.io/group=:NoSchedule taint:

spec:
  tolerations:
    - key: "node.deckhouse.io/group"
      operator: "Exists"
      effect: "NoSchedule"

Each element of the tolerations list has to match a taint on the node for the VM to be placed on that node.

To view the taints on cluster nodes, run the following command:

d8 k get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

To view detailed information about a node, run the following command:

d8 k describe node <NODE_NAME>

Simple label binding (nodeSelector)

nodeSelector is the simplest way to control the placement of virtual machines using a set of labels. It lets you specify which nodes virtual machines can start on by selecting nodes with the required labels.

spec:
  nodeSelector:
    disktype: ssd

In this example, the cluster has three nodes, two of them with fast disks (disktype=ssd) and one with slow ones (disktype=hdd). The virtual machine is placed only on nodes that have the disktype label with the ssd value.

To perform the operation in the web interface in the placement section:

1. Enable the Co-location toggle.
2. In the Select rule mode field, select Required.
3. In the Placement rule field, select On selected nodes.
4. In the How to identify the node group field, select By labels and specify the node labels (for example, disktype: ssd); the By name option lets you select specific nodes.
5. Click the Save button that appears.

Preferred binding (Affinity)

Affinity provides more flexible and powerful tools compared to nodeSelector. It lets you set “preferences” and “requirements” for the placement of virtual machines. Affinity supports two kinds: nodeAffinity and virtualMachineAndPodAffinity.

nodeAffinity defines the nodes to run a VM on using label selector expressions.

Here is an example of using nodeAffinity with a hard rule:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: disktype
                operator: In
                values:
                  - ssd

In this example, the cluster has three nodes, two of them with fast disks (disktype=ssd) and one with slow ones (disktype=hdd). The virtual machine is placed only on nodes that have the disktype label with the ssd value.

If you use a soft requirement (preferredDuringSchedulingIgnoredDuringExecution), then when there are no resources to run the VM on nodes with disktype=ssd disks, it’s scheduled on a node with disktype=hdd disks.

virtualMachineAndPodAffinity controls the placement of virtual machines relative to other virtual machines. It lets you set a preference for placing virtual machines on the same nodes where certain virtual machines are already running.

Here is an example of a soft rule:

spec:
  affinity:
    virtualMachineAndPodAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          podAffinityTerm:
            labelSelector:
              matchLabels:
                server: database
            topologyKey: "kubernetes.io/hostname"

In this example, the virtual machine is placed only on nodes that already run a virtual machine with the server: database label. The rule is soft (preferred), so if there are no such nodes, the machine starts on any suitable one.

To place VMs across availability zones (instead of pinning them to specific nodes), set topologyKey: topology.kubernetes.io/zone (Placing VMs across availability zones).

To set “preferences” and “requirements” for the placement of virtual machines in the web interface, in the placement section:

1. Enable the Co-location toggle, which corresponds to the spec.affinity.virtualMachineAndPodAffinity settings.
2. In the Select rule mode field, select Required or Preferred.
3. In the Placement rule field, select On nodes with selected VMs.
4. In the Select labels field, select the labels of the VMs you need from the list or enter your own in the key: value format.
5. Click the Save button that appears.

Avoiding co-location (AntiAffinity)

AntiAffinity is used to prevent VMs from being placed together on nodes. It’s useful for fault tolerance or load balancing.

The terms Affinity and AntiAffinity describe the relationships between virtual machines. There’s no such antonym for nodes, but you can achieve the same result through nodeAffinity with the NotIn operator, excluding the nodes you need.

Here is an example of using virtualMachineAndPodAntiAffinity:

spec:
  affinity:
    virtualMachineAndPodAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              server: database
          topologyKey: "kubernetes.io/hostname"

In this example, the virtual machine being created isn’t placed on the same node as a virtual machine with the server: database label.

To place VMs across availability zones (instead of pinning them to specific nodes), set topologyKey: topology.kubernetes.io/zone (Placing VMs across availability zones).

To configure the prevention of co-locating VMs on nodes in the web interface, in the placement section:

1. Enable the Separate placement toggle, which corresponds to the spec.affinity.virtualMachineAndPodAntiAffinity settings.
2. In the Select rule mode field, select Required or Preferred.
3. In the Placement rule field, select On nodes with selected VMs.
4. In the Select labels field, select the labels of the VMs you don’t want the machine placed next to, or enter your own label in the key: value format.
5. Click the Save button that appears.

Placing VMs across availability zones

Placement rules work not only at the node level, but also at the availability zone level.

The examples above use topologyKey: "kubernetes.io/hostname", which places VMs on the same node. To place VMs across availability zones instead of nodes, use topologyKey: "topology.kubernetes.io/zone".

With Affinity and topologyKey: "topology.kubernetes.io/zone", VMs are placed in the same availability zone where a virtual machine with the specified labels is present.

With AntiAffinity and topologyKey: "topology.kubernetes.io/zone", VMs aren’t placed in the same availability zone as a virtual machine with the specified labels. This is useful for fault tolerance when distributing VMs across different availability zones.

To view the availability zones on cluster nodes (if those zones are set), run the following command:

d8 k get nodes -o custom-columns=NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone

Attaching block devices (disks and images)

You can attach disks and images to a virtual machine (VM). They’re described as block devices (BlockDevices).

Block device types and access modes:

There are two ways to attach devices:

• Through the VM specification (.spec.blockDeviceRefs): The disks are listed in the VirtualMachine configuration, and the boot order is set for them (by position in the list or through the bootOrder field). Recommended when configuring a VM manually, and when you need control over the boot order (for example, an ISO for OS installation).
• Through VirtualMachineBlockDeviceAttachment (vmbda): The disk is attached as a separate resource and doesn’t take part in the boot order. Disks are attached through the virtio-scsi bus, regardless of the enableParavirtualization value. Recommended for automation and when you don’t have the rights to edit the VM.

With enableParavirtualization: true, both ways let you attach and detach disks on a running VM without a reboot, if the disk is available on the node where it runs. With enableParavirtualization: false, the contents of .spec.blockDeviceRefs on a running VM change only after a reboot; to attach and detach disks without a reboot, use VirtualMachineBlockDeviceAttachment (vmbda).

You can attach a disk to a running VM only when the storage is available on the cluster node where the virtual machine runs. When creating and updating a VM, and when creating a VirtualMachineBlockDeviceAttachment, the placement rules (nodeSelector, affinity, tolerations) of the volume, the virtual machine, and the VM class are taken into account, and they have to share at least one valid placement. If the VM is already running on a specific node, the new disk has to be available on that node.

While a live migration of the machine is preparing the target node, a disk can be neither attached nor detached. A new VirtualMachineBlockDeviceAttachment resource stays in the Pending phase with the BlockedByMigration reason in the Attached condition, and a deleted one stays in the Terminating phase, and both finish once the migration completes. While the migration is still queued and the target node isn’t being prepared yet, attaching and detaching work as usual.

Attaching through the VM specification

The devices listed in the machine specification are attached at startup and stay in place for the whole run. Their list is set in the .spec.blockDeviceRefs field of the VirtualMachine resource.

By default, the boot order matches the order of the devices in the list, and the optional bootOrder field lets you set it explicitly (a lower value means a higher priority). If bootOrder is specified for at least one device, only the devices with a set bootOrder get into the boot chain. Integers from 1 and up are allowed, unique within the list. When a device is removed from the list, the boot order is recalculated for the remaining devices.

A change to the order of devices in the list or to the bootOrder values takes effect after the VM reboots. For example, you can attach an ISO image for OS installation with the boot priority you need, and remove it from the list after the installation. If the VM has paravirtualization disabled (enableParavirtualization: false), edits to .spec.blockDeviceRefs on a running VM, including ones with an ISO image, apply after the VM reboots.

spec:
  blockDeviceRefs:
    - kind: VirtualDisk
      name: <VD_NAME>
      bootOrder: 1
    - kind: VirtualImage
      name: <VI_NAME>
      bootOrder: 2

spec:
  blockDeviceRefs:
    - kind: VirtualDisk
      name: <VD_NAME>
    - kind: VirtualImage
      name: <VI_NAME>
    - kind: VirtualDisk
      name: <ADDITIONAL_DISK_NAME>

Attaching through VirtualMachineBlockDeviceAttachment

The VirtualMachineBlockDeviceAttachment resource attaches and detaches a block device on a VM without touching its specification. It suits automation and the cases where the user doesn’t have the rights to edit the machine.

The device is attached when the resource moves to the Attached phase. The other phases are described in the .status.phase field, and the .status.conditions block shows the reason for a delay.

d8 k apply -f - <<EOF
apiVersion: virtualization.deckhouse.io/v1alpha2
kind: VirtualMachineBlockDeviceAttachment
metadata:
  name: attach-blank-disk
spec:
  blockDeviceRef:
    kind: VirtualDisk
    name: blank-disk
  virtualMachineName: linux-vm
EOF

d8 k get vmbda attach-blank-disk

NAME                PHASE      VIRTUALMACHINE   AGE
attach-blank-disk   Attached   linux-vm         3m7s

d8 v ssh cloud@linux-vm --command "lsblk"

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda       8:0    0   10G  0 disk <--- statically attached linux-vm-root disk
|-sda1    8:1    0  9.9G  0 part /
|-sda14   8:14   0    4M  0 part
`-sda15   8:15   0  106M  0 part /boot/efi
sdb       8:16   0    1M  0 disk <--- cloudinit
sdc       8:32   0 95.9M  0 disk <--- dynamically attached blank-disk disk

d8 k delete vmbda attach-blank-disk

d8 k apply -f - <<EOF
apiVersion: virtualization.deckhouse.io/v1alpha2
kind: VirtualMachineBlockDeviceAttachment
metadata:
  name: attach-ubuntu-iso
spec:
  blockDeviceRef:
    kind: VirtualImage # or ClusterVirtualImage
    name: ubuntu-iso
  virtualMachineName: linux-vm
EOF

Disk naming in the guest OS

Disk names in the guest system aren’t stable, so relying on them in configuration is risky.

Example:

After the first VM boot:

$ lsscsi
[0:0:0:1]  disk    QEMU     QEMU HARDDISK   /dev/sda
[0:0:0:2]  disk    QEMU     QEMU HARDDISK   /dev/sdb

After a VM reboot:

$ lsscsi
[0:0:0:1]  disk    QEMU     QEMU HARDDISK   /dev/sdb
[0:0:0:2]  disk    QEMU     QEMU HARDDISK   /dev/sda

The SCSI addresses (0:0:0:1, 0:0:0:2) stay the same, but the device names (/dev/sda, /dev/sdb) swap places.

Use stable identifiers instead of /dev/sdX:

• /dev/disk/by-uuid/: By partition UUID (preferable for /etc/fstab).
• /dev/disk/by-path/: By the SCSI connection path.
• /dev/disk/by-id/: By the SCSI device ID.

In configuration files and scripts, use partition UUIDs or symbolic links from /dev/disk/by-* instead of /dev/sdX names.

Network interface naming in the guest OS

In systems without predictable network interface naming, network interface names (eth0, eth1, eth2, and so on) are assigned by the Linux kernel in the order the devices are discovered at boot. When you add new network interfaces or change the order of networks in .spec.networks, the interface order can change, and IP addresses can end up assigned to the wrong interfaces.

Using ethX in configuration files (for example, /etc/network/interfaces, netplan, systemd-networkd) or in scripts when adding new interfaces or changing the order of networks can lead to network failures or a connection to the wrong network.

Modern distributions with systemd (Ubuntu 16.04+, Debian 9+, CentOS 7+, RHEL 7+) use predictable interface names by default (enpXsY, ensX, enoX), which are based on the physical characteristics of the device (PCI coordinates) and stay stable between reboots and when new interfaces are added.

But even with predictable names, bind the network configuration to the MAC addresses of the interfaces, especially if the order of networks in .spec.networks changes or new interfaces are added.

An example for systems without predictable naming:

Initially, the VM has two interfaces:

$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500

After adding a new interface at the beginning of the .spec.networks list and rebooting the VM:

$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500  # New interface
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500  # Former eth0
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500  # Former eth1

The MAC addresses stay the same, but the interface names (eth0, eth1) shift, which can lead to IP addresses being assigned to the wrong interfaces.

Use stable identifiers instead of ethX:

• enpXsY: Predictable names based on the physical location (systemd networkd naming scheme, enabled by default in modern systems).
• Binding by MAC address: In the netplan, systemd-networkd, or /etc/network/interfaces configuration (preferable for guaranteed stability).

In configuration files and scripts, use stable interface names (enpXsY) or binding by MAC address instead of ethX names.

To open a machine application to other machines or to users outside the cluster, configure a service or Ingress as described in Accessing applications on a virtual machine.

Live VM migration

Live migration of virtual machines (VMs) is the process of moving a running VM from one physical node to another without shutting it down. This feature plays a key role in managing virtualized infrastructure, keeping applications running during maintenance, load balancing, or updates.

How live migration works

The live migration process consists of several stages:

1. A new VM is created on the target node in a paused state. Its configuration (CPU, disks, network) is copied from the source node.

2. All the RAM of the VM is copied to the target node over the network. This is called the initial transfer.

3. While the memory is being transferred, the VM keeps running on the source node and can modify some memory pages. Such pages are called dirty pages, and the hypervisor marks them.

4. After the initial transfer, only the modified pages are sent again. This process repeats in several cycles:

   • The higher the load on the VM, the more dirty pages appear, and the longer the migration takes.
   • With good network bandwidth, the amount of unsynchronized data gradually decreases.

5. When the number of dirty pages becomes minimal, the VM on the source node is paused (usually for 100 milliseconds):

   • The remaining memory changes are transferred to the target node.
   • The state of the CPU, devices, and open connections is synchronized.
   • The VM starts on the new node, and the original copy is deleted.

Until the VM switches to the new node (step 5), the VM on the source node keeps running as usual and serving users.

Requirements and limitations

A live migration doesn’t always succeed. The following is what has to match on the source and target nodes.

Disk availability. All disks attached to the VM have to be available on the target node. With network storage such as NFS or Ceph, this requirement is met on its own, because the disks are visible from all cluster nodes. Local storage needs to be able to create a new local volume on the target node, and if such storage exists only on the source node, the migration doesn’t run.

Attaching and detaching disks. While a migration is preparing the target node, disks can be neither attached to the machine with a VirtualMachineBlockDeviceAttachment resource nor detached by deleting one. An attachment stays in the Pending phase with the BlockedByMigration reason in the Attached condition, and a deleted one stays in the Terminating phase, until the migration completes. While the migration is still queued and the target node isn’t being prepared yet, for example when it waits for the project quota to free up, attaching and detaching work as usual. The reverse is also true, a migration waits for an attach or detach request that has already been sent, and all that time the VirtualMachineOperation resource stays in the Pending phase with the WaitingForBlockDeviceAttachment reason. If the request doesn’t complete within 5 minutes, the operation fails.

Network bandwidth. The slower the network, the more memory synchronization iterations the migration goes through and the longer the VM downtime at the final stage, and in the worst case the migration doesn’t fit into the timeout. The migration policy controls how the migration runs, and the AutoConverge mechanism helps with a slow network.

Kernel versions. All cluster nodes have to run the same Linux kernel version. Differences in versions lead to incompatible interfaces, system calls, and resource handling, which breaks the migration.

CPU compatibility. The CPU type in the virtual machine class sets the CPU requirements. The Host type allows migration only between nodes with similar CPUs, so it works neither between Intel and AMD nor between different CPU generations with different instruction sets. The HostPassthrough type requires exactly the same CPU on the target node as on the source one. To let a machine migrate between nodes with different CPUs, set the Discovery, Model, or Features type in the class.

Duration. A migration has a completion timeout of 800 seconds per gibibyte of VM memory, plus 800 seconds per gibibyte of disk when the disks move along with it. For example, a machine with 4 GiB of memory and a 20 GiB disk gets 800 × (4 + 20) = 19200 seconds, or about 5.3 hours. A migration that doesn’t fit into this time is considered failed and is canceled, which happens with a slow network or a high load on the VM.

Checking whether a VM is ready for migration

The type: Migratable condition in the VM status shows whether the VM can be moved by live migration. It takes into account both the VM itself (disks, passed-through devices, CPU type) and the state of the cluster (whether there’s a node to move it to). The True value answers the question of whether the move is possible, not whether the VM will move at this very moment, so look at the reason along with the value.

The overall picture for all VMs:

d8 k get vm -o wide

The value in the MIGRATABLE column shows the result, and the condition describes the reason:

d8 k get vm <VM_NAME> -o json | jq '.status.conditions[] | select(.type=="Migratable")'

The most common reasons:

Moving disks along with a VM is available in commercial DP editions, so the VirtualMachineDisksShouldBeMigrating reason appears only there. In DP Open, a VM with disks in storage available from one node gets the VirtualMachineDisksNotMigratable reason.

Specifics of the Migratable condition

A few specifics of the condition that matter when planning maintenance and reading the status:

• The Migratable condition describes not only the VM but the cluster itself. If you remove the required label from the only suitable node without changing the VM parameters, the condition still becomes False. When a suitable node appears, the condition returns to True.

• When a node is made unschedulable, the VM stays capable of migrating. Cordoning, rebooting, and node maintenance happen on their own, so the condition stays True and only the reason changes to VirtualMachineWaitingForMigrationTarget. Otherwise, planned maintenance of a neighboring node would turn a CPU or memory change into a VM restart. In DP Open, such changes require a reboot in any case.

• The True value means that the VM is capable of migrating, not that it’s ready to migrate right now. Before a migration, look at the reason. The VirtualMachineMigratable reason means there’s a node to migrate to, and VirtualMachineWaitingForMigrationTarget means there’s no suitable node at this moment. The d8_virtualization_virtualmachine_migratable metric carries the same answer in the reason label, so a dashboard that filters VMs only by value counts waiting VMs together with those ready to migrate.

• A migration started with the VirtualMachineWaitingForMigrationTarget reason doesn’t wait for a node indefinitely. If the target pod can’t be scheduled within five minutes, the operation fails, and the Completed condition of the VirtualMachineOperation resource gets the TargetUnschedulable reason. If maintenance drags on, restart the migration.

• A stopped VM has no such condition, because the capability to migrate is computed only for a running VM. During the downtime, the disks could have moved to different storage and a device could have been detached.

• In DP Open, placement changes are taken into account after the VM restarts. While the VM runs, it uses the parameters it was started with, and the condition describes exactly those. New nodeSelector, affinity, or VM class values get into the calculation only after a restart. In commercial DP editions, such changes apply without a restart and get into the condition calculation right away.

• Local disks don’t prevent migration, but a node is still needed. In commercial DP editions, a VM with local disks moves along with them, so the condition stays True with the VirtualMachineDisksShouldBeMigrating reason. But if no cluster node matches its placement rules, the condition is False, because there’s nowhere to move the disks along with the VM.

Starting a live migration

A migration is started by the Migrate operation of the VirtualMachineOperation resource, which you create manually or with a d8 command. You can interrupt the migration while it’s in the Pending or InProgress phase by deleting that resource.

d8 k get vm

NAME       PHASE     UPTIME   NODE           IPADDRESS     AGE
linux-vm   Running   79m      virtlab-pt-1   10.66.10.14   79m

d8 v migrate -n <NAMESPACE> <VM_NAME> [--force] [--target-node-name string]

d8 v migrate -n project-1 linux-vm --target-node-name production-1

d8 k create -f - <<EOF
apiVersion: virtualization.deckhouse.io/v1alpha2
kind: VirtualMachineOperation
metadata:
  generateName: migrate-linux-vm-
  namespace: project-1
spec:
  # Virtual machine name.
  virtualMachineName: linux-vm
  # Operation for migration.
  type: Migrate
  # Defines the virtual machine migration operation.
  migrate:
    nodeSelector:
      # You can also set any suitable node selector.
      kubernetes.io/hostname: production-1
  # Allow CPU throttling by the AutoConverge mechanism to guarantee that the migration completes.
  force: true
EOF

d8 k get vm -w

NAME       PHASE       UPTIME   NODE           IPADDRESS     AGE
linux-vm   Running     79m      virtlab-pt-1   10.66.10.14   79m
linux-vm   Migrating   79m      virtlab-pt-1   10.66.10.14   79m
linux-vm   Migrating   79m      virtlab-pt-1   10.66.10.14   79m
linux-vm   Running     79m      virtlab-pt-2   10.66.10.14   79m

Configuring the migration policy

The migration policy determines when to use the AutoConverge mechanism (CPU throttling) to guarantee that a migration completes.

The AutoConverge mechanism helps a migration complete even with low network bandwidth, which makes a successful operation highly likely. However, it throttles the virtual machine CPU, which can affect the performance of applications running on the virtual machine.

The AutoConverge mechanism works in two stages:

1. Throttling the virtual machine CPU

   The hypervisor gradually lowers the CPU frequency of the source virtual machine. This reduces the rate at which new dirty pages appear. The higher the load on the virtual machine, the stronger the throttling.

2. Automatic migration completion

   As soon as the data transfer rate exceeds the memory change rate, the final synchronization starts and the virtual machine switches to the new node.

To configure the migration policy, use the .spec.liveMigrationPolicy parameter in the virtual machine configuration. Allowed values:

• AlwaysSafe: The migration always runs without CPU throttling (AutoConverge isn’t used). Suitable when maximum virtual machine performance matters, but it requires high network bandwidth.
• PreferSafe (used as the default policy): The migration runs without CPU throttling (AutoConverge isn’t used). However, you can start a migration with CPU throttling using a VirtualMachineOperation resource with the type=Migrate and force=true parameters.
• AlwaysForced: The migration always uses AutoConverge, that is, the CPU is throttled when required. This guarantees that the migration completes even on a poor network, but it can reduce virtual machine performance.
• PreferForced: The migration uses AutoConverge, that is, the CPU is throttled when required. However, you can start a migration without CPU throttling using a VirtualMachineOperation resource with the type=Migrate and force=false parameters.

Migrations with insufficient network bandwidth

During a live migration of a virtual machine, the network bandwidth may not be enough to transfer data faster than it changes in the virtual machine memory. In that case, the number of dirty pages keeps growing and the migration may not complete within the timeout.

The AutoConverge mechanism, configured through the migration policy, solves this problem.

To tell that the network bandwidth isn’t enough for a live migration of a virtual machine, check the charts in Namespace / Virtual Machine → VM Status details → Live migration memory metrics:

• Processed memory rate is lower than Dirty memory rate;
• Remaining memory rate doesn’t decrease for a long time.

This means the network has become the bottleneck for the migration.

Here is an example of a situation where the migration can’t complete because of insufficient network bandwidth. Inside the virtual machine, memory changes continuously with stress-ng.

Here is an example of migrating the same virtual machine with the --force flag of the d8 v migrate command (which enables the AutoConverge mechanism). You can clearly see that the CPU frequency is lowered in stages to reduce the rate of memory content changes.

If the network limits the migration speed, you can do the following:

1. Wait until the operation fails because of a timeout.

2. Cancel the current migration operation by deleting the VirtualMachineOperation resource, where <VMOP_NAME> is the name of that resource:

   d8 k delete vmop <VMOP_NAME>

3. Restart the migration with the --force flag to enable the AutoConverge mechanism. Using the --force flag has to match the current virtual machine migration policy.

Migrations started by the system

DP starts some migrations itself, by creating a VirtualMachineOperation resource of the Evict type. The prefix of the resource name shows what caused such a migration:

The migration has completed successfully when the resource moves to the Completed phase. The other phases are described in the .status.phase field, and to cancel a migration you delete the resource.

The following example shows how to view the list of such operations:

d8 k get vmop

NAME                    PHASE       PROGRESS   TYPE    VIRTUALMACHINE   AGE
firmware-update-fnbk2   Completed   100%       Evict   linux-vm         1m

Live VM migration on a placement parameter change

When the placement rules of a running machine change, DP moves it to a suitable node with a live migration.

The following example shows the migration mechanism in a cluster with two node groups, green and blue. Suppose a virtual machine initially runs on a node of the green group, and its configuration has no placement restrictions.

First, add a requirement to be placed in the green group to the VM specification:

spec:
  nodeSelector:
    node.deckhouse.io/group: green

After you save the changes, the VM keeps running on the current node, because the nodeSelector condition is already met.

Now change the requirement to the blue group:

spec:
  nodeSelector:
    node.deckhouse.io/group: blue

The current node from the green group no longer meets the new conditions. DP creates a VirtualMachineOperation resource of the Evict type and starts a live migration of the VM to an available node of the blue group.

Example output:

NAME                         PHASE       PROGRESS   TYPE    VIRTUALMACHINE   AGE
nodeplacement-update-dabk4   Completed   100%       Evict   linux-vm         1m

Collecting debug information

If a machine doesn’t behave as expected, collect its state and the state of the related resources into a single archive.

The archive contains the following information:

• the virtual machine configuration;
• operations on the virtual machine;
• migration information;
• block devices;
• related PVCs and PVs;
• pods related to the VM, including their logs (the last 10000 lines);
• events for all related resources;
• the XML configuration of the VM domain.

The collected data is stored in the archive in YAML format (for resources) and in text files (for logs). You can send such an archive to technical support to analyze a problem.

The following example shows how to collect such an archive:

# Collect debug information for the 'linux-vm' virtual machine
d8 v collect-debug-info linux-vm > debug-info.tar.gz

# Collect debug information for a VM with the namespace specified
d8 v collect-debug-info linux-vm -n mynamespace > debug-info.tar.gz

# Collect debug information for a VM with the full name specified (name.namespace)
d8 v collect-debug-info linux-vm.mynamespace > debug-info.tar.gz

Virtual machine networking

Every virtual machine (VM) gets an address in the main cluster network and, if needed, connects to additional networks. This section describes how to manage the addresses of a machine, open access to its applications, and attach additional interfaces.

IP addresses of VMs

Two resources describe the address of a machine in the main cluster network, the cluster-wide address lease and the address reserved for the project.

The .spec.settings.virtualMachineCIDRs block in the module settings defines the subnets that machines get IP addresses from. All addresses of a subnet are available except the first and the last one.

The cluster-wide VirtualMachineIPAddressLease (vmipl) resource manages leases of IP addresses from the shared pool specified in virtualMachineCIDRs, while the project VirtualMachineIPAddress (vmip) resource reserves leased addresses and binds them to machines. An address is assigned to a machine when the resource moves to the Attached phase. The other phases are described in the .status.phase field.

By default, DP assigns an address to the machine itself and keeps it assigned until the machine is deleted. This works as follows:

• You create a virtual machine named <VM_NAME>.
• DP creates a VirtualMachineIPAddress resource named <VM_NAME>-<HASH> to request an IP address and bind it to the virtual machine.
• For this VirtualMachineIPAddress, a VirtualMachineIPAddressLease lease resource is created, which picks a random IP address from the shared pool.
• As soon as the VirtualMachineIPAddress resource is created, the virtual machine gets the assigned IP address.

After the machine is deleted, the VirtualMachineIPAddress resource is deleted too, but the address itself stays assigned to the project for a while, and you can request it again.

d8 k get vmipl

NAME             VIRTUALMACHINEIPADDRESS                             STATUS   AGE
ip-10-66-10-14   {"name":"linux-vm-7prpx","namespace":"default"}     Bound    12h

d8 k get vmip

NAME             ADDRESS       STATUS     VM         AGE
linux-vm-7prpx   10.66.10.14   Attached   linux-vm   12h

Assigning a specific IP address

Instead of a random address from the pool, you can give a machine an address you choose in advance.

Keeping an IP address in the project

To keep the automatically allocated IP address of a virtual machine from being deleted along with the virtual machine itself, do the following.

Get the name of the VirtualMachineIPAddress resource for the given virtual machine:

d8 k get vm linux-vm -o jsonpath="{.status.virtualMachineIPAddressName}"

Example output:

linux-vm-7prpx

Remove the .metadata.ownerReferences block from the resource you found:

d8 k patch vmip linux-vm-7prpx --type=merge --patch '{"metadata":{"ownerReferences":null}}'

# Or make the same changes by editing the resource.

d8 k edit vmip linux-vm-7prpx

After the virtual machine is deleted, the VirtualMachineIPAddress resource is preserved and you can reuse it in a newly created virtual machine:

spec:
  virtualMachineIPAddressName: linux-vm-7prpx

Even if the VirtualMachineIPAddress resource is deleted, the IP address stays leased to the current project or namespace for another 10 minutes. So you can claim it again on request:

d8 k apply -f - <<EOF
apiVersion: virtualization.deckhouse.io/v1alpha2
kind: VirtualMachineIPAddress
metadata:
  name: linux-vm-custom-ip
spec:
  staticIP: 10.66.20.77
  type: Static
EOF

Accessing applications on a virtual machine

You can reach a virtual machine (VM) directly by its IP address, but this approach has limitations. You have to know the address in advance, it can change when the machine is recreated, and you can’t reach a group of machines at once. Kubernetes services solve all of these tasks.

A service gives a machine or a group of machines a permanent name that hides their addresses, and distributes requests evenly among them. The name is formed as <SERVICE_NAME>.<NAMESPACE>.svc.<CLUSTER_NAME>, and within the same namespace the short form <SERVICE_NAME> is enough.

Which service type to choose depends on the task:

• Headless: Direct access to specific machines inside the cluster without a single entry point.
• ClusterIP: A single internal address with balancing between machines.
• NodePort: External access through a port on the cluster nodes.
• LoadBalancer: External access through an external load balancer.

A machine gets into a service by labels. Assign the machine the label that the service looks for:

d8 k label vm linux-vm app=nginx

virtualmachine.virtualization.deckhouse.io/linux-vm labeled

Headless service

A headless service doesn’t allocate an IP address of its own, but returns the addresses of the machines themselves. This way you reach a specific machine by its DNS name without setting up a separate entry point for it. Even for a single machine, this is more convenient than a fixed address, because the name doesn’t change when the machine is recreated.

d8 k apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: http
  namespace: default
spec:
  clusterIP: None
  selector:
    # The label the service uses to select virtual machines.
    app: nginx
EOF

Service of the ClusterIP type

A service of this type gives the machine application a stable address inside the cluster.

ClusterIP is the standard service type that provides an internal IP address for accessing the service inside the cluster. This IP address is used to route traffic between different system components and lets virtual machines interact with each other through a predictable and stable address.

d8 k apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: http
spec:
  selector:
    # The label the service uses to decide which virtual machine to route traffic to.
    app: nginx
EOF

Service of the NodePort type

A service of this type opens the machine application on a port of every cluster node.

NodePort is an extension of the ClusterIP service that provides access to the service through a specified port on all cluster nodes. This makes the service reachable from outside the cluster through the combination of a node IP address and a port, and it suits the cases where an external load balancer isn’t needed.

d8 k apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: linux-vm-nginx-nodeport
spec:
  type: NodePort
  selector:
    # The label the service uses to decide which virtual machine to route traffic to.
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 31880
EOF

Service of the LoadBalancer type

A service of this type gives the application an external address through a load balancer.

LoadBalancer is a service type that automatically creates an external load balancer with a permanent IP address. This balancer distributes incoming traffic among virtual machines, making the service available from the internet.

d8 k apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: linux-vm-nginx-lb
spec:
  type: LoadBalancer
  selector:
    # The label the service uses to decide which virtual machine to route traffic to
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
EOF

Publishing VM services with Ingress

Ingress opens the machine application by a domain name and handles TLS termination.

Ingress lets you manage incoming HTTP/HTTPS requests and route them to different servers within your cluster. This is the most suitable method if you want to use domain names and SSL termination to access your virtual machines.

d8 k apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: linux-vm-nginx
spec:
  selector:
    # the label the service uses to decide which virtual machine to route traffic to
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
EOF

d8 k apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: linux-vm
spec:
  rules:
    - host: linux-vm.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: linux-vm-nginx
                port:
                  number: 80
EOF

Additional network interfaces

Besides the main cluster network, a machine can connect to additional networks, both project ones (Network) and cluster ones (ClusterNetwork). You list the networks you need in the .spec.networks block, and if that block isn’t set, the machine works only in the main cluster network.

Specifics of additional network interfaces:

• the order of networks in .spec.networks determines the order in which interfaces are attached inside the virtual machine;
• adding or removing an additional network (Network or ClusterNetwork) on a running VM applies without a reboot. The ACPI indexes of existing interfaces are preserved when adding or removing, so interface names in the guest OS stay stable;
• adding or removing the main network (type: Main) still requires a VM reboot, because it’s bound to the main network interface of the pod and can’t be changed on a running pod;
• to preserve the order of network interfaces inside the guest operating system, add new networks to the end of the .spec.networks list and don’t change the order of existing ones;
• network security policies (NetworkPolicy) don’t apply to additional network interfaces;
• the network parameters (IP addresses, gateways, DNS, and so on) for additional networks are configured manually from inside the guest OS (for example, with Cloud-Init), unless IPAM is configured for the network, as covered in IPAM for additional network interfaces.

For each additional network interface, a unique MAC address is created and reserved automatically, which prevents MAC address collisions. The VirtualMachineMACAddress (vmmac) and VirtualMachineMACAddressLease (vmmacl) resources are used for this.

A MAC address is generated at random from a pool of allowed ranges.

• Ranges: x2-xx-xx-xx-xx-xx, x6-xx-xx-xx-xx-xx, xA-xx-xx-xx-xx-xx, xE-xx-xx-xx-xx-xx.
• The first three octets (OUI) are formed from the cluster UUID, and the last three (NIC) are picked at random from 16 million possible combinations.

The cluster-wide VirtualMachineMACAddressLease (vmmacl) resource manages leases of addresses from the shared pool, while the project VirtualMachineMACAddress (vmmac) resource reserves leased addresses and binds them to machines. An address stays assigned to the machine until it’s deleted.

When a network is removed from the VM configuration:

• The MAC address of the interface is released.
• The related VirtualMachineMACAddress and VirtualMachineMACAddressLease resources are deleted automatically.
• The allocated IPAddress resource is deleted automatically (if IPAM was used).

The following example shows how to connect a virtual machine to an additional network:

spec:
  networks:
    - type: Main # If specified, it has to be first
    - type: Network # Network type (Network \ ClusterNetwork)
      name: user-net # Network name

spec:
  networks:
    - type: Main # If specified, it has to be first
    - type: Network
      name: user-net
    - type: ClusterNetwork
      name: corp-net # Network name

spec:
  networks:
    - type: Network
      name: isolated-net
    - type: ClusterNetwork
      name: corp-net

status:
  networks:
    - type: Main
    - type: Network
      name: user-net
      macAddress: aa:bb:cc:dd:ee:01
    - type: ClusterNetwork
      name: corp-net
      macAddress: aa:bb:cc:dd:ee:02

d8 k get vmmacl

NAME                    VIRTUALMACHINEMACADDRESS                      STATUS   AGE
mac-5e-e6-19-22-0f-d8   {"name":"vm-01-fz9cr","namespace":"pr-sdn"}   Bound    45s
mac-5e-e6-19-29-89-cf   {"name":"vm-01-99qj6","namespace":"pr-sdn"}   Bound    45s
mac-5e-e6-19-54-f9-be   {"name":"vm-01-5jqxg","namespace":"pr-sdn"}   Bound    45s

d8 k get vmmac

NAME          ADDRESS             STATUS     VM      AGE
vm-01-5jqxg   5e:e6:19:54:f9:be   Attached   vm-01   5m42s
vm-01-99qj6   5e:e6:19:29:89:cf   Attached   vm-01   5m42s
vm-01-fz9cr   5e:e6:19:22:0f:d8   Attached   vm-01   5m42s

Interface naming and ARP flux in the guest OS

When configuring network interfaces in the guest OS, use stable identifiers (predictable enpXsY names or binding by MAC address) instead of ethX names, as described in Network interface naming in the guest OS.

On a Linux guest system with several interfaces in the same subnet, the ARP flux problem can occur, where the kernel answers ARP requests through an arbitrary interface rather than the one the request arrived on, which leads to an unstable connection and packet loss because of an incorrect MAC address in the router caches.

To fix this, set the parameters that make the system answer requests strictly through the interface with the target IP and use the correct source address:

sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2

Example for cloud-init:

write_files:
- path: /etc/sysctl.d/90-arp-strict.conf
content: |
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

The parameter values are described in the IP sysctl documentation.

IPAM for additional network interfaces

DP can hand out addresses in an additional network itself, if an administrator has configured an address pool for that network.

If IPAM is configured for an additional network in the sdn module (an IP address pool bound to the network through spec.ipam.ipAddressPoolRef), the virtualization module can automatically allocate IP addresses for the additional VM interfaces and deliver them to the guest OS over DHCP.

Two modes are supported:

• Automatic (DHCP): If the ipAddressName field isn’t specified in .spec.networks[], the controller automatically creates an IPAddress resource (of the Auto type) bound to the VM through ownerReferences, and passes it to the sdn module. The sdn module allocates an address from the pool and delivers it to the guest OS over DHCP. The address is preserved across VM reboots and migrations, because it’s bound to the VM rather than to the pod. For this mode to work, the DHCP client has to be enabled on the corresponding interface in the guest OS.

• Static: If the ipAddressName field is specified in .spec.networks[], the controller uses the IPAddress resource provided by the user (of the Static type, network.deckhouse.io/v1alpha1). The address is defined by the user and doesn’t change automatically.

If an additional network has no IPAM pool configured, the IPAM feature isn’t enabled. The interface works in L2-only mode, and IP addressing has to be configured manually in the guest OS.

spec:
  networks:
    - type: Main
    - type: ClusterNetwork
      name: corp-net
      # ipAddressName isn't specified → automatic mode (DHCP) is used

spec:
  networks:
    - type: Main
    - type: ClusterNetwork
      name: corp-net
      ipAddressName: my-static-ip # Name of the IPAddress resource (SDN)

apiVersion: network.deckhouse.io/v1alpha1
kind: IPAddress
metadata:
  name: my-static-ip
  namespace: my-namespace
spec:
  networkRef:
    kind: ClusterNetwork
    name: corp-net
  type: Static
  static:
    ip: 192.168.200.42

status:
  ipAddress: 10.66.10.2                     # IP address of the main network.
  virtualMachineIPAddressName: vm-01-main-ip # IPAddress name of the main network.
  networks:
    - type: Main
    - type: ClusterNetwork
      name: corp-net
      macAddress: 32:a6:a1:0a:92:48
      virtualMachineMACAddressName: vm-01-rxzd6
      ipAddress: 192.168.200.4               # IP address of the additional network (from IPAM).

Configuring the guest OS for interfaces added on the fly

When an additional network interface is attached to an already running VM, the guest OS has to be configured to bring new network interfaces up automatically and request a DHCP lease. By default, Linux doesn’t start a DHCP client on interfaces added on the fly.

To make such interfaces configure themselves, use one of the following approaches in the guest OS:

• NetworkManager (Ubuntu, RHEL, CentOS): Configures new interfaces with DHCP automatically, if the network-manager service is running.

• A udev rule (Alpine and other systems without network-manager): Add a udev rule to bring new interfaces up:

  write_files:
    - path: /etc/udev/rules.d/90-hotplug-network.rules
      content: |
        SUBSYSTEM=="net", ACTION=="add", RUN+="/sbin/ifup %k"

Interfaces present at VM boot (included in the initial network configuration) don’t need any extra setup, because the guest OS configures them at startup through Cloud-Init.

Snapshots, recovery, and cloning

Snapshots let you capture the current state of a resource for later recovery or cloning. A disk snapshot saves only the data of the selected disk, while a virtual machine (VM) snapshot includes its parameters and the state of all its disks.