Enable volume snapshots (vSphere CSI)

What you’ll accomplish

You’ll enable Velero to capture persistent volume data using vSphere CSI snapshots on an RKE2 cluster, so your backups include both Kubernetes resources and on-disk application state.

Prerequisites

• UDS CLI installed
• Access to an RKE2 cluster with UDS Core deployed
• Velero storage backend configured (see Configure Velero storage backends)
• vSphere environment with a user account that has the required CSI roles and privileges (see Broadcom vSphere Roles and Privileges)
• Ability to apply HelmChartConfig overrides to RKE2 system charts

Before you begin

By default, UDS Core backs up Kubernetes resources only. Volume snapshots are disabled:

Setting	Default
snapshotsEnabled	false
schedules.udsbackup.template.snapshotVolumes	false

Note

If your applications use PersistentVolumes and you need to restore the actual on-disk data (not just the PVC resource definitions), you must enable volume snapshots. Without them, a restore will recreate the PVC but the underlying data will be lost.

Danger

The default vSphere limit of 3 snapshots per block volume is insufficient for UDS Core’s 10-day backup retention. Each daily backup creates approximately one snapshot per volume, so the default is exhausted after 3 days and further backups fail silently. You must set global-max-snapshots-per-block-volume to at least 10 (12 recommended for buffer) in the CSI driver configuration. This is configured in step 1.

Steps

1. Install and configure the vSphere CSI driver

   On your RKE2 cluster, set the cloud provider in your RKE2 configuration:

   config.yaml

   cloud-provider-name: rancher-vsphere

   Note

   While RKE2 deploys the rancher-vsphere-cpi and rancher-vsphere-csi Helm charts automatically, they will not function correctly until configured with vSphere credentials and other settings. The HelmChartConfig overrides below are essential.

   Provide HelmChartConfig overrides for the CPI and CSI drivers. Three CSI overrides are critical: blockVolumeSnapshot must be enabled, configTemplate must be overridden to include the snapshot limit, and global-max-snapshots-per-block-volume must be set high enough for your retention policy.

   helmchartconfig.yaml

   ---
   apiVersion: helm.cattle.io/v1
   kind: HelmChartConfig
   metadata:
     name: rancher-vsphere-cpi
     namespace: kube-system
   spec:
     valuesContent: |-
       vCenter:
         host: "<vsphere-server>"
         port: 443
         insecureFlag: true
         datacenters: "<vsphere-datacenter-name>"
         username: "<vsphere-csi-username>"
         password: "<vsphere-csi-password>"
         credentialsSecret:
           name: "vsphere-cpi-creds"
           generate: true
   ---
   apiVersion: helm.cattle.io/v1
   kind: HelmChartConfig
   metadata:
     name: rancher-vsphere-csi
     namespace: kube-system
   spec:
     valuesContent: |-
       vCenter:
         datacenters: "<vsphere-datacenter-name>"
         username: "<vsphere-csi-username>"
         password: "<vsphere-csi-password>"
         configSecret:
           configTemplate: |
             [Global]
             cluster-id = "<rke2-cluster-id>"
             user = "<vsphere-csi-username>"
             password = "<vsphere-csi-password>"
             port = 443
             insecure-flag = "1"
             [VirtualCenter "<vsphere-server>"]
             datacenters = "<vsphere-datacenter-name>"
             [Snapshot]
             global-max-snapshots-per-block-volume = 12
       csiNode:
         tolerations:
         - operator: "Exists"
           effect: "NoSchedule"
       blockVolumeSnapshot:
         enabled: true
       storageClass:
         reclaimPolicy: Retain

   Note

   Some pre-created roles in vSphere may be named differently than the Broadcom documentation suggests (for example, CNS-Datastore may appear as CNS-Supervisor-Datastore).

2. Create a VolumeSnapshotClass

   Define a VolumeSnapshotClass that tells Velero how to create snapshots using the vSphere CSI driver. Deploy this as a manifest in a Zarf package included in your bundle:

   volumesnapshotclass.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshotClass
   metadata:
     name: vsphere-csi-snapshot-class
     labels:
       velero.io/csi-volumesnapshot-class: "true"
   driver: csi.vsphere.vmware.com
   deletionPolicy: Retain

   Tip

   The velero.io/csi-volumesnapshot-class: "true" label is required for Velero to discover and use this VolumeSnapshotClass.

3. Enable CSI snapshots in Velero

   Add the following overrides to enable CSI-based volume snapshots:

   uds-bundle.yaml

   packages:
     - name: core
       repository: registry.defenseunicorns.com/public/core
       ref: x.x.x-upstream
       overrides:
         velero:
           velero:
             values:
               - path: configuration.features
                 value: EnableCSI
               - path: snapshotsEnabled
                 value: true
               - path: configuration.volumeSnapshotLocation
                 value:
                   - name: default
                     provider: velero.io/csi
               - path: schedules.udsbackup.template.snapshotVolumes
                 value: true

4. Create and deploy your bundle

   uds create <path-to-bundle-dir>
   uds deploy uds-bundle-<name>-<arch>-<version>.tar.zst

Verification

# Verify snapshots are enabled on the schedule
uds zarf tools kubectl get schedule -n velero velero-udsbackup -o jsonpath='{.spec.template.snapshotVolumes}'

# Verify the VolumeSnapshotLocation exists
uds zarf tools kubectl get volumesnapshotlocation -n velero

# After a backup completes, check for volume snapshots
uds zarf tools kubectl get volumesnapshot -A

Success criteria:

• snapshotVolumes is true on the schedule
• A VolumeSnapshotLocation with provider velero.io/csi exists in the velero namespace
• After a backup completes, VolumeSnapshot resources are created for each PVC
• Snapshot count matches the number of PVCs in backed-up namespaces

To trigger a manual backup for testing, see Perform a manual backup.

Troubleshooting

Problem: Snapshot limit reached

Symptoms: Backups fail with a FailedPrecondition error in the Velero logs:

error executing custom action: rpc error: code = FailedPrecondition desc =
the number of snapshots on the source volume reaches the configured maximum (3)

Solution: Increase global-max-snapshots-per-block-volume in the vSphere CSI HelmChartConfig. A value of at least 10 is required for the default 10-day retention, with 12 recommended for buffer. See the snapshot limit guidance in Before you begin and update the [Snapshot] section in the CSI configTemplate in step 1.

Problem: VolumeSnapshotContents remain after backup deletion

Symptoms: Deleting a backup does not clean up the associated VolumeSnapshotContents in Kubernetes or in vSphere.

Solution: Be cautious when deleting backups that have been used for restores — Velero may attempt to delete VolumeSnapshotContents that are still in use by restored volumes. Velero’s garbage collection runs hourly by default.

Tip

The pyvmomi-community-samples repository contains scripts for interacting with vSphere directly. The fcd_list_vdisk_snapshots script is useful for listing snapshots stored in vSphere that cannot be viewed in the vSphere UI — particularly when snapshots and VolumeSnapshotContents are deleted from the cluster but not cleaned up in vSphere.

Related documentation

• Velero: CSI Snapshot Support — CSI integration details and configuration
• Kubernetes: Volume Snapshots — CSI snapshot API reference
• Rancher vSphere Charts — CPI and CSI driver Helm charts
• vSphere CSI Snapshot Limits — snapshot per volume configuration
• Backup & restore concepts — how Velero fits into UDS Core

Next steps

These guides and concepts may be useful to explore next:

Perform a manual backup Verify your scheduled backups are running and trigger a manual backup on demand.

Configure Velero storage backends Set up S3 or Azure Blob storage, provide credentials, and customize the backup schedule.