Velero is an open-source tool tailored for Kubernetes environments. It facilitates backup, restoration, disaster recovery, and resource migration tasks. It captures the state of a Kubernetes cluster, including all associated resources, and stores it in a designated backup location. During crises or migrations, Velero efficiently restores the backed-up resources to their original state, ensuring high availability and resilience. Its selective backup feature and pre- and post-backup hooks offer customization options for tailored management of Kubernetes clusters.
Storage for backup
First, we need storage to save backups. There are plenty of supported types, but I chosen the Azure Storage Account.
az login AZURE_BACKUP_RESOURCE_GROUP=velero AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')" az storage account create \ --name $AZURE_STORAGE_ACCOUNT_ID \ --resource-group $AZURE_BACKUP_RESOURCE_GROUP \ --sku Standard_GRS \ --encryption-services blob \ --https-only true \ --min-tls-version TLS1_2 \ --kind BlobStorage \ --access-tier Hot BLOB_CONTAINER=velero az storage container create -n $BLOB_CONTAINER --public-access off --account-name $AZURE_STORAGE_ACCOUNT_ID AZURE_STORAGE_ACCOUNT_ACCESS_KEY=`az storage account keys list --account-name $AZURE_STORAGE_ACCOUNT_ID --query "[?keyName == 'key1'].value" -o tsv`
Kubernetes clusters
First of all, we need actual clusters to perform our tests. To avoid unnecessary complexity, I've chosen to use Kind.
Kind's configuration file is the same for the backup and restore cluster.
kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane kubeadmConfigPatches: - | kind: InitConfiguration nodeRegistration: kubeletExtraArgs: node-labels: "ingress-ready=true"
Kubernetes cluster to backup.
This will be a source to backup from.
$ kind create cluster --name cluster1 --config=kind-noports.config Creating cluster "cluster1" ... â Ensuring node image (kindest/node:v1.25.3) đŧ â Preparing nodes đĻ â Writing configuration đ â Starting control-plane đšī¸ â Installing CNI đ â Installing StorageClass đž Set kubectl context to "kind-cluster1" You can now use your cluster with: kubectl cluster-info --context kind-cluster1 Not sure what to do next? đ Check out https://kind.sigs.k8s.io/docs/user/quick-start/
Kubernetes cluster to restore
This will be a destination to restore to.
$ kind create cluster --name cluster2 --config=kind-noports.config Creating cluster "cluster2" ... â Ensuring node image (kindest/node:v1.25.3) đŧ â Preparing nodes đĻ â Writing configuration đ â Starting control-plane đšī¸ â Installing CNI đ â Installing StorageClass đž Set kubectl context to "kind-cluster1" You can now use your cluster with: kubectl cluster-info --context kind-cluster2 Not sure what to do next? đ Check out https://kind.sigs.k8s.io/docs/user/quick-start/
Velero CLI - local part
The Velero CLI is the primary tool for interacting with Velero to perform tasks like backup and restore.
# Get binary $ wget https://github.com/vmware-tanzu/velero/releases/download/v1.13.2/velero-v1.13.2-linux-amd64.tar.gz # Extraxt $ tar -xvf velero-v1.13.2-linux-amd64.tar.gz # Cd $ cd velero-v1.13.2-linux-amd64/ # Move to PATH $ sudo cp velero /usr/local/bin/ # Check $ velero version Client: Version: v1.13.2 Git commit: 4d961fb6fec384ed7f3c1b7c65c818106107f5a6 Server: Version: v1.13.2
Velero - Kubernetes part
Velero supports various Kubernetes resources and can be installed using either the Velero CLI or Helm charts. Additionally, both methods provide a straightforward setup process, ensuring flexibility and ease of use. I've used Velero CI. To install Azure provider, we have to select two things: version and authentication method to Storage Account.
Version velero/velero-plugin-for-microsoft-azure:v1.9.2
When it comes to authorization, we have the following mechanisms:
- SPN
- Azure AD Workload Identity
- Storage account access key
The simplest option is the last one; I'll follow that approach for this post. However, I recommend investigating SPN or Workload Identity options for a production-ready solution. Moreover, these alternatives provide enhanced security and scalability for enterprise environments.
The first point is to create a file that contains the Storage Account Access Key:
cat << EOF > ./credentials-velero AZURE_STORAGE_ACCOUNT_ACCESS_KEY=${AZURE_STORAGE_ACCOUNT_ACCESS_KEY} AZURE_CLOUD_NAME=AzurePublicCloud EOF
Then, run velero install
velero install \ --provider azure \ --plugins velero/velero-plugin-for-microsoft-azure:v1.9.2 \ --bucket $BLOB_CONTAINER \ --secret-file ./credentials-velero \ --backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,storageAccountKeyEnvVar=AZURE_STORAGE_ACCOUNT_ACCESS_KEY \ --use-volume-snapshots=false # check velero backup-location get
Kubernetes backup
After installation, we can check how many backups we have already created.
$ velero backup get
I see nothing, which means that there is nothing yet.
Create simply deployment
This deploys a pod and replicaset, providing something to back up.
$ kubectl create namesapce ngx $ kubectl create deployment nginx-project --image=nginx -n ngx $ kubectl -n ngx get all NAME READY STATUS RESTARTS AGE pod/nginx-project-7b5d9946b6-jxsfv 1/1 Running 1 (48m ago) 10h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-project 1/1 1 1 10h NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-project-7b5d9946b6 1 1 1 10h
Triggering backup at-hoc
and getting the details
# trigger $ velero backup create nginx-backup-1 --include-namespaces ngx --default-volumes-to-fs-backup=false --snapshot-volumes=false --include-cluster-resources=false # check $ velero backup get NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR nginx-backup-1 Completed 0 0 2024-05-27 14:19:59 +0200 CEST 29d default <none> # more details $ velero backup describe nginx-backup-1 Name: nginx-backup-1 Namespace: velero Labels: velero.io/storage-location=default Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.30.0 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=30 Phase: Completed Namespaces: Included: ngx Excluded: <none> Resources: Included: * Excluded: <none> Cluster-scoped: excluded Label selector: <none> Or label selector: <none> Storage Location: default Velero-Native Snapshot PVs: false Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: <none> Backup Format Version: 1.1.0 Started: 2024-05-27 14:19:59 +0200 CEST Completed: 2024-05-27 14:20:04 +0200 CEST Expiration: 2024-06-26 14:19:59 +0200 CEST Total items to be backed up: 5 Items backed up: 5 Backup Volumes: <none included> HooksAttempted: 0 HooksFailed: 0
Kubernetes restore
Two steps need to be completed to restore the backup into another cluster (essentially a cluster migration process). First, install the Velero resource on Kubernetes Cluster 2. Subsequently, run the velero restore
command. By following these steps, you can efficiently migrate your cluster.
First, install the Velero resource on cluster 2. Did you see it? This is the same command executed in a different context.
$ velero install \ --provider azure \ --plugins velero/velero-plugin-for-microsoft-azure:v1.9.2 \ --bucket $BLOB_CONTAINER \ --secret-file ./credentials-velero \ --backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,storageAccountKeyEnvVar=AZURE_STORAGE_ACCOUNT_ACCESS_KEY \ --use-volume-snapshots=false
Before restoring, let's confirm there is nothing out there.
$ k -n ngx get all No resources found in ngx namespace.
Restore
$ velero restore create --from-backup nginx-backup-1
Checking
$ k -n ngx get all NAME READY STATUS RESTARTS AGE pod/nginx-project-7b5d9946b6-jxsfv 1/1 Running 2 (6m30s ago) 24h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-project 1/1 1 1 24h NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-project-7b5d9946b6 1 1 1 24h
Velero - other feature
Scheduling - you can set up a backup to run at specific intervals, such as daily or every Monday. This is cron-like syntax.
$ velero schedule create SCHEDULE_NAME --schedule "0 0 * * *" --include-namespaces NAMESPACE
Helm - you can install Velero exclusively using the Helm chart.
$ helm install --set-file credentials.secretContents.cloud=./credentials-velero stable/velero
You can exclude namespaces, resources, or individual resources by label.
$ kubectl label -n <ITEM_NAMESPACE> <RESOURCE>/<NAME> velero.io/exclude-from-backup=true
Summary of Velero Kubernetes backup
Velero is a widely used, open-source tool designed to handle the backup, restoration, and migration of Kubernetes cluster resources. It offers robust features that ensure data integrity and facilitate smooth cluster management. By leveraging Velero, users can efficiently protect their Kubernetes environments and streamline the process of moving resources between clusters.