Velero Kubernetes backup

Velero

Velero is an open-source tool tailored for Kubernetes environments. It facilitates backup, restoration, disaster recovery, and resource migration tasks. It captures the state of a Kubernetes cluster, including all associated resources, and stores it in a designated backup location. During crises or migrations, Velero efficiently restores the backed-up resources to their original state, ensuring high availability and resilience. Its selective backup feature and pre- and post-backup hooks offer customization options for tailored management of Kubernetes clusters.

Storage for backup

First, we need storage to save backups. There are plenty of supported types, but I chosen the Azure Storage Account.

az login

AZURE_BACKUP_RESOURCE_GROUP=velero

AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"

az storage account create \
    --name $AZURE_STORAGE_ACCOUNT_ID \
    --resource-group $AZURE_BACKUP_RESOURCE_GROUP \
    --sku Standard_GRS \
    --encryption-services blob \
    --https-only true \
    --min-tls-version TLS1_2 \
    --kind BlobStorage \
    --access-tier Hot

BLOB_CONTAINER=velero
az storage container create -n $BLOB_CONTAINER --public-access off --account-name $AZURE_STORAGE_ACCOUNT_ID

AZURE_STORAGE_ACCOUNT_ACCESS_KEY=`az storage account keys list --account-name $AZURE_STORAGE_ACCOUNT_ID --query "[?keyName == 'key1'].value" -o tsv`

Kubernetes clusters

First of all, we need actual clusters to perform our tests. To avoid unnecessary complexity, I've chosen to use Kind.

Kind's configuration file is the same for the backup and restore cluster.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"

Kubernetes cluster to backup.

This will be a source to backup from.

$ kind create cluster --name cluster1 --config=kind-noports.config
Creating cluster "cluster1" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) đŸ–ŧ 
 ✓ Preparing nodes đŸ“Ļ  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹ī¸ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-cluster1"
You can now use your cluster with:

kubectl cluster-info --context kind-cluster1

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

Kubernetes cluster to restore

This will be a destination to restore to.

$ kind create cluster --name cluster2 --config=kind-noports.config
Creating cluster "cluster2" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) đŸ–ŧ 
 ✓ Preparing nodes đŸ“Ļ  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹ī¸ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-cluster1"
You can now use your cluster with:

kubectl cluster-info --context kind-cluster2

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

Velero CLI - local part

The Velero CLI is the primary tool for interacting with Velero to perform tasks like backup and restore.

# Get binary
$ wget https://github.com/vmware-tanzu/velero/releases/download/v1.13.2/velero-v1.13.2-linux-amd64.tar.gz

# Extraxt
$ tar -xvf velero-v1.13.2-linux-amd64.tar.gz

# Cd
$ cd velero-v1.13.2-linux-amd64/

# Move to PATH
$ sudo cp velero /usr/local/bin/

# Check
$ velero version

Client:
        Version: v1.13.2
        Git commit: 4d961fb6fec384ed7f3c1b7c65c818106107f5a6
Server:
        Version: v1.13.2

Velero - Kubernetes part

Velero supports various Kubernetes resources and can be installed using either the Velero CLI or Helm charts. Additionally, both methods provide a straightforward setup process, ensuring flexibility and ease of use. I've used Velero CI. To install Azure provider, we have to select two things: version and authentication method to Storage Account.

Version velero/velero-plugin-for-microsoft-azure:v1.9.2

When it comes to authorization, we have the following mechanisms:

  • SPN
  • Azure AD Workload Identity
  • Storage account access key

The simplest option is the last one; I'll follow that approach for this post. However, I recommend investigating SPN or Workload Identity options for a production-ready solution. Moreover, these alternatives provide enhanced security and scalability for enterprise environments.

The first point is to create a file that contains the Storage Account Access Key:

cat << EOF  > ./credentials-velero
AZURE_STORAGE_ACCOUNT_ACCESS_KEY=${AZURE_STORAGE_ACCOUNT_ACCESS_KEY}
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

Then, run velero install

velero install \
    --provider azure \
    --plugins velero/velero-plugin-for-microsoft-azure:v1.9.2 \
    --bucket $BLOB_CONTAINER \
    --secret-file ./credentials-velero \
    --backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,storageAccountKeyEnvVar=AZURE_STORAGE_ACCOUNT_ACCESS_KEY \
    --use-volume-snapshots=false
    
 # check 
 velero backup-location get

Kubernetes backup

After installation, we can check how many backups we have already created.

$ velero backup get

I see nothing, which means that there is nothing yet.

Create simply deployment

This deploys a pod and replicaset, providing something to back up.

$ kubectl create namesapce ngx
$ kubectl create deployment nginx-project --image=nginx -n ngx

$ kubectl -n ngx get all
NAME                                 READY   STATUS    RESTARTS      AGE
pod/nginx-project-7b5d9946b6-jxsfv   1/1     Running   1 (48m ago)   10h

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-project   1/1     1            1           10h

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-project-7b5d9946b6   1         1         1       10h

Triggering backup at-hoc and getting the details

# trigger
$ velero backup create nginx-backup-1 --include-namespaces ngx --default-volumes-to-fs-backup=false --snapshot-volumes=false --include-cluster-resources=false

# check
$ velero backup get
NAME             STATUS      ERRORS   WARNINGS   CREATED                          EXPIRES   STORAGE LOCATION   SELECTOR
nginx-backup-1   Completed   0        0          2024-05-27 14:19:59 +0200 CEST   29d       default            <none>

# more details
$ velero backup describe nginx-backup-1
Name:         nginx-backup-1
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.30.0
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=30

Phase:  Completed


Namespaces:
  Included:  ngx
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  excluded

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  false
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-05-27 14:19:59 +0200 CEST
Completed:  2024-05-27 14:20:04 +0200 CEST

Expiration:  2024-06-26 14:19:59 +0200 CEST

Total items to be backed up:  5
Items backed up:              5

Backup Volumes: <none included>

HooksAttempted:  0
HooksFailed:     0

Kubernetes restore

Two steps need to be completed to restore the backup into another cluster (essentially a cluster migration process). First, install the Velero resource on Kubernetes Cluster 2. Subsequently, run the velero restore command. By following these steps, you can efficiently migrate your cluster.

First, install the Velero resource on cluster 2. Did you see it? This is the same command executed in a different context.

$ velero install \
    --provider azure \
    --plugins velero/velero-plugin-for-microsoft-azure:v1.9.2 \
    --bucket $BLOB_CONTAINER \
    --secret-file ./credentials-velero \
    --backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,storageAccountKeyEnvVar=AZURE_STORAGE_ACCOUNT_ACCESS_KEY \
    --use-volume-snapshots=false

Before restoring, let's confirm there is nothing out there.

$ k -n ngx get all
No resources found in ngx namespace.

Restore

$ velero restore create --from-backup nginx-backup-1

Checking

$ k -n ngx get all
NAME                                 READY   STATUS    RESTARTS        AGE
pod/nginx-project-7b5d9946b6-jxsfv   1/1     Running   2 (6m30s ago)   24h

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-project   1/1     1            1           24h

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-project-7b5d9946b6   1         1         1       24h

Velero - other feature

Scheduling - you can set up a backup to run at specific intervals, such as daily or every Monday. This is cron-like syntax.

$ velero schedule create SCHEDULE_NAME --schedule "0 0 * * *" --include-namespaces NAMESPACE

Helm - you can install Velero exclusively using the Helm chart.

$ helm install --set-file credentials.secretContents.cloud=./credentials-velero stable/velero

You can exclude namespaces, resources, or individual resources by label.

$ kubectl label -n <ITEM_NAMESPACE> <RESOURCE>/<NAME> velero.io/exclude-from-backup=true

Summary of Velero Kubernetes backup

Velero is a widely used, open-source tool designed to handle the backup, restoration, and migration of Kubernetes cluster resources. It offers robust features that ensure data integrity and facilitate smooth cluster management. By leveraging Velero, users can efficiently protect their Kubernetes environments and streamline the process of moving resources between clusters.

Leave a Reply

Your email address will not be published. Required fields are marked *