End-to-End Kubernetes with Rancher, RKE2, K3s, Fleet, Longhorn, and NeuVector

Snapshots and Backups

For disaster recovery scenarios, Longhorn provides built-in support for snapshots and backups. Both concepts are interconnected and serve complementary purposes for protection and recovery.

A snapshot, working by capturing changes incrementally in the primary storage, is a point-in-time representation of a volume's data. Snapshots are stored locally as part of the volume’s replica chain and are used to preserve the history of changes.

A backup, on the other hand, is created using a snapshot as its source and represents a flattened version of the snapshot chain. Unlike snapshots, backups are stored in secondary storage (e.g., an object storage service like S3 or Minio), outside the Kubernetes cluster. Backups contain the state of the volume’s data at the moment the snapshot was taken but do not preserve the detailed history of changes.

ℹ️ Backups are incremental, meaning only the blocks that changed since the last snapshot are transmitted and stored. To optimize storage, Longhorn reuses unchanged blocks across multiple backups, compresses the blocks (using lz4 by default but can be changed to gzip), and uses checksums for deduplication.

As a practical example, let's see how to use S3 as a backup target for Longhorn. We will:

• Install AWS CLI on the workspace server.
• Create an S3 bucket in AWS.
• Create an IAM user with the necessary permissions to access the bucket.
• Create a Kubernetes secret with the IAM user's credentials.
• Configure Longhorn to use the S3 bucket as a backup target.

Start by installing the AWS CLI:

ssh root@$WORKSPACE_PUBLIC_IP

# Pip install AWS CLI
pip install awscli --break-system-packages

Export your credentials:

export AWS_ACCESS_KEY_ID=[YOUR_ACCESS_KEY_ID]
export AWS_SECRET_ACCESS_KEY=[YOUR_SECRET]

Create an S3 bucket in AWS:

export AWS_REGION=eu-west-3
export BUCKET_NAME=longhorn-backups-$(date +%s)

aws s3api create-bucket \
    --bucket $BUCKET_NAME \
    --region $AWS_REGION \
    --create-bucket-configuration LocationConstraint=$AWS_REGION \
    --acl private

Save the policy to a file using cat and a heredoc:

cat < /tmp/longhorn-s3-policy.json
# Create the policy file
cat > longhorn-s3-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LonghornS3Access",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": [
        "arn:aws:s3:::$BUCKET_NAME"
      ]
    },
    {
      "Sid": "LonghornS3ObjectAccess",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": [
        "arn:aws:s3:::$BUCKET_NAME/*"
      ]
    }
  ]
}
EOF

Create the IAM policy in AWS: