End-to-End Kubernetes with Rancher, RKE2, K3s, Fleet, Longhorn, and NeuVector

Best Practices for Cluster Backup and Restore

Automated Backups

To automate backups, you need to follow these steps:

• Click on Global > Cluster Management.
• Next to the cluster you want to take a snapshot of, click on the three dots and select Edit Config.
• In the Cluster Configuration section, click on etcd.
• You can now enable the Automated Snapshots option and set the cron expression for the schedule. Here are some examples you can set up:
  • 0 0 * * *: at 00:00.
  • 0 */5 * * *: at minute 0 past every 5th hour.
  • 0 0 * * 0: at 00:00 on Sunday.
  • ..etc. You can use crontab.guru to generate your own cron expressions.
• You can also choose how many snapshots you want to keep (e.g., 3, 5, 10, etc.).

ℹ️ Make sure the size of the snapshots directory is large enough to store a few snapshots. If the directory runs out of space, the snapshots will fail.

Offsite Storage

To guard against data loss, it is recommended to store the snapshots in an offsite location. You can use an S3-compatible object storage service like Amazon S3 or Minio to store the snapshots. To do this, you can follow the same steps as above, and enable the Backup Snapshot to S3 option and provide the following information:

• Authentication: Access Key and Secret Key.
• Endpoint: The URL of the S3-compatible object storage service.
• Bucket: The name of the bucket where the snapshots will be stored.
• Folder: The folder within the bucket where the snapshots will be stored.
• Region: The region where the bucket is located.
• Insecure: If the S3-compatible object storage service uses an insecure connection.
• CACert: The CA certificate for the S3-compatible object storage service you are using.

Backup the Management Cluster

In Rancher, a good practice, as we have seen, is to create a management cluster (Rancher Manager) that is separate from the workload clusters. This cluster is used to manage the downstream workload clusters. To ensure that you can recover from a disaster, it is important to back up the management cluster too. However, its backup and restore is not the same as the workload clusters. Instead, you should follow these steps:

• Click on Global > Cluster Management.
• Next to the local cluster, click on Explore and navigate to the Apps section.
• Install the Helm chart Rancher Backups in the system project (or any other project you prefer).
• Choose between the following options:
  • No default storage location: This is not the recommended option.
  • Use an S3-compatible object store: This is a better option as the backups will be stored in an offsite location.
  • Use an existing storage class: This is another option if you want to use an existing storage class.
  • Use an existing persistent volume: In case you have an existing persistent volume that you want to use.
  • For the sake of simplicity, we will use local-path as the storage class, but I recommend using an S3-compatible object store for production environments.
• Click Install to install the Helm chart. The cattle-resources-system namespace will be created, and the Rancher Backups tool will be installed in this namespace.
• Once installed, you should go to the Rancher Backups tab and create a new backup.

Before proceeding, let's discuss the encryption of the backups. Each backup could be encrypted with a Kubernetes EncryptionConfig secret. You have the choice of not encrypting it, but it is not recommended for security reasons. If you choose not to encrypt the backup and if someone gains access to the backup, they could potentially access all the data in the backup, including access to all the clusters managed by Rancher.

To create the encryption key (Secret), you can follow these steps:

• Generate a 32-byte base64-encoded encryption key:

BASE64_ENCODED_KEY=$(head -c 32