diff --git a/documentation/custom-backup-and-restore.md b/documentation/custom-backup-and-restore.md new file mode 100644 index 00000000..90f2e6d7 --- /dev/null +++ b/documentation/custom-backup-and-restore.md @@ -0,0 +1,174 @@ +# Custom Backup and Restore Providers + +With enough effort one can create a custom backup and restore provider +for the Jenkins Operator. + +## Requirements + +Two commands (e.g. scripts) are required: +- a backup command, e.g. `backup.sh` that takes one argument, a **backup number** +- a restore command, e.g. `backup.sh` that takes one argument, a **backup number** + +Both scripts need to return an exit code of `0` on success and `1` or greater for failure. + +One of those scripts (or the entry point of the container) needs to be responsible +for backup cleanup or rotation if required, or an external system. + +## How it works + +The mechanism relies on basic Kubernetes and UNIX functionalities. + +The backup (and restore) container runs as a sidecar in the same +Kubernetes pod as the Jenkins master. + +Name of the backup and restore containers can be set as necessary using +`spec.backup.containerName` and `spec.restore.containerName`. +In most cases it will be the same container, but we allow for less common use cases. + +The operator will call a backup or restore commands inside a sidecar container when necessary: +- backup command (defined in `spec.backup.action.exec.command`) + will be called every `N` seconds configurable in: `spec.backup.interval` + and on pod shutdown (if enabled in `spec.backup.makeBackupBeforePodDeletion`) + with an integer representing the current backup number as first and only argument +- restore command (defined in `spec.restore.action.exec.command`) + will be called at Jenkins startup + with an integer representing the backup number to restore as first and only argument + (can be overridden using `spec.restore.recoveryOnce`) + +## Example AWS S3 backup using the CLI + +This example shows abbreviated version of a simple AWS S3 backup implementation +using: `aws-cli`, `bash` and `kube2iam`. + +In addition to your normal `Jenkins` `CustomResource` some additional settings +for backup and restore are required, e.g.: +```yaml +kind: Jenkins +apiVersion: jenkins.io/v1alpha1 +metadata: + name: example + namespace: jenkins +spec: + master: + masterAnnotations: + iam.amazonaws.com/role: "my-example-backup-role" # tell kube2iam where the AWS IAM role is + containers: + - name: jenkins-master + ... + - name: backup # container responsible for backup and restore + image: quay.io/virtuslab/aws-cli:1.16.263-2 + workingDir: /home/user/bin/ + command: # our container entry point + - sleep + - infinity + env: + - name: BACKUP_BUCKET + value: my-example-bucket # the S3 bucket name to use + - name: BACKUP_PATH + value: my-backup-path # the S3 bucket path prefix to use + - name: JENKINS_HOME + value: /jenkins-home # the path to mount jenkins home dir in the backup container + volumeMounts: + - mountPath: /jenkins-home # Jenkins home volume + name: jenkins-home + - mountPath: /home/user/bin/backup.sh + name: backup-scripts + subPath: backup.sh + readOnly: true + - mountPath: /home/user/bin/restore.sh + name: backup-scripts + subPath: restore.sh + readOnly: true + volumes: + - name: backup-scripts + configMap: + defaultMode: 0754 + name: jenkins-operator-backup-s3 + securityContext: # make sure both containers use the same UID and GUID + runAsUser: 1000 + fsGroup: 1000 + ... + backup: + containerName: backup # container name responsible for backup + interval: 3600 # how often make a backup in seconds + makeBackupBeforePodDeletion: true # trigger backup just before deleting the pod + action: + exec: + command: + # this command is invoked on "backup" container to create a backup, + # is passed by operator, + # for example /home/user/bin/backup.sh + - /home/user/bin/backup.sh + restore: + containerName: backup # container name is responsible for restore backup + action: + exec: + command: + # this command is invoked on "backup" container to restore a backup, + # is passed by operator + # for example /home/user/bin/restore.sh + - /home/user/bin/restore.sh +# recoveryOnce: # if want to restore specific backup configure this field and then Jenkins will be restarted and desired backup will be restored +``` + +The actual backup and restore scripts will be provided in a `ConfigMap`: + +```yaml +kind: ConfigMap +apiVersion: v1 +metadata: + name: jenkins-operator-backup-s3 + namespace: jenkins + labels: + app: jenkins-operator +data: + backup.sh: |- + #!/bin/bash -xeu + [[ ! $# -eq 1 ]] && echo "Usage: $0 backup_number" && exit 1; + [[ -z "${BACKUP_BUCKET}" ]] && echo "Required 'BACKUP_BUCKET' env not set" && exit 1; + [[ -z "${BACKUP_PATH}" ]] && echo "Required 'BACKUP_PATH' env not set" && exit 1; + [[ -z "${JENKINS_HOME}" ]] && echo "Required 'JENKINS_HOME' env not set" && exit 1; + + backup_number=$1 + echo "Running backup #${backup_number}" + + BACKUP_TMP_DIR=$(mktemp -d) + tar -C ${JENKINS_HOME} -czf "${BACKUP_TMP_DIR}/${backup_number}.tar.gz" --exclude jobs/*/config.xml --exclude jobs/*/workspace* -c jobs && \ + + aws s3 cp ${BACKUP_TMP_DIR}/${backup_number}.tar.gz s3://${BACKUP_BUCKET}/${BACKUP_PATH}/${backup_number}.tar.gz + echo Done + + restore.sh: |- + #!/bin/bash -xeu + [[ ! $# -eq 1 ]] && echo "Usage: $0 backup_number" && exit 1 + [[ -z "${BACKUP_BUCKET}" ]] && echo "Required 'BACKUP_BUCKET' env not set" && exit 1; + [[ -z "${BACKUP_PATH}" ]] && echo "Required 'BACKUP_PATH' env not set" && exit 1; + [[ -z "${JENKINS_HOME}" ]] && echo "Required 'JENKINS_HOME' env not set" && exit 1; + + backup_number=$1 + echo "Running restore #${backup_number}" + + BACKUP_TMP_DIR=$(mktemp -d) + aws s3 cp s3://${BACKUP_BUCKET}/${BACKUP_PATH}/${backup_number}.tar.gz ${BACKUP_TMP_DIR}/${backup_number}.tar.gz + + tar -C ${JENKINS_HOME} -zxf "${BACKUP_TMP_DIR}/${backup_number}.tar.gz" + echo Done +``` + +In our example we will use S3 bucket lifecycle policy to keep +the number of backups under control, e.g. Cloud Formation fragment: +```yaml + Type: AWS::S3::Bucket + Properties: + BucketName: my-example-bucket + ... + LifecycleConfiguration: + Rules: + - Id: BackupCleanup + Status: Enabled + Prefix: my-backup-path + ExpirationInDays: 7 + NoncurrentVersionExpirationInDays: 14 + AbortIncompleteMultipartUpload: + DaysAfterInitiation: 3 +```