Backup Konzepte für GitLab prüfen

Wie im Abstimmungsworkshop vom 20.1.2022 besprochen:

Wir sollten uns überlegen, wie wir Gitlab Artemis und Gitlab Sharing "vernünftig" sichern bzw. im Notfall wiederherstellen können.

changed milestone to %2022 März Sprint

assigned to @simon.priller

Backup GitLab

based on https://docs.gitlab.com/ee/raketasks/backup_restore.html

Create a backup

docker exec -t <container name> gitlab-backup create STRATEGY=copy

Also backup /etc/gitlab/gitlab-secrets.json and /etc/gitlab/gitlab.rb separately.

Uploading to locally mounted shares

Mounting the backup destination folder in gitlab is necessary! This can be easily done in the docker-compose.yaml like we do it for the filehooks project mounts.

Edit /etc/gitlab/gitlab.rb:

gitlab_rails['backup_upload_connection'] = {
  :provider => 'Local',
  :local_root => '/mnt/backups'
}

# The directory inside the mounted folder to copy backups to
# Use '.' to store them in the root directory
gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups'

• Configuring cron to make daily backups

  Edit the crontab for the root user:

  sudo su -
  crontab -e

  There, add the following line to schedule the backup for everyday at 2 AM:

  0 2 * * * docker exec -t <container name> gitlab-backup create STRATEGY=copy CRON=1

• Prune old backups

  Edit /etc/gitlab/gitlab.rb

  ## Limit backup lifetime to 7 days - 604800 seconds
  gitlab_rails['backup_keep_time'] = 604800

Restoring the gitlab container from a backup is quite straight forward:

• Restore prerequisites
• Restore for Docker image

  # Stop the processes that are connected to the database
  docker exec -it <name of container> gitlab-ctl stop puma
  docker exec -it <name of container> gitlab-ctl stop sidekiq
  
  # Verify that the processes are all down before continuing
  docker exec -it <name of container> gitlab-ctl status
  
  # Run the restore. NOTE: "_gitlab_backup.tar" is omitted from the name
  docker exec -it <name of container> gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce
  
  # Restart the GitLab container
  docker restart <name of container>
  
  # Check GitLab
  docker exec -it <name of container> gitlab-rake gitlab:check SANITIZE=true

changed milestone to %2022 April/Mai Sprint

Concept looks good and will be applied to our Artemis Gitlab instance (codeability-git.uibk.ac.at) soon, as a restart is needed in order to mount the additional volumes gitlab_backups and gitlab_config_backup.

In order to backup the configs and secrets separately, a copy command will be added to the cronjob line:

0 2 * * * docker exec -t <container name> bash -c "gitlab-backup create STRATEGY=copy CRON=1;cp -r /etc/gitlab /var/opt/gitlab/config_backup"

assigned to @dancraz and unassigned @simon.priller

Backup strategy has now been set up for both codeability-git.uibk.ac.at and sharing-codeability.uibk.ac.at

A wiki page describing the backup strategy has been added at https://sharing-codeability.uibk.ac.at/sharing/codeability-sharing-platform/-/wikis/technical/Backup-strategies

@simon.priller can you take a look at it if it's okay?

assigned to @simon.priller and unassigned @dancraz

i checked it. it looks ok!

closed