Last year Zalando Tech offered its engineers to host their company source code on GitHub Enterprise. This was based on the feedback we received from Zalando Tech’s GitHub Enterprise and Stash Workshop.

At Zalando Tech, we’ve setup GitHub Enterprise with a High Availability configuration based on AWS, including backup. Whenever the main GitHub enterprise instance is down, you have two options with which to proceed:

1. Promote the replication instance to a new main instance.
2. Restore from the backup instance.

We were faced with two challenges to include an HA setup in Zalando Tech:

1. Usage of Taupage AMI with a Docker container on top is mandatory to be Zalando company compliant on AWS. It’s an Amazon machine image created by Zalando STUPS. The first backup approach on AWS was just an Ubuntu box.
2. Source code data might be affected if the AWS account where GitHub Enterprise instances run gets compromised. Often, source code is a company’s intellectual property and you don’t want it to be exploited. On top of that, source code affection may result in malicious code added without anyone noticing.

AWS delivers scalable cloud-based IT infrastructure services and resources, and Docker images are designed as a composition of layers. A minimal amount of data is sent over the network – like in the ghe-backup case to be deployed to AWS – due to the composition of layers that a Docker image consists of.

Hence, both technologies make perfect sense for a GitHub Enterprise backup approach.

Based on this idea, we created the open source ghe-backup project. This will help people out there facing the same challenges we had. Deploying updated backup instances into different AWS accounts might be one such use case. However, using Taupage will not be a requirement for people outside of Zalando.

The basic idea of ghe-backup is to have two backup instances based on Taupage running in two AWS accounts:

Both backup instances fetch the latest changes every hour and persist the data on EBS volumes within the AWS accounts.

In order to connect a GitHub Enterprise Backup host with the master, the backup host requires a private ssh key that matches its public ssh key present on the GitHub Enterprise primary instance.

The private ssh key can’t be shipped for security reasons with the actual deployment, therefore KMS is used to encrypt the private key. The KMS key is defined within ghe-backup’s senza yaml file:

  ...
        kms_private_ssh_key: "aws:kms:myAWSregion:123456789:key/myrandomstringwithnumbers123456567890"
  ...

See here for a full sample senza yaml file.

Once a deployment is triggered, the encrypted private ssh key value is fetched from KMS, decrypted, and written to a file. Most of this is defined in the appropriate Dockerfile. Afterwards, a cron job triggers the actual hourly backups within extended business hours.

The KMS key defined in senza yaml is taken on a deployment to fetch the encrypted private ssh that gets decrypted, and put into the Docker container running inside of Taupage.

With the above method, we have GitHub Enterprise backups in separate AWS accounts as well as the backups that are fetched in a company-compliant way using Taupage.

We're hiring! Do you like working in an ever evolving organization such as Zalando? Consider joining our teams as a Software Engineer!