GitHub Enterprise backup using Kubernetes Jobs and upload to AWS S3

We have a Enterprise GitHub environment, where the ask was to take hourly backup and send that to S3 bucket.

We have moved it form traditional process to microservice, Here I am going to demonstrate how I have transferred this load to AWS EKS (Amazon Elastic Kubernetes Service).

I have used Docker file from GitHub Backup utility to create docker image and added aws cli ( AWS Command Line Interface) into Docker file and created Docker Image and uploaded it to AWS ECR (Amazon Elastic Container Registry)

This demonstration code can be found at GitHub

Clone upstream Git repository

$ mkdir work
$ cd work
$ git clone https://github.com/github/backup-utils.git
$ cd backup-utils

Build container image and push it to ECR

$docker build -t github-backup-utils-aws

Note :- Before pushing image into ECR you have to setup registry into ECR, also your AWS EKS Cluster should have AWS S3 bucket access.

Explanation:

Converted private key into secret and created secret from this

$kubectl create secret generic --from-file=private_key_file -o yaml --dry-run=client / > private-key-secret.yaml

Now we will deploy things using yaml files

$ kubectl apply -f ns.yaml
$ kubectl apply -f private-key-secret.yaml
$ kubectl apply -f known-hosts-configmap.yaml
$ kubectl apply -f backup-config-configmap.yaml
$ kubectl apply -f backup_compress_upload_s3.yaml
$ kubectl apply -f pvc.yaml
$ kubectl apply -f github_cron_aws.yaml

If you find any difficulties while deploying kubernetes jobs, you can use deployment file and debug the issue.

Edit below files and change values as per environment

1. "S3_upload" and change "S3_BUCKET" as per environment
2. "back-config-example"
3. "known-hosts-configmap.yaml"
4. "private-key-secret.yaml"

Please comment and let me know if you find this helpful or any suggestion.