GitOps Flow on Gitlab CI

There is a new *ops thing every other day. I don’t usually pay much attention to hype, but this one got my interest. The main reason is I had thought of it before I read about it!

Background

We are using Kubernetes to deploy our applications. To achieve this, after usual compiling and testing steps, we create an image and push it to GCR. After that we have a manual build step to deploy the image. We get the image tag, and pass it as a variable to CI step, and run it. We use Helm to generate our Kubernetes templates and apply them.

The main problem with this approach is that it is not traceable. We pass many environment variables to Gitlab and Helm, and they are not tracked anywhere. We get the Gitlab CI execution log, but that is not what we want to rely on. As our releases are not tracked anywhere, we cannot reliably find the state of our application in any point in time, neither we can reproduce an environment.

Beside this, release pipeline is very hard to reproduce. What it means that if for any reason, you want to release from another place beside the originally implemented one, you will have a hard time doing that.

Idea

The main idea of GitOps is to capture your whole environment in a git repository. Why git repository? Git is developers holy grail of reliable storage. It is very accessible for developers. We used it for many years, and we trust it. Other reason is all of our code is in there, and we use it every day, so why not just add another one? On the other hand If we capture our environment in a git repository, we have a time based store of our releases and environments, we can reproduce it any time by simply checking out the repository at that point in time.

At the same time the whole release process is not a convoluted thing in the CI pipeline that nobody can reproduce, it should be easy to simply check out the repository, and run it to release the application.

Tools

You can use any tool you like pretty much, but of course there are tools aligned with gitops. The most important tool we chose for this is the new Kustomize from Kubernetes team. Kustomize is design with philosophy of gitops, it does not accept any environment variables out of the box, and everything should be in the configuration files. Also I really like the way it generates the files with the concept of overlays. It is like what we putting a translucence layer of yaml on top of the other one. The good part is all the partial yamls are reasonably readable. If you have used Helm in the past, but not to familiar with it, looking at a helm configuration file looks completely gibberish. It is just a bunch of semi-random variables in yaml format. But Kustomize keeps the format of the overlays close to the Kubernetes resources themselves.

Other nice feature about Kustomize is the command like editor to change the image name(s). What is means that you can issue a command from the command line, and change the image name or tag. This is very important when you want to automate the flow. Saves us some fragile bash scripting and sedding. We will see some sample of this below.

One thing to note is the structure of the repository that holds the kustimization code. In Kustomization, each environment will have a different “overlay”, and each overlay has a directory. So each of your environments gets a directory in the repository. This contrast the usual relationship that repository has with environment. It is usually different branches that runs on different environments. For example you might have your “develop” branch run in “DEV” environment, then “promote” that code to “production” by merging the develop to “master” branch. In a way, as your code moves between different environments, they are also move “up” in the branch hierarchy. This is nice concept, and breaking from it is not so easy, and very confusing.

Beside Kustomize, we used gitlab, but the reason is completely different. We use gitlab for all our build so it is a company standard. I won’t go into this comparison.

Flow

First of all I should give credit to This article from google to help me wrap my head around this. Even though the idea is known, and I had thought of that personally, this article put it in perspective.

We started with two repositories, one for the application code and the other one for the environment code. This might look overkill, but when you think about the principle of “keep stuff that change together close”, you will notice that the Kubernetes files change is very rare, and the reason is usually environment load related not code related. So it makes sense from a pure conceptual perspective to separate them. Anyway we have a valid reason to keep them separate in this case.

Once we approved a change in application repository, we merge it to master, if the merge is successful we consider it “production ready” at least from the view point of the application repository. We then create an image, version it and push it to a docker repository, in our case GCR.

Now the fun begins, the application repository sends a signal to environment repository indicating that there is a new “good” image ready to deploy. In our case this signal was in the form of merge request to the environment repository. This can be done by exporting a build artifact and using cross repository triggers, which can be a little cleaner, but this is the approach we are chosen, I will blog about that approach later.

When environment repository receives the merge request, it triggers a build process. This build process tries to release the merge request on master, and deploy the result to “DEV” environment. If the deploy is successful, merge the PR to repository. The idea of a pipeline changing the repository is a little strange, but you will get used to it.

After this point, the rest of the process is manual for us. That means deploying to UAT and Production environments are triggered manually, but you can choose to make them automatic by simply removing the constrain on them.

Deployment to UAT is basically follows the same logic as DEV, except that we need to find out the release version the hard way, by looking at “dev” overlay, and finding the image name and tag.

Implementation

I will go through some implementation detail that was tricky.

Env repo

First off, we need to setup out env-repository. This repository is going to host our kustomization templates, and the gitlab ci file.

The layout of the repository looks like standard Kustomization repository.

env: 
- kustomizations:
  - base:
    - deployment.yaml
    - hpa.yaml
    - kustomization.yaml
  - overlays:
    - dev:
      - kustomization.yaml
      - deployment.yaml
    - prod:
      - kustomization.yaml
    ...
- .gitlab-ci.yml
- functions.sh

You got the idea. The files under kustomizations directory are just kustomizations files for your application. The .gitlab-ci.yaml file is our build script for env, which is responsible for deploying and promoting our application. The functions.sh file are bash script helper functions to ease writing the gitlab build file.

Suggesting new Image

Second, we need to add a step to our application repository to send the newly created image to the environment repository:

 suggest-dev:
  tags:
  image: image-with-git:v2.13.1
  stage: suggest-image
  script:
    - export DOCKER_IMAGE="$DOCKER_IMAGE_NAME":`cat ./build/docker_image_tag_value.txt`
    - export GIT_ANNOTATION_TAG_VALUE=`cat ./build/docker_image_tag_value.txt`
    - git config --global user.email "xxx"
    - git config --global user.name "xxx"
    - git clone https://oauth2:${GITLAB_ACCESS_TOKEN}@gitlab.ca/env-repo.git
    - cd pick-manager-env/kustomizations/overlays/dev
    - git checkout candidates
    - curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash
    - chmod +x ./kustomize
    - mv ./kustomize /usr/local/bin/kustomize
    - /usr/local/bin/kustomize edit set image application-image=$DOCKER_IMAGE
    - /usr/local/bin/kustomize edit set image order-manager-image=$DOCKER_IMAGE
    - message="CI Pipeline tagged commit '${CI_COMMIT_SHORT_SHA}' [commit sha - ${CI_COMMIT_SHA}] on ref '${CI_COMMIT_REF_NAME}' in build id '${CI_PIPELINE_ID}' of '${CI_PROJECT_NAME}' \nupstream message:\n "`git log -1 --pretty=%B`
    - git commit -a -m "$message"
    - git push --set-upstream origin $newBranch -o merge_request.create -o merge_request.target=master -o merge_request.merge_when_pipeline_succeeds -o merge_request.label="$newBranch" -o merge_request.remove_source_branch
  only:
    - master

Deploying Images

In the environment repository we have the bulk of stuff that we need. First of all we need couple of shell functions. The reason is there are many repetitive scripts to run, and they are longish.

    function install_kubetools {
        echo " installing kube tools"
        gcloud auth list
        gcloud container clusters get-credentials --region us-east1 --project ld-shipyard prod-bluenose
        cd /tmp
        curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash
        chmod +x ./kustomize
        mv ./kustomize /usr/local/bin/kustomize
        curl -LO "https://storage.googleapis.com/kubernetes-release/release/v1.17.0/bin/linux/amd64/kubectl"
        chmod +x ./kubectl
        mv ./kubectl /usr/local/bin/kubectl
        cd -
    }

This function is to install kubectl and kustomize on the image. You can easily bake this in a custom image and use that as your build image.

    function deploy {
    /usr/local/bin/kustomize build . > target.yaml
    echo "Applying the following target kubernetes yaml:"
    cat ./target.yaml
    /usr/local/bin/kubectl apply -f ./target.yaml
    echo "sleeping for 4 seconds"
    sleep 4

The second function is to deploy the image to your cluster. This can also be done by directly invoking kubectl -k so it is no strictly necessary. I used wrote it this way to log the generated templates.

    function check_success {
        for i in 1 2 3 4 5 6
        do
            /usr/local/bin/kubectl -n $1 rollout status deployment name-of-deployment
            if [ $? -eq "0" ] ; then
                return 0
            else
                echo "waiting for deployment"
            fi
            sleep 10
        done
        echo "Deployment failed"
        return -1

    }
This function checks to see if the deployment was successful. It is rudimentary, but does the job. You are going to need to replace the name of the deployment with yours.

With all of this function in hand, implementing the actual pipeline becomes rather simple.

stages:
- deploy
- promote

deploy-dev:
  stage: deploy
  script:
    - echo "Going to deploy to dev"
    - . ./functions.sh
    - install_kubetools
    - cd kustomizations/overlays/dev
    - deploy
  
deploy-uat:
  stage: deploy
  script:
  - . ./functions.sh
  - install_kubetools
  - export newTag=`find_current_image`
  - export message="$newTag promoted to UAT"
  - cd kustomizations/overlays/uat
  - echo "setting image tag to $newTag"
  - /usr/local/bin/kustomize edit set image image-name="some-image:$newTag"
  - git_commit kustomization.yaml  "$message"
  - deploy
  - cd -
  - check_success environment
  - git push --set-upstream origin master

  only:
    refs:
      - master
  when: manual

Please leave a comment, and let me know what you what you think.

Thanks for reading!