Running minimal golang apps in kubernetes cluster

I’ve recently been working at migrating some of my projects to docker, and ultimately kubernetes to run inside a cluster. It’s an interesting workflow that results in very rapid development cycles and the ability to constantly verify an application via a rolling deploy and testing with GoConvey.

The process looks a bit like this:

  • Write code
  • Build & unit test code
  • Build minimal docker container with statically linked go app
  • Push docker container to local registry
  • Deploy into kubernetes cluster as a rolling deploy

This process ensures that its very easy for developers to constantly be testing in a local environment, which for me is the perfect development setup.

A quick caveat, as this is quite complex to setup I have chosen to skip a few things like the unit tests and rolling deployments. These are well documented elsewhere, but if you get stuck just comment and I’ll do a follow up post.

The setup

So here is the overall setup:

Kubernetes

  • Everything runs on a single computer, in this case my mac
  • The docker registry runs inside a boot2docker VM
  • Kubernetes runs locally via vagrant
  • Kubernetes runs a master and a pair of nodes (minions in the old kubernetes speak)
  • The nodes run 3 copies of our container
    • We don’t care which nodes they run on
  • A service exposes our application to the host machine via a fixed port on both nodes

Note: Blue = The computer, Red = VM, Green = Docker Container, Purple = Relevant parts of kubernetes (there’s more, but its easier to think this way)

First Steps

For doing this work, you need to have:

  • Go 1.4 installed (and setup for cross compilation if you’re not on linux)
  • Git
  • Boot2docker running (for mac users)
  • Make
  • Vagrant & VirtualBox (VirtualBox was tested but should work with any provider)

To get up and running I have created a sample project that contains all the files needed for this work. As you are working in go the best way to get hold of this will be

go get github.com/chriswhitcombe/minimal-docker-go

This will clone the project under your $GOPATH/src/ structure for you ready to be used.

Run your registry

Before we build our code you need to have a working local docker registry. You could skip this stage and push to the public docker repository, but that slows the cycle time for feedback and ideally you should only push releases to that, not just ad-hoc development builds.

Luckily the docker team have made it very easy to get your own local registry up and running. Make sure you have boot2docker ready to go:

boot2docker init
boot2docker up

Then you can setup your registry with one line:

docker run -p 5000:5000 -d --name=basic_registry registry

Finally you need to check the IP this is running on from boot2docker

boot2docker ip

You can then load <ipaddress>:5000 into your browser and confirm you get a message stating you have hit the registry server. Take a note of the IP address and port, you will need this later!

Build the app and docker image

Open a terminal prompt in the minimal-docker-go project and run the make file, this does a few things:

  • This builds your code
    • It cross compiles for linux as the target system
    • It statically links the c libraries go needs
    • It forces all packages we depend on to recompile under these rules also
    • In the end you get a binary for linux that is totally self contained
  • This builds and pushes your docker image to your local docker registry
    • Thanks to our self contained binary we build off the scratch image, resulting in a tiny file
    • Note you may need to change localhost here if you have issues
  • Finally it cleans up the binary to keep things tidy

Once you are done here you should have a local docker registry with a copy of your container, in which it has your code.

Note: The use of the scratch container here is really pretty amazing, you are essentially sitting on top of a kernel with hardly anything else in your way. The final image size ends up being around 6mb for a container, your app, the garbage collector, everything!

Boot kubernetes

Your first job is to download and extract kubernetes from github to somewhere useful on your machine. As noted above we will be using vagrant to simulate our cluster. However before we get started there is one important change to make to the configuration. Normally docker pulls images from a valid https endpoint, however in our example we’re using a simple local registry that doesn’t have a valid https endpoint. We need to ensure that we can pull our docker images in kubernetes.

Open a terminal and cd into the kubernetes folder.

Open kubernetes/cluster/vagrant/config-default.sh and search for the line containing “insecure-registry”, in my release it looks as such:

EXTRA_DOCKER_OPTS="-b=cbr0 --insecure-registry 10.0.0.0/8"

We need to change this so images from our insecure registry are accepted, as such take the IP from your boot2docker VM and update accordingly, mine ended up looking as such:

EXTRA_DOCKER_OPTS="-b=cbr0 --insecure-registry 192.168.59.103:5000"

With that in place, we can bring up our cluster by running:

export KUBERNETES_PROVIDER=vagrant
export NUM_MINIONS=2
./cluster/kube-up.sh

This ensures our script knows to boot via vagrant and creates a cluster with a master and 2 minions (now known as nodes). Go get a cup of tea, this takes a while!

Once its finally booted you can get the IP’s of your nodes

./cluster/kubectl.sh get nodes
NAME       LABELS                            STATUS
10.245.1.3 kubernetes.io/hostname=10.245.1.3 Ready
10.245.1.4 kubernetes.io/hostname=10.245.1.4 Ready

Deploy the pods

There are enough tutorials on kubernetes terminology around on youtube, I won’t go into it. What we will do here is bring up a controller that ensures we have 3 replicas of our container (from our local docker registry) running across our 2 nodes. We don’t care how many run and where, we just care that we have 3.

As can be seen from the gist we run a controller, which runs 3 replicas of our controller. You may need to adjust the IP of your docker registry host accordingly. It’s important to note we are demanding the latest container version, which ensures docker asks our registry for the latest image and doesn’t use its cached copy. To get our cluster into this state we submit this file to kubernetes, it then takes responsibility of booting docker containers on various nodes and keeping the correct number of replicas up. To do this run:

./cluster/kubectl.sh create -f 01-goapp-controller.json

You can check on the progress of this by running:

./cluster/kubectl.sh get pods

In the end your cluster should look like this:

NAME                   READY STATUS  RESTARTS AGE
goapp-controller-08xl7 1/1   Running 0        15s
goapp-controller-33lr2 1/1   Running 0        15s
goapp-controller-ev3bp 1/1   Running 0        15s

Create the services

Now that our pods are running we need to be able to access them. In our example we don’t need access from elsewhere in our cluster, but from outside our cluster, however the approach is the same. We need to create a service, which proxies to our pods.

Most of this is pretty standard, however its important to note that we have specified a type of Nodeport to ensure that the service is exposed on the host machines (in this case our minions/nodes), we have also specified a port to expose this service on just to make our lives easier.  In a real world deployment there would be a managed load balancer in front of the service, but for now this does the job.

As above you can run this config into your cluster as such:

./cluster/kubectl.sh create -f 01-goapp-service.json

And you can confirm its all looking good as such:

./cluster/kubectl.sh get services
NAME          LABELS              SELECTOR      IP(S)          PORT(S)
goapp-service <none>              app=goapp-app 10.247.249.142 80/TCP
kubernetes    component=apiserver <none>        10.247.0.1     443/TCP

Test

Finally we have our service deployed, so we can test. Remember this is now being exposed on our host machines, to get their IP’s just run:

./cluster/kubectl.sh get nodes
NAME       LABELS                            STATUS
10.245.1.3 kubernetes.io/hostname=10.245.1.3 Ready
10.245.1.4 kubernetes.io/hostname=10.245.1.4 Ready

In the above example you can hit either 10.245.1.3:31000 or 10.245.1.4:31000 and our service will respond. Whats really cool is even if our service isn’t running on that node/minion then kubernetes is smart enough to proxy us to another one! You can change the replication to 1 and redeploy, you will still get results from both IP’s!

You can also tear the cluster down as such:

./cluster/kubectl.sh delete -f 01-goapp-controller.json
./cluster/kubectl.sh delete -f 01-goapp-service.json

Summary

Now that you have a cluster up and running you can change code and run make to get a tiny container that is entirely self contained. Then using kubernetes you can quickly tear down and redeploy an entire, highly resilient, cluster of the containers to arbitrary machines.

In the future I will blog about how to test using goconvey and use rolling deploys to ensure zero downtime in your cluster, but hopefully this is a good starting point for those exploring kubernetes and go. Any questions just drop them in the comments.

One thought on “Running minimal golang apps in kubernetes cluster

Leave a Reply