Implementing graceful shutdown for docker containers in go (part 1)

As part of ensuring the servers I write are compatible with docker and kubernetes I wanted to ensure all my http servers shutdown gracefully so as not to drop any transactions. This is especially useful when scaling down in kubernetes or as part of rolling deploys of new pods, ensuring your containers have the best chance to not drop any transactions.

The standard shutdown procedure for a docker container is to send the container a SIGTERM to indicate a desire to shutdown, if the container doesn’t shutdown it is issued with a SIGKILL after a grace period. As such we need to hook the SIGTERM and ensure we finish all our communications.

My solution for this is as per the following gist:

There are a couple of important things to point out here. Firstly you will notice the use of manners, which is a wrapper for the standard http server to allow for graceful shutdown.

You will also notice the all important hooks for both SIGTERM and also Ctrl+C, which makes things a lot easier for quick tests and also operation outside of docker.

This approach ends up working nicely, you can boot a number of instances of this server and for the most part scale up and down without seeing any dropped requests. However I have noticed that when I scale to a single instance I do see some errors for a very brief period. I’m not sure if this is due to it being a single instance, or just less than the number of test nodes I was running but I’m wondering if there is some issue around etcd timing its changes across the cluster as requests come in.

Overall the approach works for the use case in mind, as a cluster system I don’t expect to run single instances of services that get scaled up and down rapidly, but I’m hoping I can investigate more to find out where the issue is coming from.

3 thoughts on “Implementing graceful shutdown for docker containers in go (part 1)

  1. Hello! Things may get very interesting when it comes to graceful shutdown, I’ve been working on these problems too for my servers. Finally I came with a solution similar to “defer”, but instead it ensures that certain functions are called upon app’s exit. Check this: https://github.com/xlab/closer and its examples for a general picture.

    I don’t like the way github.com/braintree/manners did their Close() function, indeed it SHOULD lock, but instead we must add a channel to ensure that ListenAndServe is done. Instead of just 1 additional line of code I’ve got a few for a channel: https://gist.github.com/2ea988fca7c8e0129f4a and an additional function that checks the channel upon exit.
    This looks messy. I just submitted an issue https://github.com/braintree/manners/issues/31 (check out the example with BlockingClose() that looks like a candy)

    Finally, it worth mentioning two things in this article:
    1) Docker by default will SIGKILL any container after 10 seconds of waiting. So stopping these apps with a 60-second override may be a good option: docker stop -t 60
    2) The snippet in the article lacks of “time” import. Also many people may run this main.go using “go run main.go” which is INCOMPATIBLE with signals catching, I just wasted ~8 minutes of my time trying to figure out why neither yours or mine code doesn’t block! 😀

    Have fun.

    1. Cheers for pointing out that missing time import, thats patched up now.

      Interestingly part of my work was to ensure that I had a shutdown method in docker that was safe, but my focus was making sure this also works in kubernetes inside the cluster. Though the shutdown as above works I had to do some extra work to ensure the proxy didn’t route requests to a container that was not ready to serve them, I just uploaded that here: http://www.bite-code.com/2015/07/27/implementing-graceful-shutdown-for-docker-containers-in-go-part-2/

      I’ll take a look at your code, I agree it would be nicer to have both blocking and non blocking close operations as in this case a blocking operation would make it a bit easier to avoid the channels etc.

Leave a Reply