02 February 2018

Docker for developers: part 2

Tags: Docker

In a previous post, I wrote about some useful commands developers should know when using docker as a development environment. In this post, I'll add two more topics, namely multi-stage builds and permission problems with host mounted volumes. Let's start with docker multi-stage builds.

Docker multi-stage  builds

In docker version 17.05 multi-stage builds were introduced which simplified life for us who used docker in production and testing. Back when applications, where deployed to production servers over SCP or similar, production servers were kept lean. Only packages related to the apps where added to the OS and nothing else. Keeping the production OS lean was a way of reducing the attacker surface and thus smaller probabilities of vulnerabilities.

Similar to that logic when you deploy containers to production you want the images to be as lean as possible. Build-essentials in the container? No way.
However, some packages require that you have compilers installed when doing package installs. The way to solve this puzzle prior to docker multi-stage builds was to have two separate Dockerfiles. One for building the binaries and one for running the actual application. With docker multi-stage builds you can now combine the two dockerfiles in one and you will end up with a lean image for production. An example might be useful to show this

20 January 2018

Docker for developers

Tags: Docker

With the rise of microservices, the complexity of dev environments has increased. Technology stacks can be used independently and thus the choices of programming languages have increased. Companies used to be proficient in one language and implement their solutions using the tools/libraries available to them in the specific language in a monolithic way. Today with containers it's totally different. If you are a Ruby on Rails shop but need to implement a real-time service you might dip into NodeJS. Or you are a PHP shop and need to implement a high traffic service you might look into Golang. Or you have been using python2 but want the new service to use asyncio (python3). Now imagine the time it will take onboarding developers and setting up their dev environments. That can be quite time-consuming for the developer and for you! So what is the solution? Docker of course :)

Usually, docker is utilized for CI (building / testing / QA..etc) but I think that every project should begin with a Dockerfile and a docker-compose file. That way when somebody new comes and joins the team he would just need to do a docker-compose up to start his work. NodeJS 6 or 8? No problem just define it in the Dockerfile and the developer doesn't have to bother with installing anything.

For optimum results, the developer has to know some useful docker commands. In this post, I intend to go over some of the good to know docker commands that I have found myself using on a regular basis while using docker as my development environment.

docker in dev envs

12 December 2017

Aerospike vs Redis vs Tarantool

I have been working on a project where we have to store application data (clicks, views, mentions,...etc) in order to do some analytics later on on the data. There are several ways to architect a solution for this kind of problem and although in the past I architected a similar solution to this problem using Apache Kafka and the Citus extension for PostgreSQL this time we decided to use an in-memory storage engine and implement a batch job that regularly aggregates the data and stores it in PostgreSQL.

The question was which in-memory storage engine should we go for? In this post, I will compare three promising tools and do some performance testing on them in order to better evaluate which one to go with. Before I start I have to mention that I am slightly biased towards Redis because I have used it regularly in the past and know my way around it but after having now worked with both Aerospike and Tarantool I have to say that all of them are worthy candidates.

Test Setup

The tests are run on AWS. I use terraform to provision the infrastructure. The picture below shows an overview of the infrastructure. The tools were provisioned as single instances on their own nodes:
YCSB-tester on  a c4.2xlarge (This instance was used to run YCSB tool to perform the tests)
Redis-tester(v. 4.0.6) on a m4.xlarge
Tarantool-tester(v. 1.8) on a m4.xlarge
Aerospike-tester(v. on a m4.xlarge
Elasticache-tester(v. 3.2.4) on a cache.m4.xlarge


30 November 2017

AWS Fargate

In re:Invent 2017 AWS announced AWS Fargate among many other new services. AWS describes Fargate as

A technology that allows you to use containers as a fundamental compute primitive without having to manage the underlying instances

Image Source and Credits (AWS Blog)

I have experience with ECS and Kubernetes and the underlying machines haven't posed a problem for me to manage. In Kubernetes draining a node for maintenance or update is achieved via the CLI

kubectl drain <node name>

30 September 2017

Fix slow container shutdown

Tags: Docker

Today I was working in a dev environment where everything was done within containers and docker-compose was used to define the services. Usually, when I work on my own projects in this way I don't have to rebuild containers every time I make changes to my code to reflect those changes but in this special case, there was no getting around it. Every time I made changes to the code I had to rebuild the container and run it again. The annoying part was that it took such a long time to re-up the container because the stop process took such a long time. I investigated it and it seems I have been running my apps in the container as the PID 1 process. How does this happen?

There are two ways of running your containers. With CMD or ENTRYPOINT or a combination of both. For example


FROM alpine:3.6

CMD ["ping"]


12 June 2017

Building and Shipping Docker images

Most of the time we don't have to deal with building and shipping docker images because the community has already built quite a number of useful images for us. However, if you practice DevOps with Docker (you should) chances are you have to build custom images for your services. Maybe some developers in your organization have a service running NodeJS, maybe others are writing Java APIs, obviously for these services you have to build your own images and today I would like to describe two ways of building and shipping docker images.

Note that I haven't used the word container in the above paragraph although in many places online the word container is sometimes used interchangeably with the meaning of images. Which is not correct. A Docker container is a running instance of a Docker image. A Docker image can be considered as the "source code" for the container. Docker images are made up of union file systems layered over each other and are build step-by-step using a series of instructions. We will see this in action now by building an image in two ways.

Image Source and Credits (Fabio Ferrari)

13 March 2017

Web app deployment without downtime

Tags: Docker

Hi there today I'll show you how I deploy my site to production without incurring downtime. My site isn't big and is deployed on a single server where I have NGINX, HAproxy and redis running. Here a diagram that shows the setup

service diagram

23 February 2017

Awesome Docker

Tags: Docker

Docker is a disrupting technology for software delivery, infrastructure, and development. Just look at the adoption numbers (courtesy to datadog)

docker adoptions

But let us step back for a second and answer the common question, What is Docker?