24 June 2019

Streaming kubernetes logs from multiple replicas

I have been really busy these days with kubernetes. Not only managing it but also developing microservices on top of it. One thing that has been particularly annoying is viewing logs in the command line from multiple replicas. First I installed stern and then I tried kubetail. Both are fine but require me to remember some commands on top of all the commands I have to remember using kubectl. 

I was reading this document the other day and noticed the paragraph:

Begin streaming the logs from all containers in pods defined by label app=nginx
kubectl logs -f -l app=nginx --all-containers=true

I am not sure when this feature was added but I wanted to try it out and see if it was working the way I expected it. So I tried it out on one of the microservices I am working on.
I have a microservice called user-information it is of kind deployment and has the replica count set to 3. I want to stream all pod logs of this service so I tried:

kubectl logs -f --namespace user-information -l app=user-information

and see there all logs from all pods displayed in my console! Nice. No more external tools, just plain kubectl  and grep


If your services are structured and all of them have labels (which they should) you can further simplify the above command by adding a function to your ~/.functions file. Open the ~/.functionsfile with your preferred editor and paste the following block inside it:

logs() {
  kubectl logs -f -n $1 -l app=$1 --max-log-requests 5 --all-containers=true --timestamps=true

in a new terminal I can then simply view the logs with:

logs user-information


Unfortunately, this works only on deployments / statefulSets with replication count <= 5. If you try the above and encounter this

error: you are attempting to follow 6 log streams, but maximum allowed concurrency is 5

You might need to resort to something else then kubectl.

05 May 2018

Kubernetes in a private AWS network

I have been working on setting up a kubernetes cluster on AWS. Usually, the setup isn't difficult and there are many tools that can assist, for example, kops, spray, conjure-up and probably many others I am forgetting. The problem I had with these tools is that they are configured to create all the resources a kubernetes cluster might need. For example, trying to use kops to create a cluster in a private subnet fails if no Internet gateway exists, thus it will try to create an internet gateway. But what if you are in a corporate environment and no internet gateway has been provisioned? What if the internet breakout should go through your own data center? Now your options are limited to:

  1. Setting up the cluster manually (hard way)

  2. Dig deep into the kops / spray code and modify it to do what you need 

Obviously, both options are time-consuming. What if there is another way? The semi-automated way. Actually, the creators of kubernetes thought about the case where flexibility and customizability are needed. That's why they gave us kubeadm currently my go-to tool for provisioning and managing kubernetes clusters. I call it the semi-automated way because I have to ssh into the master / nodes and issue kubeadm commands and write some config files but with a little bash scripting and terraform knowledge it's rather easy to automate everything.