05 May 2018

Kubernetes in a private AWS network

I have been working on setting up a kubernetes cluster on AWS. Usually, the setup isn't difficult and there are many tools that can assist, for example, kops, spray, conjure-up and probably many others I am forgetting. The problem I had with these tools is that they are configured to create all the resources a kubernetes cluster might need. For example, trying to use kops to create a cluster in a private subnet fails if no Internet gateway exists, thus it will try to create an internet gateway. But what if you are in a corporate environment and no internet gateway has been provisioned? What if the internet breakout should go through your own data center? Now your options are limited to:

  1. Setting up the cluster manually (hard way)

  2. Dig deep into the kops / spray code and modify it to do what you need 

Obviously, both options are time-consuming. What if there is another way? The semi-automated way. Actually, the creators of kubernetes thought about the case where flexibility and customizability are needed. That's why they gave us kubeadm currently my go-to tool for provisioning and managing kubernetes clusters. I call it the semi-automated way because I have to ssh into the master / nodes and issue kubeadm commands and write some config files but with a little bash scripting and terraform knowledge it's rather easy to automate everything.

22 December 2017

When to use DynamoDB and when not to

Having seen projects that use DynamoDB as their persistence layer with good results, I have also seen many projects where DynamoDB wasn't suitable but was used regardless. Upon asking the question why DynamoDB was chosen the answers were usually unclear and defensive. All good Architects and Engineers should ask themselves before using a new technology what the benefits are, the technology will bring and if these benefits warrant the inclusion of the technology.
In this post, I will list a couple of answers to the simple question "Why did you go with DynamoDB" with my remarks based on my experience with DynamoDB. Let's start with the most common answer:

We don't have to think about scaling the system it is done for us

Yes, scaling is done behind the scenes by AWS, for every 10GB of data a new Node is added but what people forget is that when you initially provision 30 RCU/WCU and suddenly your data grows and you have 2-3 Nodes the provisioned RCUs/WCUs is a total of all Nodes. Hence if a query throws a ProvisionedThroughputExceededException you can't provision more throughput on a Node by Node basis you will have to overprovision your throughput or re-design your data model. Also with the newly introduced auto-scaling feature for provisioned throughput, care must be taken to not exceed budget limits because of a bad initial data model design.

30 November 2017

AWS Fargate

In re:Invent 2017 AWS announced AWS Fargate among many other new services. AWS describes Fargate as

A technology that allows you to use containers as a fundamental compute primitive without having to manage the underlying instances

Image Source and Credits (AWS Blog)

I have experience with ECS and Kubernetes and the underlying machines haven't posed a problem for me to manage. In Kubernetes draining a node for maintenance or update is achieved via the CLI

kubectl drain <node name>

16 September 2017

AWS Network Load Balancer

Tags: AWS Cloud

The other day I received an email from AWS with their latest announcements. I went through the email and saw that AWS now offers a new kind of ELB namely the Network Load Balancer (NLB). Compared to the ALB and the ELB classic the NLB is a layer 4 load balancer (transport). When I clicked the link, the AWS webpage states the following "The Network Load Balancer for the Elastic Load Balancing service is designed to handle millions of requests per second while maintaining ultra-low latencies" and I thought to myself why didn't AWS offer this ELB a year ago when I needed a network load balancer :)

A year ago I worked on a project where the requirements were to capture NetFlow traffic and query CPEs (Customer Premises Equipment) of their status through the SNMP protocol. The SNMP part wasn't that big of a challenge but the NetFlow was more of a challenge. Having 10.000 CPE each with at least 10 interfaces bombing your backend is quite a challenge. One of the requirements was data integrity since the data would be used in a commercial manner. Our initial design consisted of having 5x EC2 instances with NetFlow parsers that would parse and forward (produce) the data to an Apache Kafka cluster. The following picture shows a simplified design

15 January 2017

State of the Cloud

Tags: AWS Cloud

Cloud technologies and services are evolving rapidly and startups are not the only adopters anymore. I can see that many German companies are moving their infrastructure to the cloud, either entirely or partly (hybrid clouds). 

In this post I want to take you for a cloud journey! Follow along if you have time.