Highly available and scalable Kubernetes on AWS #builderscon

More than 1 year has passed since last update.

Table of contents

  1. A bit of context

  2. About this presentation

  3. Overview of Kubernetes on AWS

  4. Detailed guide for kube-aws

A bit of context

  •  Kubernetes :point_left:

  •  AWS

  •  Why Kubernetes on AWS

  •  Why they matter to you

:shell: Kubernetes in a nutshell

An open-source system for automating deployment, scaling, and management of containerized applications.


:shell: Kubernetes does

  • Automatically keep your applications running according to your requirements

    • Types of an application: long-running/one-shot, stateful/stateless, standalone/distributed

    • Required resources like CPU, memory, storage, machine, network capability, permissions, etc.

:shell: Kubernetes does

  • Provides APIs to CRUD desired states of your applications

  • Converges your applications to their desired states

Kubernetes Alternatives

  • Docker Swarm

  • Hashicorp Nomad

  • Kontena

  • AWS ECS(EC2 Container Service)

  • Mesos

  • etc.

3 catchphrases of Kubernetes

  • Planet Scale

  • Never Outgrow

  • Runs Anywhere

:globe_with_meridians: "Planet Scale" with Kubernetes

Designed on the same principles that allows Google to run billions of containers a week


:lifter: You "Never Outgrow" Kubernetes

Whether testing locally or running a global enterprise, Kubernetes flexibility grows with you to deliver your applications consistently and easily no matter how complex your need is


How we can never outgrow?

  • Model your application in Kubernetes way:

    • Containers, Pods, Jobs, ReplicaSets, Deployments, StatefulSets, etc.

Kubernetes Models 1/4

  • Containers = cgroups + namespaces w/ copy-on-write

  • Pods = VMs in the container era w/ shared CPU, memory, volumes, permissions

Kubernetes Models 2/4

  • Services = Named sets of pods load balanced via kubernetes internal DNS


Kubernetes Models 3/4

  • Jobs = One-shot or long-running jobs backed by pod(s), with optional parallelism

  • ReplicaSets = A set of identical pods with its number maintained

Kubernetes Models 4/4

  • Deployments = Desired state of your ReplicaSet

  • StatefulSets = A set of named, persistent pods to host e.g. stateful distributed system like databases, filesystems

:airplane: Kubernetes "Runs Anywhere"

Kubernetes is open source giving you the freedom to take advantage of on-premise, hybrid, or public cloud infrastructure, letting you effortlessly move workloads to where it matters to you.


Anywhere = ?

  • Run your Kubernetes locally for development with minikube

  • On IaaSes including Baremetal, GCP, AWS, OpenStack, etc.

  • On Ubuntu, CentOS, CoreOS, etc.

Context: AWS

  • :ballot_box_with_check: Kubernetes

  •  AWS :point_left:

  •  Why Kubernetes on AWS

  •  Why they matter to you

AWS is a leader in IaaS

:link: AWS named as a leader in the Infrastructure as a Service (IaaS) Magic Quadrant report for 6th consecutive year

AWS is IaaS

  • VPS

  • IaaS :point_left:

  • PaaS

  • SaaS

:computer: VPS

  • Virtual Private Server

  • Equipped with a common set of services(including installed softwares, public IPs, storages, CPUs, memories, etc.)

  • e.g. DigitalOcean, さくらのVPS

:gear: IaaS

  • Infrastructure as a Service

  • Serves you a series of resources and APIs to build your own virtual data centers

  • e.g. Azure, GCP, AWS, etc

:station: PaaS

  • Platform as a Service

  • Serves you a platform to run your application without taking care of the infrastructure

  • e.g. Heroku, Google App Engine

:fork_knife_plate: SaaS

  • Software as a Service

  • Serves you software(s) or managed applications to assists part of your business(es)

Context: Why Kubernetes on AWS

  • :ballot_box_with_check: Kubernetes

  • :ballot_box_with_check: AWS

  •  Why Kubernetes on AWS :point_left:

  •  Why they matter to you

Recap: What is Kubernetes?

  • Kubernetes is all about how to run your application in an unified way regardless of where it runs: from home to a datacenter, from a laptop to a rack of blade servers

Kubernetes on What?

  • VPS can be used to run Kubernetes

  • There're PaaSes to host your Kubernetes clusters

  • There're SaaSes hosted by Kubernetes

  • There're IaaSes provides foundations to build/host your Kubernetes clusters

So: Why Kubernetes on IaaS?

  • IaaS allow us to build our own PaaS based on Kubernetes to host our SaaS according to our requirements

Context: Why they matter to you

  • :ballot_box_with_check: Kubernetes

  • :ballot_box_with_check: AWS

  • :ballot_box_with_check: Why Kubernetes on AWS

  •  Why they matter to you :point_left:

:ticket: builderscon is

  • IMHO builderscon is almost a conference for programmers.

  • Why all those things matter to us programmers?

From a programmer's viewpoint: API

  • Kubernetes provides a rich set of APIs to manage Kubernetes resources

  • AWS provides a rich set of APIs to manage AWS resources

From a programmer's viewpoint: Kubernetes and AWS

  • Kubernetes is a just framework to program how my applications are managed

  • AWS is a just framework to program how my Kubernetes clusters are managed

:keyboard: I'd like to program apps, not manage them

  • Kubernetes allows us to focus on programming

  • Also: Kubernetes and its surrounding ecosystem themselves would be interesting to be programmed :wink:

:ok_woman: Now, you understand the context

  • :ballot_box_with_check: Kubernetes

  • :ballot_box_with_check: AWS

  • :ballot_box_with_check: Why they matter to you

Let's keep going :muscle:

Table of contents

  1. A bit of context

  2. About this presentation :point_left:

  3. Overview of Kubernetes on AWS

  4. Detailed guide for kube-aws

About this presentation

  • Topics explained in this presentation

  • Questions answered in this presentation

:information_source: Topics explained in this presentation

  • Overview of Kubernetes on AWS

  • Detailed guide for kube-aws

  • Related personal experience(s) from the presenter

:question: Questions answered in this presentation

  • How to achieve high scalability w/ Kubernetes on AWS?

  • How to achieve high availability w/ Kubernetes on AWS?

  • Are there any tool(s) to deploy and manage Kubernetes clusters?

  • How to select tool(s) for your use?

    • Requirements which arises frequently today

    • What is kube-aws? How to use it?

  • What's my use-case?

    • Requirements

    • Tools

Table of contents

  1. A bit of context

  2. About this presentation

  3. Overview of Kubernetes on AWS :point_left:

  4. Detailed guide for kube-aws

Overview of Kubernetes on AWS

  • How to achieve high availability and scalability

  • Tools

  • How to select those tools

High availability with Kubernetes on AWS

  • High availability in nutshell

  • How to achieve it

:shell: High availability in nutshell

Entirely/Almost no downtime when a expected/unexpected shutdown of 1 or more nodes and pods

  • Pods = Sets of containers running on EC2 instances

  • Nodes = EC2 instances

High availability and Redundancy

  • Pods = Sets of containers

    • 2 or more identical pods

    • ReplicaSets in Kubernetes

  • Nodes = EC2 instances

    • 2 or more nodes usable to host the same set of pods

    • Auto Scaling Group and/or Spot Fleet in AWS

High scalability with Kubernetes on AWS

  • High scalability in nutshell

  • How to achieve it

:shell: High scalability in nutshell

No redundant cost + No user visible frustration in less/more workload

  • Pods = Sets of containers running on EC2 instances

  • Nodes = EC2 instances

Cost = Number of nodes * unit price

Number of nodes = Total resources required by pods

High scalability for pods

High scalability for nodes

  • Nodes = EC2 instances

    • Less nodes when less pods, more pods when more workload

    • Kubernetes's cluster-autoscaler

      • Adds nodes to meet capacity enough for scheduling all the pending pods

    • AWS Auto Scaling

      • Adds/removes nodes according to CPU and/or memory usage

:tools: Are there any tool(s) to deploy and manage Kubernetes clusters?

  • Yes,

  • A lot

    • klondike, kubernetes-anywhere, kops, kube-aws, kope, kube-cluster, kubeadm-aws, halcyon-kubernetes, tack, etc.

  • :bookmark: An overview of those tools

:question: How to select tool(s) for your use?

Selecting a tool is all about:

  • knowing your requirements,

  • investing in tools, and then

  • connecting them

Knowing your requirements

Requirements which aries frequently today:

  • Highly available

  • Highly scalable

  • Highly secure

  • Be locked in to technologies give you higher leverage

Lock-in to what gives you higher leverage?

  • An IaaS frequently reduce pricing

  • An IaaS frequently add new features, APIs, SDKs, etc., which are ready to be used for your business

  • // AWS for me

Table of contents

  1. A bit of context

  2. About this presentation

  3. Overview of Kubernetes on AWS

  4. Detailed guide for kube-aws :point_left:

Detailed guide for kube-aws

  • For all:

    • kube-aws in a nutshell

  • For users:

    • Usage guide for kube-aws

  • For developers:

    • Development guide for kube-aws

  • For potential contributors:

    • Contribution guide for kube-aws

:shell: kube-aws in a nutshell

  • A single binary to create/update/destroy Kubernetes cluster(s) on AWS

  • Infrastructure as Code achieved via AWS CloudFormation + Golang

  • Originally developed by CoreOS, currently maintained by community

:hamburger: Under the hood of kube-aws

  • Standard Golang application based on Golang 1.7, glide, cobra, etc., coupled w/

  • CoreOS, which has official support for running Kubernetes

  • Kubernetes 1.4.x, upstream releases followed timely

  • Various AWS services

    • VPC, EC2, ELB, Route53, KMS, CloudFormation

:no_entry: kube-aws does not use

  • Terraform

  • Ansible

  • Chef, Puppet, Saltstack, etc.

Usage guide for kube-aws

Create your first Kubernetes cluster on AWS :smile:

* Install kube-aws

* Generate cluster.yaml

* Customize your cluster

* Render the contents of the assets directory

* Validate your configuration

* Updating the cluster without downtime

* Destroying the cluster

Creating your first Kubernetes cluster on AWS

  • Create keys

  • Install the latest version of kube-aws command

  • Customizing the cluster

Prepare keys

  • EC2 Keypair

    • Used to authenticate SSH access to your server

  • KMS key

    • Used to encrypt/decrypt TLS assets generated by kube-aws

Install the kube-aws command

$ platform=darwin-amd64

$ version=v0.9.2-rc.1
$ curl -L${version}/kube-aws-${platform}.tar.gz | tar zxv ${platform}/kube-aws && \
mv ${platform}/kube-aws /usr/local/bin/kube-aws && \
chmod +x /usr/local/bin/kube-aws
$ kube-aws version
kube-aws version v0.9.2-rc.1


Customizing the cluster

Generate a cluster.yaml:

$ mkdir my-cluster

$ cd my-cluster

$ kube-aws init \
--cluster-name=my-cluster-name \
--external-dns-name=my-cluster-endpoint \
--region=us-west-1 \
--availability-zone=us-west-1c \
--key-name=key-pair-name \

Customizing the cluster


clusterName: my-cluster-name

releaseChannel: stable
createRecordSet: true
recordSetTTL: 300
hostedZoneId: "XASDASDASD" # DEV private only route53 zone
keyName: my-ssh-keypair
region: ap-northeast-1
availabilityZone: ap-northeast-1c
kmsKeyArn: "arn:aws:kms:ap-northeast-1:0w123456789:key/d345fcd1-c77c-4fca-acdc-asdasdf3234232"
controllerCount: 1
controllerInstanceType: m3.medium
controllerRootVolumeSize: 30
controllerRootVolumeType: gp2
workerCount: 1
workerInstanceType: m3.medium
workerRootVolumeSize: 30
workerRootVolumeType: gp2
etcdCount: 1
etcdInstanceType: m3.medium
etcdRootVolumeSize: 30
etcdDataVolumeSize: 30
vpcId: vpc-xxxccc45
routeTableId: "rtb-aaabbb12" # main external no NAT uses internet gateway
vpcCIDR: ""
instanceCIDR: ""
serviceCIDR: ""
podCIDR: ""
Name: "my-cluster-name"
Environment: "development"

Render the contents of the assets directory

$ kube-aws render

$ tree
├── cluster.yaml
├── credentials
│ ├── admin-key.pem
│ ├── admin.pem
│ ├── apiserver-key.pem
│ ├── apiserver.pem
│ ├── ca-key.pem
│ ├── ca.pem
│ ├── worker-key.pem
│ └── worker.pem
│ ├── etcd-key.pem
│ └── etcd.pem
│ ├── etcd-client-key.pem
│ └── etcd-client.pem
├── kubeconfig
├── stack-template.json
└── userdata
├── cloud-config-controller
└── cloud-config-worker

Credential files

  • admin*.pem, used on your local machine and/or build servers

  • worker*.pem, used in worker nodes

  • apiserver*.pem, used in controller nodes

  • etcd-client*pem, used in worker and controller nodes

  • etcd*pem, used in etcd nodes

Credential file naming

  • foo-key.pem

    • The private key for foo access

  • foo.pem

    • THe public key for foo access

That's a lot!

  • Every little mistake in configuration would result in non-functional clusters

  • That's why we need kube-aws validate :smile:

Validate your configuration

$ kube-aws validate

  • Fails only when there're syntax and/or semantic errors in your configuration files

    • Especially in:

      • cloud-config

      • CloudFormation templates

      • No need to wait for minutes until you find easy mistakes like typos :wink:

It is just "validating"

  • kube-aws validate doesn't(can't) test functional correctness of the resulting Kubernetes cluster

  • It is covered by our End-to-end testing

Launch your cluster

$ kube-aws up

Creating AWS resources. This should take around 5 minutes.
Success! Your AWS resources have been created:
Cluster Name: my-cluster-name

The containers that power your cluster are now being downloaded.

You should be able to access the Kubernetes API once the containers finish downloading.

Or just export your cluster

$ kube-aws up --export

  • Generates a CloudFormation stack template

  • A template is a JSON file which defines which AWS resources to create and how they're connected.

Destroy your cluster

$ kube-aws destroy

Development guide for kube-aws

  • Understanding how kube-aws is built

  • Build your own kube-aws

  • Make targets

  • End-to-end testing in kube-aws

Understanding how kube-aws is built

  • golang 1.7.x

  • glide for package management

    • Alternatives: Godep, trash, govendor, etc.

  • go gen + go run for code generation

    • To embed the default configuration files into kube-aws binaries

  • go test for unit testing

  • gofmt for code formatting and validation

Build your own kube-aws

  • Install required tools

  • git clone

  • make build

Installing required tools

brew update

brew install go

brew install glide
brew install make

Clone the repository to an appropriate path

mkdir -p $GOPATH/src/

git clone \


Build a kube-aws binary

$ make build

Building kube-aws 102cb5b8c9d71af1a5074b77955c4913e7d0b84d

$ bin/kube-aws version
kube-aws version 102cb5b8c9d71af1a5074b77955c4913e7d0b84d

FYI: Make targets

  • make build to build bin/kube-aws

  • make format to automatically format all the code of kube-aws

  • make test to run all the unit tests

End-to-end testing in kube-aws

  • Create a main cluster

  • Create a node pool

  • Update the main cluster

  • Run Kubernetes conformance tests (:exclamation:)

Kubernetes conformance tests

  • are a subset of Kubernetes E2E tests

  • to confirm that a minimum feature set of functions required for a cluster to be a Kubernetes cluster is properly working

Kubernetes E2E tests

End-to-end (e2e) tests for Kubernetes provide a mechanism to test end-to-end behavior of the system, and is the last signal to ensure end user operations match developer specifications.

:link: End to end testing in Kubernetes

Why E2E tests are needed

Although unit and integration tests provide a good signal, in a distributed system like Kubernetes it is not uncommon that a minor change may pass all unit and integration tests, but cause unforeseen changes at the system level.

:link: End to end testing in Kubernetes

Running End-to-end tests

  • cd e2e && KUBE_AWS_CLUSTER_NAME=my-cluster-name ./ all

  • It takes approximately 1 hour to run each test case :cry:

  • A good news is that we run it against multiple combinations of settings and options

  • Before releasing new kube-aws releases

  • For you

Contribution guide for kube-aws

  • Where to start: GitHub issues

  • What are discussed and answered

  • Please file your own issue(s)!

Where to start

Our github issues page is the single place to start your contribution:

Discussed and answered in github issues

  • Feature Request

  • Proposal

    • For possible features

    • For possible improvement(s) in the documentation

  • Bug reports

  • Questions

What a proposal would look like


Please file your own issue(s)!

Every little feedback is welcomed to shape our goals and plans toward production-ready Kubernetes clusters on AWS


Thanks for your attention and interest on Kubernetes on AWS,

From @mumoshu, the primary maintainer of kube-aws