More than 5 years have passed since last update.

posted at

updated at

Highly available and scalable Kubernetes on AWS #builderscon

1 / 89

Table of contents

  1. A bit of context
  2. About this presentation
  3. Overview of Kubernetes on AWS
  4. Detailed guide for kube-aws

A bit of context

  •  Kubernetes :point_left:
  •  AWS
  •  Why Kubernetes on AWS
  •  Why they matter to you

:shell: Kubernetes in a nutshell

An open-source system for automating deployment, scaling, and management of containerized applications.
:link: http://kubernetes.io/

:shell: Kubernetes does

  • Automatically keep your applications running according to your requirements
    • Types of an application: long-running/one-shot, stateful/stateless, standalone/distributed
    • Required resources like CPU, memory, storage, machine, network capability, permissions, etc.

:shell: Kubernetes does

  • Provides APIs to CRUD desired states of your applications
  • Converges your applications to their desired states

Kubernetes Alternatives

  • Docker Swarm
  • Hashicorp Nomad
  • Kontena
  • AWS ECS(EC2 Container Service)
  • Mesos
  • etc.

3 catchphrases of Kubernetes

  • Planet Scale
  • Never Outgrow
  • Runs Anywhere

:globe_with_meridians: "Planet Scale" with Kubernetes

Designed on the same principles that allows Google to run billions of containers a week
:link: http://kubernetes.io/

:lifter: You "Never Outgrow" Kubernetes

Whether testing locally or running a global enterprise, Kubernetes flexibility grows with you to deliver your applications consistently and easily no matter how complex your need is
:link: http://kubernetes.io/

How we can never outgrow?
  • Model your application in Kubernetes way:
    • Containers, Pods, Jobs, ReplicaSets, Deployments, StatefulSets, etc.

Kubernetes Models 1/4
  • Containers = cgroups + namespaces w/ copy-on-write
  • Pods = VMs in the container era w/ shared CPU, memory, volumes, permissions

Kubernetes Models 2/4
  • Services = Named sets of pods load balanced via kubernetes internal DNS


Kubernetes Models 3/4
  • Jobs = One-shot or long-running jobs backed by pod(s), with optional parallelism
  • ReplicaSets = A set of identical pods with its number maintained

Kubernetes Models 4/4
  • Deployments = Desired state of your ReplicaSet
  • StatefulSets = A set of named, persistent pods to host e.g. stateful distributed system like databases, filesystems

:airplane: Kubernetes "Runs Anywhere"

Kubernetes is open source giving you the freedom to take advantage of on-premise, hybrid, or public cloud infrastructure, letting you effortlessly move workloads to where it matters to you.
:link: http://kubernetes.io/

Anywhere = ?
  • Run your Kubernetes locally for development with minikube
  • On IaaSes including Baremetal, GCP, AWS, OpenStack, etc.
  • On Ubuntu, CentOS, CoreOS, etc.

Context: AWS

  • :ballot_box_with_check: Kubernetes
  •  AWS :point_left:
  •  Why Kubernetes on AWS
  •  Why they matter to you

AWS is a leader in IaaS

:link: AWS named as a leader in the Infrastructure as a Service (IaaS) Magic Quadrant report for 6th consecutive year

AWS is IaaS

  • VPS
  • IaaS :point_left:
  • PaaS
  • SaaS

:computer: VPS

  • Virtual Private Server
  • Equipped with a common set of services(including installed softwares, public IPs, storages, CPUs, memories, etc.)
  • e.g. DigitalOcean, さくらのVPS

:gear: IaaS

  • Infrastructure as a Service
  • Serves you a series of resources and APIs to build your own virtual data centers
  • e.g. Azure, GCP, AWS, etc

:station: PaaS

  • Platform as a Service
  • Serves you a platform to run your application without taking care of the infrastructure
  • e.g. Heroku, Google App Engine

:fork_knife_plate: SaaS

  • Software as a Service
  • Serves you software(s) or managed applications to assists part of your business(es)

Context: Why Kubernetes on AWS

  • :ballot_box_with_check: Kubernetes
  • :ballot_box_with_check: AWS
  •  Why Kubernetes on AWS :point_left:
  •  Why they matter to you

Recap: What is Kubernetes?

  • Kubernetes is all about how to run your application in an unified way regardless of where it runs: from home to a datacenter, from a laptop to a rack of blade servers

Kubernetes on What?

  • VPS can be used to run Kubernetes
  • There're PaaSes to host your Kubernetes clusters
  • There're SaaSes hosted by Kubernetes
  • There're IaaSes provides foundations to build/host your Kubernetes clusters

So: Why Kubernetes on IaaS?

  • IaaS allow us to build our own PaaS based on Kubernetes to host our SaaS according to our requirements

Context: Why they matter to you

  • :ballot_box_with_check: Kubernetes
  • :ballot_box_with_check: AWS
  • :ballot_box_with_check: Why Kubernetes on AWS
  •  Why they matter to you :point_left:

:ticket: builderscon is

  • IMHO builderscon is almost a conference for programmers.
  • Why all those things matter to us programmers?

From a programmer's viewpoint: API

  • Kubernetes provides a rich set of APIs to manage Kubernetes resources
  • AWS provides a rich set of APIs to manage AWS resources

From a programmer's viewpoint: Kubernetes and AWS

  • Kubernetes is a just framework to program how my applications are managed
  • AWS is a just framework to program how my Kubernetes clusters are managed

:keyboard: I'd like to program apps, not manage them

  • Kubernetes allows us to focus on programming
  • Also: Kubernetes and its surrounding ecosystem themselves would be interesting to be programmed :wink:

:ok_woman: Now, you understand the context

  • :ballot_box_with_check: Kubernetes
  • :ballot_box_with_check: AWS
  • :ballot_box_with_check: Why they matter to you

Let's keep going :muscle:

Table of contents

  1. A bit of context
  2. About this presentation :point_left:
  3. Overview of Kubernetes on AWS
  4. Detailed guide for kube-aws

About this presentation

  • Topics explained in this presentation
  • Questions answered in this presentation

:information_source: Topics explained in this presentation

  • Overview of Kubernetes on AWS
  • Detailed guide for kube-aws
  • Related personal experience(s) from the presenter

:question: Questions answered in this presentation

  • How to achieve high scalability w/ Kubernetes on AWS?
  • How to achieve high availability w/ Kubernetes on AWS?
  • Are there any tool(s) to deploy and manage Kubernetes clusters?
  • How to select tool(s) for your use?
    • Requirements which arises frequently today
    • What is kube-aws? How to use it?
  • What's my use-case?
    • Requirements
    • Tools

Table of contents

  1. A bit of context
  2. About this presentation
  3. Overview of Kubernetes on AWS :point_left:
  4. Detailed guide for kube-aws

Overview of Kubernetes on AWS

  • How to achieve high availability and scalability
  • Tools
  • How to select those tools

High availability with Kubernetes on AWS

  • High availability in nutshell
  • How to achieve it

:shell: High availability in nutshell

Entirely/Almost no downtime when a expected/unexpected shutdown of 1 or more nodes and pods

  • Pods = Sets of containers running on EC2 instances
  • Nodes = EC2 instances

High availability and Redundancy

  • Pods = Sets of containers
    • 2 or more identical pods
    • ReplicaSets in Kubernetes
  • Nodes = EC2 instances
    • 2 or more nodes usable to host the same set of pods
    • Auto Scaling Group and/or Spot Fleet in AWS

High scalability with Kubernetes on AWS

  • High scalability in nutshell
  • How to achieve it

:shell: High scalability in nutshell

No redundant cost + No user visible frustration in less/more workload

  • Pods = Sets of containers running on EC2 instances
  • Nodes = EC2 instances

Cost = Number of nodes * unit price
Number of nodes = Total resources required by pods

High scalability for pods

High scalability for nodes

  • Nodes = EC2 instances
    • Less nodes when less pods, more pods when more workload
    • Kubernetes's cluster-autoscaler
      • Adds nodes to meet capacity enough for scheduling all the pending pods
    • AWS Auto Scaling
      • Adds/removes nodes according to CPU and/or memory usage

:tools: Are there any tool(s) to deploy and manage Kubernetes clusters?

  • Yes,
  • A lot
    • klondike, kubernetes-anywhere, kops, kube-aws, kope, kube-cluster, kubeadm-aws, halcyon-kubernetes, tack, etc.
  • :bookmark: An overview of those tools

:question: How to select tool(s) for your use?

Selecting a tool is all about:

  • knowing your requirements,
  • investing in tools, and then
  • connecting them

Knowing your requirements

Requirements which aries frequently today:

  • Highly available
  • Highly scalable
  • Highly secure
  • Be locked in to technologies give you higher leverage

Lock-in to what gives you higher leverage?

  • An IaaS frequently reduce pricing
  • An IaaS frequently add new features, APIs, SDKs, etc., which are ready to be used for your business
  • // AWS for me

Table of contents

  1. A bit of context
  2. About this presentation
  3. Overview of Kubernetes on AWS
  4. Detailed guide for kube-aws :point_left:

Detailed guide for kube-aws

  • For all:
    • kube-aws in a nutshell
  • For users:
    • Usage guide for kube-aws
  • For developers:
    • Development guide for kube-aws
  • For potential contributors:
    • Contribution guide for kube-aws

:shell: kube-aws in a nutshell

  • A single binary to create/update/destroy Kubernetes cluster(s) on AWS
  • Infrastructure as Code achieved via AWS CloudFormation + Golang
  • Originally developed by CoreOS, currently maintained by community https://github.com/coreos/kube-aws

:hamburger: Under the hood of kube-aws

  • Standard Golang application based on Golang 1.7, glide, cobra, etc., coupled w/
  • CoreOS, which has official support for running Kubernetes
  • Kubernetes 1.4.x, upstream releases followed timely
  • Various AWS services
    • VPC, EC2, ELB, Route53, KMS, CloudFormation

:no_entry: kube-aws does not use

  • Terraform
  • Ansible
  • Chef, Puppet, Saltstack, etc.

Usage guide for kube-aws

Create your first Kubernetes cluster on AWS :smile:
* Install kube-aws
* Generate cluster.yaml
* Customize your cluster
* Render the contents of the assets directory
* Validate your configuration
* Updating the cluster without downtime
* Destroying the cluster

Creating your first Kubernetes cluster on AWS

  • Create keys
  • Install the latest version of kube-aws command
  • Customizing the cluster

Prepare keys

  • EC2 Keypair
    • Used to authenticate SSH access to your server
  • KMS key
    • Used to encrypt/decrypt TLS assets generated by kube-aws

Install the kube-aws command

$ platform=darwin-amd64
$ version=v0.9.2-rc.1
$ curl -L https://github.com/coreos/kube-aws/releases/download/${version}/kube-aws-${platform}.tar.gz | tar zxv ${platform}/kube-aws && \
  mv ${platform}/kube-aws /usr/local/bin/kube-aws && \
  chmod +x /usr/local/bin/kube-aws
$ kube-aws version
kube-aws version v0.9.2-rc.1

:link: https://github.com/coreos/kube-aws/releases

Customizing the cluster

Generate a cluster.yaml:

$ mkdir my-cluster
$ cd my-cluster

$ kube-aws init \
--cluster-name=my-cluster-name \
--external-dns-name=my-cluster-endpoint \
--region=us-west-1 \
--availability-zone=us-west-1c \
--key-name=key-pair-name \

Customizing the cluster


clusterName: my-cluster-name
externalDNSName: my-cluster-name-api.my.domain
releaseChannel: stable
createRecordSet: true
recordSetTTL: 300
hostedZoneId: "XASDASDASD" # DEV private only route53 zone
keyName: my-ssh-keypair
region: ap-northeast-1
availabilityZone: ap-northeast-1c
kmsKeyArn: "arn:aws:kms:ap-northeast-1:0w123456789:key/d345fcd1-c77c-4fca-acdc-asdasdf3234232"
controllerCount: 1
controllerInstanceType: m3.medium
controllerRootVolumeSize: 30
controllerRootVolumeType: gp2
workerCount: 1
workerInstanceType: m3.medium
workerRootVolumeSize: 30
workerRootVolumeType: gp2
etcdCount: 1
etcdInstanceType: m3.medium
etcdRootVolumeSize: 30
etcdDataVolumeSize: 30
vpcId: vpc-xxxccc45
routeTableId: "rtb-aaabbb12" # main external no NAT uses internet gateway
vpcCIDR: ""
instanceCIDR: ""
serviceCIDR: ""
podCIDR: ""
  Name: "my-cluster-name"
  Environment: "development"

Render the contents of the assets directory

$ kube-aws render
$ tree
├── cluster.yaml
├── credentials
│   ├── admin-key.pem
│   ├── admin.pem
│   ├── apiserver-key.pem
│   ├── apiserver.pem
│   ├── ca-key.pem
│   ├── ca.pem
│   ├── worker-key.pem
│   └── worker.pem
│   ├── etcd-key.pem
│   └── etcd.pem
│   ├── etcd-client-key.pem
│   └── etcd-client.pem
├── kubeconfig
├── stack-template.json
└── userdata
    ├── cloud-config-controller
    └── cloud-config-worker

Credential files

  • admin*.pem, used on your local machine and/or build servers
  • worker*.pem, used in worker nodes
  • apiserver*.pem, used in controller nodes
  • etcd-client*pem, used in worker and controller nodes
  • etcd*pem, used in etcd nodes

Credential file naming

  • foo-key.pem
    • The private key for foo access
  • foo.pem
    • THe public key for foo access

That's a lot!

  • Every little mistake in configuration would result in non-functional clusters
  • That's why we need kube-aws validate :smile:

Validate your configuration

$ kube-aws validate
  • Fails only when there're syntax and/or semantic errors in your configuration files
    • Especially in:
      • cloud-config
      • CloudFormation templates
      • No need to wait for minutes until you find easy mistakes like typos :wink:

It is just "validating"

  • kube-aws validate doesn't(can't) test functional correctness of the resulting Kubernetes cluster
  • It is covered by our End-to-end testing

Launch your cluster

$ kube-aws up
Creating AWS resources. This should take around 5 minutes.
Success! Your AWS resources have been created:
Cluster Name:   my-cluster-name

The containers that power your cluster are now being downloaded.

You should be able to access the Kubernetes API once the containers finish downloading.

Or just export your cluster

$ kube-aws up --export
  • Generates a CloudFormation stack template
  • A template is a JSON file which defines which AWS resources to create and how they're connected.

Destroy your cluster

$ kube-aws destroy

Development guide for kube-aws

  • Understanding how kube-aws is built
  • Build your own kube-aws
  • Make targets
  • End-to-end testing in kube-aws

Understanding how kube-aws is built

  • golang 1.7.x
  • glide for package management
    • Alternatives: Godep, trash, govendor, etc.
  • go gen + go run for code generation
    • To embed the default configuration files into kube-aws binaries
  • go test for unit testing
  • gofmt for code formatting and validation

Build your own kube-aws

  • Install required tools
  • git clone
  • make build

Installing required tools

brew update
brew install go
brew install glide
brew install make

Clone the repository to an appropriate path

mkdir -p $GOPATH/src/github.com/coreos/kube-aws
git clone git@github.com:coreos/kube-aws.git \

Build a kube-aws binary

$ make build
Building kube-aws 102cb5b8c9d71af1a5074b77955c4913e7d0b84d

$ bin/kube-aws version
kube-aws version 102cb5b8c9d71af1a5074b77955c4913e7d0b84d

FYI: Make targets

  • make build to build bin/kube-aws
  • make format to automatically format all the code of kube-aws
  • make test to run all the unit tests

End-to-end testing in kube-aws

  • Create a main cluster
  • Create a node pool
  • Update the main cluster
  • Run Kubernetes conformance tests (:exclamation:)

Kubernetes conformance tests

  • are a subset of Kubernetes E2E tests
  • to confirm that a minimum feature set of functions required for a cluster to be a Kubernetes cluster is properly working

Kubernetes E2E tests

End-to-end (e2e) tests for Kubernetes provide a mechanism to test end-to-end behavior of the system, and is the last signal to ensure end user operations match developer specifications.
:link: End to end testing in Kubernetes

Why E2E tests are needed

Although unit and integration tests provide a good signal, in a distributed system like Kubernetes it is not uncommon that a minor change may pass all unit and integration tests, but cause unforeseen changes at the system level.
:link: End to end testing in Kubernetes

Running End-to-end tests

  • cd e2e && KUBE_AWS_CLUSTER_NAME=my-cluster-name ./run.sh all

  • It takes approximately 1 hour to run each test case :cry:

  • A good news is that we run it against multiple combinations of settings and options

  • Before releasing new kube-aws releases

  • For you

Contribution guide for kube-aws

  • Where to start: GitHub issues
  • What are discussed and answered
  • Please file your own issue(s)!

Where to start

Our github issues page is the single place to start your contribution:

Discussed and answered in github issues

  • Feature Request
  • Proposal
    • For possible features
    • For possible improvement(s) in the documentation
  • Bug reports
  • Questions

What a proposal would look like


Please file your own issue(s)!

Every little feedback is welcomed to shape our goals and plans toward production-ready Kubernetes clusters on AWS


Thanks for your attention and interest on Kubernetes on AWS,

From @mumoshu, the primary maintainer of kube-aws

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
What you can do with signing up