LoginSignup
TrendQuestionOfficial EventOfficial ColumnOpportunitiesOrganization
7
4

More than 5 years have passed since last update.

@mumoshu

posted at

updated at

Highly available and scalable Kubernetes on AWS #builderscon

AWS,CloudFormation,CoreOS,kubernetes,builderscon
1 / 89

Table of contents

  1. A bit of context
  2. About this presentation
  3. Overview of Kubernetes on AWS
  4. Detailed guide for kube-aws

A bit of context

  • 　Kubernetes :point_left:
  • 　AWS
  • 　Why Kubernetes on AWS
  • 　Why they matter to you

:shell: Kubernetes in a nutshell

An open-source system for automating deployment, scaling, and management of containerized applications.
:link: http://kubernetes.io/

:shell: Kubernetes does

  • Automatically keep your applications running according to your requirements
    • Types of an application: long-running/one-shot, stateful/stateless, standalone/distributed
    • Required resources like CPU, memory, storage, machine, network capability, permissions, etc.

:shell: Kubernetes does

  • Provides APIs to CRUD desired states of your applications
  • Converges your applications to their desired states

Kubernetes Alternatives

  • Docker Swarm
  • Hashicorp Nomad
  • Kontena
  • AWS ECS(EC2 Container Service)
  • Mesos
  • etc.

3 catchphrases of Kubernetes

  • Planet Scale
  • Never Outgrow
  • Runs Anywhere

:globe_with_meridians: "Planet Scale" with Kubernetes

Designed on the same principles that allows Google to run billions of containers a week
:link: http://kubernetes.io/

:lifter: You "Never Outgrow" Kubernetes

Whether testing locally or running a global enterprise, Kubernetes flexibility grows with you to deliver your applications consistently and easily no matter how complex your need is
:link: http://kubernetes.io/

How we can never outgrow?
  • Model your application in Kubernetes way:
    • Containers, Pods, Jobs, ReplicaSets, Deployments, StatefulSets, etc.
Kubernetes Models 1/4
  • Containers = cgroups + namespaces w/ copy-on-write
  • Pods = VMs in the container era w/ shared CPU, memory, volumes, permissions
Kubernetes Models 2/4
  • Services = Named sets of pods load balanced via kubernetes internal DNS

image

Kubernetes Models 3/4
  • Jobs = One-shot or long-running jobs backed by pod(s), with optional parallelism
  • ReplicaSets = A set of identical pods with its number maintained
Kubernetes Models 4/4
  • Deployments = Desired state of your ReplicaSet
  • StatefulSets = A set of named, persistent pods to host e.g. stateful distributed system like databases, filesystems

:airplane: Kubernetes "Runs Anywhere"

Kubernetes is open source giving you the freedom to take advantage of on-premise, hybrid, or public cloud infrastructure, letting you effortlessly move workloads to where it matters to you.
:link: http://kubernetes.io/

Anywhere = ?
  • Run your Kubernetes locally for development with minikube
  • On IaaSes including Baremetal, GCP, AWS, OpenStack, etc.
  • On Ubuntu, CentOS, CoreOS, etc.

Context: AWS

  • :ballot_box_with_check: Kubernetes
  • 　AWS :point_left:
  • 　Why Kubernetes on AWS
  • 　Why they matter to you

AWS is a leader in IaaS

:link: AWS named as a leader in the Infrastructure as a Service (IaaS) Magic Quadrant report for 6th consecutive year

AWS is IaaS

  • VPS
  • IaaS :point_left:
  • PaaS
  • SaaS

:computer: VPS

  • Virtual Private Server
  • Equipped with a common set of services(including installed softwares, public IPs, storages, CPUs, memories, etc.)
  • e.g. DigitalOcean, さくらのVPS

:gear: IaaS

  • Infrastructure as a Service
  • Serves you a series of resources and APIs to build your own virtual data centers
  • e.g. Azure, GCP, AWS, etc

:station: PaaS

  • Platform as a Service
  • Serves you a platform to run your application without taking care of the infrastructure
  • e.g. Heroku, Google App Engine

:fork_knife_plate: SaaS

  • Software as a Service
  • Serves you software(s) or managed applications to assists part of your business(es)

Context: Why Kubernetes on AWS

  • :ballot_box_with_check: Kubernetes
  • :ballot_box_with_check: AWS
  • 　Why Kubernetes on AWS :point_left:
  • 　Why they matter to you

Recap: What is Kubernetes?

  • Kubernetes is all about how to run your application in an unified way regardless of where it runs: from home to a datacenter, from a laptop to a rack of blade servers

Kubernetes on What?

  • VPS can be used to run Kubernetes
  • There're PaaSes to host your Kubernetes clusters
  • There're SaaSes hosted by Kubernetes
  • There're IaaSes provides foundations to build/host your Kubernetes clusters

So: Why Kubernetes on IaaS?

  • IaaS allow us to build our own PaaS based on Kubernetes to host our SaaS according to our requirements

Context: Why they matter to you

  • :ballot_box_with_check: Kubernetes
  • :ballot_box_with_check: AWS
  • :ballot_box_with_check: Why Kubernetes on AWS
  • 　Why they matter to you :point_left:

:ticket: builderscon is

  • IMHO builderscon is almost a conference for programmers.
  • Why all those things matter to us programmers?

From a programmer's viewpoint: API

  • Kubernetes provides a rich set of APIs to manage Kubernetes resources
  • AWS provides a rich set of APIs to manage AWS resources

From a programmer's viewpoint: Kubernetes and AWS

  • Kubernetes is a just framework to program how my applications are managed
  • AWS is a just framework to program how my Kubernetes clusters are managed

:keyboard: I'd like to program apps, not manage them

  • Kubernetes allows us to focus on programming
  • Also: Kubernetes and its surrounding ecosystem themselves would be interesting to be programmed :wink:

:ok_woman: Now, you understand the context

  • :ballot_box_with_check: Kubernetes
  • :ballot_box_with_check: AWS
  • :ballot_box_with_check: Why they matter to you

Let's keep going :muscle:

Table of contents

  1. A bit of context
  2. About this presentation :point_left:
  3. Overview of Kubernetes on AWS
  4. Detailed guide for kube-aws

About this presentation

  • Topics explained in this presentation
  • Questions answered in this presentation

:information_source: Topics explained in this presentation

  • Overview of Kubernetes on AWS
  • Detailed guide for kube-aws
  • Related personal experience(s) from the presenter

:question: Questions answered in this presentation

  • How to achieve high scalability w/ Kubernetes on AWS?
  • How to achieve high availability w/ Kubernetes on AWS?
  • Are there any tool(s) to deploy and manage Kubernetes clusters?
  • How to select tool(s) for your use?
    • Requirements which arises frequently today
    • What is kube-aws? How to use it?
  • What's my use-case?
    • Requirements
    • Tools

Table of contents

  1. A bit of context
  2. About this presentation
  3. Overview of Kubernetes on AWS :point_left:
  4. Detailed guide for kube-aws

Overview of Kubernetes on AWS

  • How to achieve high availability and scalability
  • Tools
  • How to select those tools

High availability with Kubernetes on AWS

  • High availability in nutshell
  • How to achieve it

:shell: High availability in nutshell

Entirely/Almost no downtime when a expected/unexpected shutdown of 1 or more nodes and pods

  • Pods = Sets of containers running on EC2 instances
  • Nodes = EC2 instances

High availability and Redundancy

  • Pods = Sets of containers
    • 2 or more identical pods
    • ReplicaSets in Kubernetes
  • Nodes = EC2 instances
    • 2 or more nodes usable to host the same set of pods
    • Auto Scaling Group and/or Spot Fleet in AWS

High scalability with Kubernetes on AWS

  • High scalability in nutshell
  • How to achieve it

:shell: High scalability in nutshell

No redundant cost + No user visible frustration in less/more workload

  • Pods = Sets of containers running on EC2 instances
  • Nodes = EC2 instances

Cost = Number of nodes * unit price
Number of nodes = Total resources required by pods

High scalability for pods

High scalability for nodes

  • Nodes = EC2 instances
    • Less nodes when less pods, more pods when more workload
    • Kubernetes's cluster-autoscaler
      • Adds nodes to meet capacity enough for scheduling all the pending pods
    • AWS Auto Scaling
      • Adds/removes nodes according to CPU and/or memory usage

:tools: Are there any tool(s) to deploy and manage Kubernetes clusters?

  • Yes,
  • A lot
    • klondike, kubernetes-anywhere, kops, kube-aws, kope, kube-cluster, kubeadm-aws, halcyon-kubernetes, tack, etc.
  • :bookmark: An overview of those tools

:question: How to select tool(s) for your use?

Selecting a tool is all about:

  • knowing your requirements,
  • investing in tools, and then
  • connecting them

Knowing your requirements

Requirements which aries frequently today:

  • Highly available
  • Highly scalable
  • Highly secure
  • Be locked in to technologies give you higher leverage

Lock-in to what gives you higher leverage?

  • An IaaS frequently reduce pricing
  • An IaaS frequently add new features, APIs, SDKs, etc., which are ready to be used for your business
  • // AWS for me

Table of contents

  1. A bit of context
  2. About this presentation
  3. Overview of Kubernetes on AWS
  4. Detailed guide for kube-aws :point_left:

Detailed guide for kube-aws

  • For all:
    • kube-aws in a nutshell
  • For users:
    • Usage guide for kube-aws
  • For developers:
    • Development guide for kube-aws
  • For potential contributors:
    • Contribution guide for kube-aws

:shell: kube-aws in a nutshell

  • A single binary to create/update/destroy Kubernetes cluster(s) on AWS
  • Infrastructure as Code achieved via AWS CloudFormation + Golang
  • Originally developed by CoreOS, currently maintained by community https://github.com/coreos/kube-aws

:hamburger: Under the hood of kube-aws

  • Standard Golang application based on Golang 1.7, glide, cobra, etc., coupled w/
  • CoreOS, which has official support for running Kubernetes
  • Kubernetes 1.4.x, upstream releases followed timely
  • Various AWS services
    • VPC, EC2, ELB, Route53, KMS, CloudFormation

:no_entry: kube-aws does not use

  • Terraform
  • Ansible
  • Chef, Puppet, Saltstack, etc.

Usage guide for kube-aws

Create your first Kubernetes cluster on AWS :smile:
* Install kube-aws
* Generate cluster.yaml
* Customize your cluster
* Render the contents of the assets directory
* Validate your configuration
* Updating the cluster without downtime
* Destroying the cluster

Creating your first Kubernetes cluster on AWS

  • Create keys
  • Install the latest version of kube-aws command
  • Customizing the cluster

Prepare keys

  • EC2 Keypair
    • Used to authenticate SSH access to your server
  • KMS key
    • Used to encrypt/decrypt TLS assets generated by kube-aws

Install the kube-aws command

$ platform=darwin-amd64
$ version=v0.9.2-rc.1
$ curl -L https://github.com/coreos/kube-aws/releases/download/${version}/kube-aws-${platform}.tar.gz | tar zxv ${platform}/kube-aws && \
  mv ${platform}/kube-aws /usr/local/bin/kube-aws && \
  chmod +x /usr/local/bin/kube-aws
$ kube-aws version
kube-aws version v0.9.2-rc.1

:link: https://github.com/coreos/kube-aws/releases

Customizing the cluster

Generate a cluster.yaml:

$ mkdir my-cluster
$ cd my-cluster

$ kube-aws init \
--cluster-name=my-cluster-name \
--external-dns-name=my-cluster-endpoint \
--region=us-west-1 \
--availability-zone=us-west-1c \
--key-name=key-pair-name \
--kms-key-arn="arn:aws:kms:us-west-1:xxxxxxxxxx:key/xxxxxxxxxxxxxxxxxxx"

Customizing the cluster

cluster.yaml:

clusterName: my-cluster-name
externalDNSName: my-cluster-name-api.my.domain
releaseChannel: stable
createRecordSet: true
recordSetTTL: 300
hostedZoneId: "XASDASDASD" # DEV private only route53 zone
keyName: my-ssh-keypair
region: ap-northeast-1
availabilityZone: ap-northeast-1c
kmsKeyArn: "arn:aws:kms:ap-northeast-1:0w123456789:key/d345fcd1-c77c-4fca-acdc-asdasdf3234232"
controllerCount: 1
controllerInstanceType: m3.medium
controllerRootVolumeSize: 30
controllerRootVolumeType: gp2
workerCount: 1
workerInstanceType: m3.medium
workerRootVolumeSize: 30
workerRootVolumeType: gp2
etcdCount: 1
etcdInstanceType: m3.medium
etcdRootVolumeSize: 30
etcdDataVolumeSize: 30
vpcId: vpc-xxxccc45
routeTableId: "rtb-aaabbb12" # main external no NAT uses internet gateway
vpcCIDR: "10.1.0.0/16"
instanceCIDR: "10.1.10.0/24"
serviceCIDR: "10.3.0.0/24"
podCIDR: "10.2.0.0/16"
dnsServiceIP: 10.3.0.10
stackTags:
  Name: "my-cluster-name"
  Environment: "development"

Render the contents of the assets directory

$ kube-aws render
$ tree
.
├── cluster.yaml
├── credentials
│   ├── admin-key.pem
│   ├── admin.pem
│   ├── apiserver-key.pem
│   ├── apiserver.pem
│   ├── ca-key.pem
│   ├── ca.pem
│   ├── worker-key.pem
│   └── worker.pem
│   ├── etcd-key.pem
│   └── etcd.pem
│   ├── etcd-client-key.pem
│   └── etcd-client.pem
├── kubeconfig
├── stack-template.json
└── userdata
    ├── cloud-config-controller
    └── cloud-config-worker

Credential files

  • admin*.pem, used on your local machine and/or build servers
  • worker*.pem, used in worker nodes
  • apiserver*.pem, used in controller nodes
  • etcd-client*pem, used in worker and controller nodes
  • etcd*pem, used in etcd nodes

Credential file naming

  • foo-key.pem
    • The private key for foo access
  • foo.pem
    • THe public key for foo access

That's a lot!

  • Every little mistake in configuration would result in non-functional clusters
  • That's why we need kube-aws validate :smile:

Validate your configuration

$ kube-aws validate
  • Fails only when there're syntax and/or semantic errors in your configuration files
    • Especially in:
      • cloud-config
      • CloudFormation templates
      • No need to wait for minutes until you find easy mistakes like typos :wink:

It is just "validating"

  • kube-aws validate doesn't(can't) test functional correctness of the resulting Kubernetes cluster
  • It is covered by our End-to-end testing

Launch your cluster

$ kube-aws up
Creating AWS resources. This should take around 5 minutes.
Success! Your AWS resources have been created:
Cluster Name:   my-cluster-name

The containers that power your cluster are now being downloaded.

You should be able to access the Kubernetes API once the containers finish downloading.

Or just export your cluster

$ kube-aws up --export
  • Generates a CloudFormation stack template
  • A template is a JSON file which defines which AWS resources to create and how they're connected.

Destroy your cluster

$ kube-aws destroy

Development guide for kube-aws

  • Understanding how kube-aws is built
  • Build your own kube-aws
  • Make targets
  • End-to-end testing in kube-aws

Understanding how kube-aws is built

  • golang 1.7.x
  • glide for package management
    • Alternatives: Godep, trash, govendor, etc.
  • go gen + go run for code generation
    • To embed the default configuration files into kube-aws binaries
  • go test for unit testing
  • gofmt for code formatting and validation

Build your own kube-aws

  • Install required tools
  • git clone
  • make build

Installing required tools

brew update

brew install go
brew install glide
brew install make

Clone the repository to an appropriate path

mkdir -p $GOPATH/src/github.com/coreos/kube-aws

git clone git@github.com:coreos/kube-aws.git \
  $GOPATH/src/github.com/coreos/kube-aws

Build a kube-aws binary

$ make build
./build
Building kube-aws 102cb5b8c9d71af1a5074b77955c4913e7d0b84d

$ bin/kube-aws version
kube-aws version 102cb5b8c9d71af1a5074b77955c4913e7d0b84d

FYI: Make targets

  • make build to build bin/kube-aws
  • make format to automatically format all the code of kube-aws
  • make test to run all the unit tests

End-to-end testing in kube-aws

  • Create a main cluster
  • Create a node pool
  • Update the main cluster
  • Run Kubernetes conformance tests (:exclamation:)

Kubernetes conformance tests

  • are a subset of Kubernetes E2E tests
  • to confirm that a minimum feature set of functions required for a cluster to be a Kubernetes cluster is properly working

Kubernetes E2E tests

End-to-end (e2e) tests for Kubernetes provide a mechanism to test end-to-end behavior of the system, and is the last signal to ensure end user operations match developer specifications.
:link: End to end testing in Kubernetes

Why E2E tests are needed

Although unit and integration tests provide a good signal, in a distributed system like Kubernetes it is not uncommon that a minor change may pass all unit and integration tests, but cause unforeseen changes at the system level.
:link: End to end testing in Kubernetes

Running End-to-end tests

  • cd e2e && KUBE_AWS_CLUSTER_NAME=my-cluster-name ./run.sh all

  • It takes approximately 1 hour to run each test case :cry:

  • A good news is that we run it against multiple combinations of settings and options

  • Before releasing new kube-aws releases

  • For you

Contribution guide for kube-aws

  • Where to start: GitHub issues
  • What are discussed and answered
  • Please file your own issue(s)!

Where to start

Our github issues page is the single place to start your contribution:

Discussed and answered in github issues

  • Feature Request
  • Proposal
    • For possible features
    • For possible improvement(s) in the documentation
  • Bug reports
  • Questions

What a proposal would look like

image

Please file your own issue(s)!

Every little feedback is welcomed to shape our goals and plans toward production-ready Kubernetes clusters on AWS

Fin.

Thanks for your attention and interest on Kubernetes on AWS,

From @mumoshu, the primary maintainer of kube-aws

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
What you can do with signing up
Sign upLogin
7
4