More than 5 years have passed since last update.

Kubernetes and Beyond Democratizing Workflow Automation

Last updated at 2019-05-15Posted at 2019-04-18

1 / 78

※本記事はCloudNative Dyas Fukuoka 2019の登壇スライドです

Intro.

「クラウドは人間を強化する」
"Cloud reinforces people"

"飛び込もう、Cloud Nativeの世界", CloudNative Days Fukuoka 2019

Life is too short to waste
I don't want to write a lot of code to get things done

Master Kelsey says...

Platform?

Cloud?
Serverless?
PaaS?

Platform?

PaaS is abbrev of Platform as a Service
Google Cloud Platform ?
Cloud Foundry - Cloud Application Platform
OpenShift - Container Application Platform
Knative is often called as Serverless Platform ?

Is Platform something to do with Cloud, Serverless , Application?

Cloud Programming Simplified: A Berkeley View on Serverless Computing
February 10, 2019

Cloud: Hardware abstracted away
Serverless: = Cloud abstracted away
- Serverless Compute: Cloud Computing abstracted away(Cloud Run, Lambda
- Serverless Storage: Cloud Storage abstracted away(Spanner, Aurora-serverless
Container: Industry-Standard Application Packaging Format
- Lambda? Use img2lambda
Application Platform: Serverless abstracted away

And there are a bunch of Applications...

Kinds of Applications

General Apps: Project Mangement, Issue Tracking, Chat, E-mail, Spreadsheet, Web Form, Workflow Management
Developer Apps: CI, CD, Observability, Security, (PaaS)

This situation is damaging our lives! (I think)

Why? How?

Ex. Enterprise Application Development & Deployment Workflow 2019

Create Issue: GitHub, Jira
Discuss: GitHub, Jira, Slack
Write Design Doc: GitHub + Markdown, Google Doc, MS Word
Write Code (Editor on local machine or cloud IDE
Pull Request (CLI, GitHub
Rewiew (GitHub
Approve (GitHub
Request Approval for Production Op: Google Form/Your Int. Workflow System
Merge to Trigger Automation: GitOps!
- Automation happens on Kubernetes(Containerized scripts, workflows, services)
Browse Web UI of your CI: CircleCI, TravisCI, Jenkins
Browse Web UI of your Observability Stack: Kibana, Grafana, Datadog, Prometheus
Debug: Terminal, Debugger, kubectl, See Logs
Repeat

WHY

Are we forced to switch apps SO many times?

GitHub
Slack
JIRA
Google Form
CI
Observability Stack
Terminal

Can't we automate it?

Kinds of Automation

Domain-specific Automations
- CI/CD, Monitoring, Log Collection&Aggregation
- Operations for Cluster, Apps, Middlewares, Services, ...
Kubernetes "Operator" Pattern
- https://coreos.com/operators/
- Pattern for building domain-specific automations
  - postgres-operator and es-operartor by Zalando

"Kubernetes Operators" - https://coreos.com/operators/

Is "Operator" the only solution?

Do we need to build an "Enterprise Application Development & Deployment Workflow" operator? In golang?

Can't we start small, then think BIG?

History

Back to the good old days of Linux

You've been used to write shell scripts to automate all the daunting tasks

Shell scripts remain an optimum choice for enhancing software leverage.
"Linux and the Unix Philosophy", Mike Gancarz, 2003

The world got more complicated...

Modern Environment(at glance)

Microservice Architecture — Learn, Build, and Deploy Applications - DZone Microservices

Modern Environment(Reality)

"Amazon"
Practicing Microservices, Lachlan Evenson

Distributed microservices...
Inside and outside of your company...
Connected with various protocols, transports, and network topology
With varying assumptions on latencies, durability, etc...

Today, it isn't simple as scripting:

# !/usr/bin/env bash
inputs | map | reduce | present > output

Enter Cloud Native

Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.
https://github.com/cncf/toc/blob/master/DEFINITION.md

Cloud Native

Does it help scripting, too!?

Let's see...

OOT: Baseline

You have no difficulty deploying things to Kubernetes:

kubectl apply -f manifests/ to install a set of manifsts
kustomize build -k env/ | kubectl apply -f to compose manifests to be applied
helm upgrade --install chart/myapp myapp to upgrade/install a single app
helmfile apply to deploy all the apps

This talk is not about those, but rather about scripting.

OOT: Another Stories

Continuous Integration: Use of CircleCI, Concourse CI, Argo CI, Drone, Jenkins, ...
Continuous Delivery: Argo CD, Weaveworks Flux, or whatever to automate builds and deployments
Observability: Datadog, Prometheus, Grafana, Zipkin, Jaeger, ...

This talk is not about those, but rather about scripting.

Hello World, Traditional Style

In old days we've been writing:

Script

echo hello world

Output

hello world

Hello World, Modern Style

kubectl run to run any command on Kubernetes:

Script

kubectl run \
  # Interactive
  -i \
  # Run only once
  --restart=Never \
  # Run the command in an Alpine Linux container
  --image alpine:3.9 \
  # Gimme an unique ID of my command runtime...
  alpine-cmd-$(date +%s)-$(LC_CTYPE=C tr -dc a-z0-9 < /dev/urandom | fold -w 3 | head -n 1) \
  # This is the command
  -- /bin/sh -c "echo hello world"

Output

hello world

Hello World, Modern Style(Abstracted)

Script

cat <<'EOS' > ~/bin/kuberun && chmod +x ~/bin/kuberun
# !/usr/bin/env bash
opts="-i"
if tty >/dev/null; then opts="${opts}t"; fi
kubectl run "$opts" --restart=Never --image alpine:3.9 alpine-cmd-$(date +%s)-$(LC_CTYPE=C tr -dc a-z0-9 < /dev/urandom | fold -w 3 | head -n 1) -- /bin/sh -c "$*"
EOS

Script

kuberun echo hello world

Output

hello world

`kubectl run` - Curious How It Works?

See "What happens when ... Kubernetes edition!" for more info:

github.com/jamiehannaford/what-happens-when-k8s

Kubernetes-as-Shell solutions

kubectl-repl runs shell with short aliases to various kubectl commands. no filc sync
kubed-sh runs ruby, node, python in containers, with periodic local-to-remote file sync

Good.

We can now run any command on Kubernetes.

But how to run multiple commands in parallel?

Run commands on Kubernetes(Parallel)

Script

kuberun echo hello & kuberun echo world & wait

You can even go further...

Script

i=0; for c in h e l l o w o r l d; do
  ((i+=1)); bash -c "sleep $i; kuberun echo $c" &
done

Output

h
e
l
l
o
w
o
r
l
d

Gotcha

Your containers don't see your local files(of course!)

Problem of "Syncing Files"

Use-cases:

Local-to-Container: Run local code in container(e.g. Webapp dev.)
Container-to-Local: Build binary in container, download the result
Container-to-Container Use-case: Run workflow across containers(e.g. CI/CD pipeline)

"Sync Files" Solutions

kubectl cp
rsync, kubectl-warp
Cloud Object Storage(e.g. AWS S3)
ReadWriteOnce volumes (Block Storage incl. AWS EBS, Ceph Block Device/RBD)
ReadWriteMany volumes (NFS, AzureFS, CephFS, AWS EFS)

Introducing kubectl-warp

Basically kubectl run + initial/periodic rsync(but no final rsync)
Runs container with local-to-remote file sync(no vice versa though)
rsync for point-to-point file sync before and while running the command

kubewarp

cat <<'EOS' > ~/bin/kubewarp && chmod +x ~/bin/kubewarp
# !/usr/bin/env bash
inc="$1"
shift
opts="-i"
if tty >/dev/null; then opts="${opts}t"; fi
kubectl warp "$opts" --include "$inc" --image alpine:3.9 alpine-cmd-$(date +%s)-$(LC_CTYPE=C tr -dc a-z0-9 < /dev/urandom | fold -w 3 | head -n 1) -- /bin/sh -c "$*"
EOS

Local

$ ls
hello.txt

Kubernetes

$ kubewarp ls
hello.txt

Gotcha

Your local machine doesn't see your remote files!

kubectl-biwarp

kube-warp + reverse rsync before stopping container

Concurrency + Aggregate Results

$ seq 1 5 | xargs -P5 -n1 -I{} kubebiwarp "hello.{}.txt" "echo hello {} > hello.{}.txt" ; wait

$ cat hello.*.txt
hello 1
hello 2
hello 3
hello 4
hello 5

Good.

We can now run simple stateful workflow on Kubernetes.

Shell Scripting Gotchas

No defacto package manager (I like bpkg. But did you know..?
Not so portable(Windows?
Testing(bats?

Can't we "graduate" to a real programming language?

JavaScript?

Package Manager: npm or yarn

Portability: Better than bash! :)

Testing: Standard node.js tools

There is...

A CNCF Sandbox Project that helps Scripting on Kubernetes

Introducing Brigade

Brigade

Use-cases:

Watch for incoming requests from services like GitHub, Docker, and Trello
Run unit tests, process data, and store results
Send notifications through services like Slack and Twitter

Anything that can be scripted in node and shell

Automating unit tests on `git push` with Brigade

Write some script:

// Run unit tests for a Github push
const { events, Job , Group} = require("brigadier");
const dest = "$GOPATH/src/github.com/technosophos/ulid";

events.on("push", (e, p) => {
  console.log(e.payload)
  var gh = JSON.parse(e.payload)
  var test = new Job("test", "golang:1.9")
  test.tasks = [
    "mkdir -p " + dest,
    "cp -a /src/* " + dest,
    "cd " + dest,
    "go get -u github.com/golang/dep/cmd/dep",
    "dep ensure",
    "make test"
  ];
  test.run()
});

Brigade turns the script into...

Workflow composed of containers!

Brigade's Abstraction

Brigade Script: Dynamic Workflow of Jobs

Job: Containerized Shell Scripts

Testing

technosophos/brigtest for testing brigade scripts:

npm run brigtest -c event.json

Back to the topic...

Life is too short to waste. I don't want to write a lot of code to get things done
Does kubectl, shell and brigade solve our problem as whole?

No (yet)

Re: Enterprise Application Development & Deployment Workflow 2019

Create Issue: GitHub, Jira
Discuss: GitHub, Jira, Slack
Write Design Doc: GitHub + Markdown, Google Doc, MS Word
Write Code (Editor on local machine or cloud IDE
Pull Request (CLI, GitHub
Rewiew (GitHub
Approve (GitHub
Request Approval for Production Op: Google Form/Your Int. Workflow System
Merge to Trigger Automation: GitOps!
- Automation happens on Kubernetes(Containerized scripts, workflows, services)
Browse Web UI of your CI: CircleCI, TravisCI, Jenkins
Browse Web UI of your Observability Stack: Kibana, Grafana, Datadog, Prometheus
Debug: Terminal, Debugger, kubectl, See Logs
Repeat

Why are we forced to switch apps so many times?

GitHub
Slack
JIRA
Google Form
CI
Observability Stack
Terminal

Can't we script the WHOLE workflow?

read -p "title? " title

create_issue --title=$title > issue.json

echo discussion in the following communication channels...
for url in $(myctl create discussions --issue $(jq .issue.channels[].url issue.json)); do
  open ${url}
done

for ch in $(myctl get channels --discussion $discussion_id -i json | jq .channelId); do
wait_for_settlement $ch
done

echo "All the discussions have been settled! Let's write some code!"

pushd $PROJECT
branch=$(escape-title "$title")
git checkout -b $branch origin/master
code .

read -p "type enter once you finish coding"

git add .
git commit -m "$title"
git push origin $branch
hub pull-request

wait_for_approval $pr_number

verify_ci_builds

# fail if (1)deplyoment failed (2)alert triggered in 10 min.
wait_for_deployment $pr_number

status=$?

if [ $status -ne 0 ]; then
  echo "deployment failed!" 1>&2
  open $datadog_url
fi

Going Production...

$ HISTTIMEFORMAT=$'\r\e[K' history > myscript
$ cat <(echo '#!/usr/bin/env bash') myscript > myscript
$ chmod +x myscript
$ $EDITOR $_

And finally deploy it as a Brigade script?

$ helm install brigade/brigade-project \
  --set-file defaultScript=brigade.js ...

Is that all you need?

No!

Remaining Gotchas

Make your command idempotent
- Rather than badly paraphrasing here, read: Designing robust and predictable APIs with idempotency
Multiple Events Involved in Single Workflow

Multiple Events Involved in Single Workflow

Webhook (Do you own a webhook-to-job-queue server?
Slack Interaction Message (Internet-facing HTTP server required in order to respond e.g. button clicks!
And do it without compromising security?

1-to-1

PubSub

Ref: https://www.knative.dev/docs/eventing/

Compromiesd "Channel Receiver" is horror!

Instead of building and running (internet-facing) webhook server per automation,

Use Event Gateway

Event Gateway Implementations

Serverless Event Gateway: API Gateway + PubSub
Knative Eventing - API Gateway + PubSub
Brigade Gateway: Kind of API Gateway (1-to-1 Msg Passing

CloudNative Wishlist

Scripting Platform that allows me to script the whole workflow:

if pr-not-created; then
  hub pull-request
fi

# Subscribe to Event Gateway!
await-events \
  -f github-pr-approvals.yaml \
  -f slack-thread-comments.yaml \
  -f circleci-builds-pass.yaml \
  -o result.json

status=$?

if [ $status -ne 0 ];
  exit 1
fi

./do-merge-pr $(./extract_pr_number result.json)

await-events \
  -f argo-cd-deployment-success.yaml \
  -t 300s

status=$?

if [ $status -ne 0 ];
  exit 1
fi

Important Point

All those works on top of Kubernetes!

Kubernetes: API and Runtime for Various Automations

Ending

Recap

WHY: Life is too short to waste. I don't want to write a lot of code to get things done
HOW: I need a Scripting Platform that automates all the things(Cloud, Kubernetes, Service Mesh, Dev/Business Process, ...)
WHAT: Shell Script + Brigade + Event Gateway. Kubernetes is just API and Runtime

Script on Kubernetes for automating everything.

Bring back your productivity!

git add progress/

git commit -m 'Finish the talk. Start coding.'

Thanks!

Any question?

@mumoshu / Yusuke KUOKA at Z Lab, Developing Kubernetes as a Service for YJ

AWS EC2上にKubernetesクラスタをベンチャー2社で構築・運用し続けて3年、AWS EKSの登場を機にこれから何をしていくべきかを考えてます。

AWS Container Hero
OSSメンテナ: kube-aws, helmfile, helm-diff, eksctl, ...
Kubernetesのお仕事 at freee, Quipper, ChatWork, ...

Democratizing Workflow Automation

WHY: 人生は短いので、普段のお仕事を自動化するのに沢山コードを書きたくない！
HOW: クラウドもサービスメッシュもPaaSもSaaSも社内システムも、何もかもを自動化するためにScripting Platformがほしい
WHAT: Shell Script + Brigade + Event Gatewayで作ってます。Kubernetesは実装詳細
](qi)

You get articles that match your needs
You can efficiently read back useful information
You can use dark theme

What you can do with signing up

Kubernetes and Beyond Democratizing Workflow Automation

Intro.

Master Kelsey says...

Platform?

Platform?

Kinds of Applications

Ex. Enterprise Application Development & Deployment Workflow 2019

WHY

Can't we automate it?

Kinds of Automation

Is "Operator" the only solution?

Can't we start small, then think BIG?

History

The world got more complicated...

Modern Environment(at glance)

Modern Environment(Reality)

Enter Cloud Native

Cloud Native

OOT: Baseline

OOT: Another Stories

Hello World, Traditional Style

Hello World, Modern Style

Hello World, Modern Style(Abstracted)

kubectl run - Curious How It Works?

Kubernetes-as-Shell solutions

Run commands on Kubernetes(Parallel)

Gotcha

Problem of "Syncing Files"

"Sync Files" Solutions

Introducing kubectl-warp

Gotcha

kubectl-biwarp

Concurrency + Aggregate Results

Shell Scripting Gotchas

Can't we "graduate" to a real programming language?

JavaScript?

There is...

Introducing Brigade

Brigade

Automating unit tests on git push with Brigade

Brigade's Abstraction

Testing

Back to the topic...

Re: Enterprise Application Development & Deployment Workflow 2019

Why are we forced to switch apps so many times?

Can't we script the WHOLE workflow?

Going Production...

Is that all you need?

Remaining Gotchas

Multiple Events Involved in Single Workflow

Event Gateway Implementations

CloudNative Wishlist

Important Point

Ending

Recap

Any question?

Democratizing Workflow Automation

`kubectl run` - Curious How It Works?

Automating unit tests on `git push` with Brigade