Help us understand the problem. What is going on with this article?

Kubernetes and Beyond Democratizing Workflow Automation

More than 1 year has passed since last update.

Kubernetes and Beyond Democratizing Workflow Automation

by mumoshu
1 / 78

※本記事はCloudNative Dyas Fukuoka 2019の登壇スライドです



Intro.

image.png
「クラウドは人間を強化する」
"Cloud reinforces people"

"飛び込もう、Cloud Nativeの世界", CloudNative Days Fukuoka 2019


  • Life is too short to waste
  • I don't want to write a lot of code to get things done

Master Kelsey says...

image.png

https://twitter.com/kelseyhightower/status/935252923721793536


Platform?

  • Cloud?
  • Serverless?
  • PaaS?

Platform?

  • PaaS is abbrev of Platform as a Service
  • Google Cloud Platform ?
  • Cloud Foundry - Cloud Application Platform
  • OpenShift - Container Application Platform
  • Knative is often called as Serverless Platform ?

Is Platform something to do with Cloud, Serverless , Application?


image.png

Cloud Programming Simplified: A Berkeley View on Serverless Computing
February 10, 2019


image.png


  • Cloud: Hardware abstracted away
  • Serverless: = Cloud abstracted away
    • Serverless Compute: Cloud Computing abstracted away(Cloud Run, Lambda
    • Serverless Storage: Cloud Storage abstracted away(Spanner, Aurora-serverless
  • Container: Industry-Standard Application Packaging Format
  • Application Platform: Serverless abstracted away

And there are a bunch of Applications...


Kinds of Applications

  • General Apps: Project Mangement, Issue Tracking, Chat, E-mail, Spreadsheet, Web Form, Workflow Management
  • Developer Apps: CI, CD, Observability, Security, (PaaS)

This situation is damaging our lives! (I think)

Why? How?


Ex. Enterprise Application Development & Deployment Workflow 2019

  • Create Issue: GitHub, Jira
  • Discuss: GitHub, Jira, Slack
  • Write Design Doc: GitHub + Markdown, Google Doc, MS Word
  • Write Code (Editor on local machine or cloud IDE
  • Pull Request (CLI, GitHub
  • Rewiew (GitHub
  • Approve (GitHub
  • Request Approval for Production Op: Google Form/Your Int. Workflow System
  • Merge to Trigger Automation: GitOps!
    • Automation happens on Kubernetes(Containerized scripts, workflows, services)
  • Browse Web UI of your CI: CircleCI, TravisCI, Jenkins
  • Browse Web UI of your Observability Stack: Kibana, Grafana, Datadog, Prometheus
  • Debug: Terminal, Debugger, kubectl, See Logs
  • :beers:
  • Repeat

WHY

Are we forced to switch apps SO many times?

  • GitHub
  • Slack
  • JIRA
  • Google Form
  • CI
  • Observability Stack
  • Terminal

Can't we automate it?


Kinds of Automation


image.png

"Kubernetes Operators" - https://coreos.com/operators/


Is "Operator" the only solution?

Do we need to build an "Enterprise Application Development & Deployment Workflow" operator? In golang?


Can't we start small, then think BIG?


History

Back to the good old days of Linux

You've been used to write shell scripts to automate all the daunting tasks

image.png


Shell scripts remain an optimum choice for enhancing software leverage.
"Linux and the Unix Philosophy", Mike Gancarz, 2003


The world got more complicated...


Modern Environment(at glance)

image.png

Microservice Architecture — Learn, Build, and Deploy Applications - DZone Microservices


Modern Environment(Reality)

image.png

"Amazon"
Practicing Microservices, Lachlan Evenson


  • Distributed microservices...
  • Inside and outside of your company...
  • Connected with various protocols, transports, and network topology
  • With varying assumptions on latencies, durability, etc...

Today, it isn't simple as scripting:

#!/usr/bin/env bash
inputs | map | reduce | present > output

Enter Cloud Native


Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.
https://github.com/cncf/toc/blob/master/DEFINITION.md


Cloud Native

Does it help scripting, too!?

Let's see...


OOT: Baseline

You have no difficulty deploying things to Kubernetes:

  • kubectl apply -f manifests/ to install a set of manifsts
  • kustomize build -k env/ | kubectl apply -f to compose manifests to be applied
  • helm upgrade --install chart/myapp myapp to upgrade/install a single app
  • helmfile apply to deploy all the apps

This talk is not about those, but rather about scripting.


OOT: Another Stories

  • Continuous Integration: Use of CircleCI, Concourse CI, Argo CI, Drone, Jenkins, ...
  • Continuous Delivery: Argo CD, Weaveworks Flux, or whatever to automate builds and deployments
  • Observability: Datadog, Prometheus, Grafana, Zipkin, Jaeger, ...

This talk is not about those, but rather about scripting.


Hello World, Traditional Style

In old days we've been writing:

Script
echo hello world
Output
hello world

Hello World, Modern Style

kubectl run to run any command on Kubernetes:

Script
kubectl run \
  # Interactive
  -i \
  # Run only once
  --restart=Never \
  # Run the command in an Alpine Linux container
  --image alpine:3.9 \
  # Gimme an unique ID of my command runtime...
  alpine-cmd-$(date +%s)-$(LC_CTYPE=C tr -dc a-z0-9 < /dev/urandom | fold -w 3 | head -n 1) \
  # This is the command
  -- /bin/sh -c "echo hello world"
Output
hello world

Hello World, Modern Style(Abstracted)

Script
cat <<'EOS' > ~/bin/kuberun && chmod +x ~/bin/kuberun
#!/usr/bin/env bash
opts="-i"
if tty >/dev/null; then opts="${opts}t"; fi
kubectl run "$opts" --restart=Never --image alpine:3.9 alpine-cmd-$(date +%s)-$(LC_CTYPE=C tr -dc a-z0-9 < /dev/urandom | fold -w 3 | head -n 1) -- /bin/sh -c "$*"
EOS
Script
kuberun echo hello world
Output
hello world

kubectl run - Curious How It Works?

See "What happens when ... Kubernetes edition!" for more info:

github.com/jamiehannaford/what-happens-when-k8s


Kubernetes-as-Shell solutions

  • kubectl-repl runs shell with short aliases to various kubectl commands. no filc sync
  • kubed-sh runs ruby, node, python in containers, with periodic local-to-remote file sync

Good.

We can now run any command on Kubernetes.


But how to run multiple commands in parallel?


Run commands on Kubernetes(Parallel)

Script
kuberun echo hello & kuberun echo world & wait

You can even go further...

Script
i=0; for c in h e l l o w o r l d; do
  ((i+=1)); bash -c "sleep $i; kuberun echo $c" &
done
Output
h
e
l
l
o
w
o
r
l
d

Gotcha

  • Your containers don't see your local files(of course!)

Problem of "Syncing Files"

Use-cases:

  • Local-to-Container: Run local code in container(e.g. Webapp dev.)
  • Container-to-Local: Build binary in container, download the result
  • Container-to-Container Use-case: Run workflow across containers(e.g. CI/CD pipeline)

"Sync Files" Solutions

  • kubectl cp
  • rsync, kubectl-warp
  • Cloud Object Storage(e.g. AWS S3)
  • ReadWriteOnce volumes (Block Storage incl. AWS EBS, Ceph Block Device/RBD)
  • ReadWriteMany volumes (NFS, AzureFS, CephFS, AWS EFS)

Introducing kubectl-warp

  • Basically kubectl run + initial/periodic rsync(but no final rsync)

  • Runs container with local-to-remote file sync(no vice versa though)

  • rsync for point-to-point file sync before and while running the command


kubewarp
cat <<'EOS' > ~/bin/kubewarp && chmod +x ~/bin/kubewarp
#!/usr/bin/env bash
inc="$1"
shift
opts="-i"
if tty >/dev/null; then opts="${opts}t"; fi
kubectl warp "$opts" --include "$inc" --image alpine:3.9 alpine-cmd-$(date +%s)-$(LC_CTYPE=C tr -dc a-z0-9 < /dev/urandom | fold -w 3 | head -n 1) -- /bin/sh -c "$*"
EOS
Local
$ ls
hello.txt
Kubernetes
$ kubewarp ls
hello.txt

Gotcha

  • Your local machine doesn't see your remote files!

kubectl-biwarp

kube-warp + reverse rsync before stopping container

https://github.com/mumoshu/kubectl-biwarp


Concurrency + Aggregate Results

$ seq 1 5 | xargs -P5 -n1 -I{} kubebiwarp "hello.{}.txt" "echo hello {} > hello.{}.txt" ; wait
$ cat hello.*.txt
hello 1
hello 2
hello 3
hello 4
hello 5

Good.

We can now run simple stateful workflow on Kubernetes.


Shell Scripting Gotchas

  • No defacto package manager (I like bpkg. But did you know..?
  • Not so portable(Windows?
  • Testing(bats?

Can't we "graduate" to a real programming language?


JavaScript?

Package Manager: npm or yarn

Portability: Better than bash! :)

Testing: Standard node.js tools


There is...

A CNCF Sandbox Project that helps Scripting on Kubernetes


Introducing Brigade

image.png


Brigade

Use-cases:

  • Watch for incoming requests from services like GitHub, Docker, and Trello
  • Run unit tests, process data, and store results
  • Send notifications through services like Slack and Twitter

Anything that can be scripted in node and shell


Automating unit tests on git push with Brigade

  1. Write some script:
// Run unit tests for a Github push
const { events, Job , Group} = require("brigadier");
const dest = "$GOPATH/src/github.com/technosophos/ulid";

events.on("push", (e, p) => {
  console.log(e.payload)
  var gh = JSON.parse(e.payload)
  var test = new Job("test", "golang:1.9")
  test.tasks = [
    "mkdir -p " + dest,
    "cp -a /src/* " + dest,
    "cd " + dest,
    "go get -u github.com/golang/dep/cmd/dep",
    "dep ensure",
    "make test"
  ];
  test.run()
});

  1. Brigade turns the script into...

image.png

Workflow composed of containers!


Brigade's Abstraction

Brigade Script: Dynamic Workflow of Jobs

Job: Containerized Shell Scripts


Testing

technosophos/brigtest for testing brigade scripts:

npm run brigtest -c event.json

Back to the topic...

  • Life is too short to waste. I don't want to write a lot of code to get things done

  • Does kubectl, shell and brigade solve our problem as whole?

No (yet)


Re: Enterprise Application Development & Deployment Workflow 2019

  • Create Issue: GitHub, Jira
  • Discuss: GitHub, Jira, Slack
  • Write Design Doc: GitHub + Markdown, Google Doc, MS Word
  • Write Code (Editor on local machine or cloud IDE
  • Pull Request (CLI, GitHub
  • Rewiew (GitHub
  • Approve (GitHub
  • Request Approval for Production Op: Google Form/Your Int. Workflow System
  • Merge to Trigger Automation: GitOps!
    • Automation happens on Kubernetes(Containerized scripts, workflows, services)
  • Browse Web UI of your CI: CircleCI, TravisCI, Jenkins
  • Browse Web UI of your Observability Stack: Kibana, Grafana, Datadog, Prometheus
  • Debug: Terminal, Debugger, kubectl, See Logs
  • :beers:
  • Repeat

Why are we forced to switch apps so many times?

  • GitHub
  • Slack
  • JIRA
  • Google Form
  • CI
  • Observability Stack
  • Terminal

Can't we script the WHOLE workflow?

read -p "title? " title

create_issue --title=$title > issue.json

echo discussion in the following communication channels...
for url in $(myctl create discussions --issue $(jq .issue.channels[].url issue.json)); do
  open ${url}
done

for ch in $(myctl get channels --discussion $discussion_id -i json | jq .channelId); do
wait_for_settlement $ch
done

echo "All the discussions have been settled! Let's write some code!"

pushd $PROJECT
branch=$(escape-title "$title")
git checkout -b $branch origin/master
code .

read -p "type enter once you finish coding"

git add .
git commit -m "$title"
git push origin $branch
hub pull-request

wait_for_approval $pr_number

verify_ci_builds

# fail if (1)deplyoment failed (2)alert triggered in 10 min.
wait_for_deployment $pr_number

status=$?

if [ $status -ne 0 ]; then
  echo "deployment failed!" 1>&2
  open $datadog_url
fi

Going Production...

$ HISTTIMEFORMAT=$'\r\e[K' history > myscript
$ cat <(echo '#!/usr/bin/env bash') myscript > myscript
$ chmod +x myscript
$ $EDITOR $_

And finally deploy it as a Brigade script?

$ helm install brigade/brigade-project \
  --set-file defaultScript=brigade.js ...

Is that all you need?

No!


Remaining Gotchas


Multiple Events Involved in Single Workflow

  • Webhook (Do you own a webhook-to-job-queue server?
  • Slack Interaction Message (Internet-facing HTTP server required in order to respond e.g. button clicks!
  • And do it without compromising security?

1-to-1

image.png

PubSub

image.png

Ref: https://www.knative.dev/docs/eventing/


image.png

Compromiesd "Channel Receiver" is horror!


Instead of building and running (internet-facing) webhook server per automation,

Use Event Gateway


Event Gateway Implementations


CloudNative Wishlist

Scripting Platform that allows me to script the whole workflow:

if pr-not-created; then
  hub pull-request
fi

# Subscribe to Event Gateway!
await-events \
  -f github-pr-approvals.yaml \
  -f slack-thread-comments.yaml \
  -f circleci-builds-pass.yaml \
  -o result.json

status=$?

if [ $status -ne 0 ];
  exit 1
fi

./do-merge-pr $(./extract_pr_number result.json)

await-events \
  -f argo-cd-deployment-success.yaml \
  -t 300s

status=$?

if [ $status -ne 0 ];
  exit 1
fi

Important Point

All those works on top of Kubernetes!

Kubernetes: API and Runtime for Various Automations


Ending


Recap

  • WHY: Life is too short to waste. I don't want to write a lot of code to get things done
  • HOW: I need a Scripting Platform that automates all the things(Cloud, Kubernetes, Service Mesh, Dev/Business Process, ...)
  • WHAT: Shell Script + Brigade + Event Gateway. Kubernetes is just API and Runtime

Script on Kubernetes for automating everything.

Bring back your productivity!


git add progress/
git commit -m 'Finish the talk. Start coding.'

Thanks! :bow: :wave:


Any question?

@mumoshu / Yusuke KUOKA at Z Lab, Developing Kubernetes as a Service for YJ

AWS EC2上にKubernetesクラスタをベンチャー2社で構築・運用し続けて3年、AWS EKSの登場を機にこれから何をしていくべきかを考えてます。

AWS Container Hero
OSSメンテナ: kube-aws, helmfile, helm-diff, eksctl, ...
Kubernetesのお仕事 at freee, Quipper, ChatWork, ...


Democratizing Workflow Automation

WHY: 人生は短いので、普段のお仕事を自動化するのに沢山コードを書きたくない!
HOW: クラウドもサービスメッシュもPaaSもSaaSも社内システムも、何もかもを自動化するためにScripting Platformがほしい
WHAT: Shell Script + Brigade + Event Gatewayで作ってます。Kubernetesは実装詳細
](qi)

mumoshu
AWS Container Hero maintaining kube-aws, eksctl, brigade, helmfile, helm-diff, etc.
https://github.com/mumoshu/
freee
スモールビジネスのバックオフィス業務をテクノロジーで自動化し、日本のスモールビジネスを元気にする
http://www.freee.co.jp/
Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
Comments
No comments
Sign up for free and join this conversation.
If you already have a Qiita account
Why do not you register as a user and use Qiita more conveniently?
You need to log in to use this function. Qiita can be used more conveniently after logging in.
You seem to be reading articles frequently this month. Qiita can be used more conveniently after logging in.
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
ユーザーは見つかりませんでした