kubernetes
brigade
cndjp
cndf2019

Kubernetes and Beyond Democratizing Workflow Automation

※本記事はCloudNative Dyas Fukuoka 2019の登壇スライドです




Intro.


image.png

「クラウドは人間を強化する」

"Cloud reinforces people"

"飛び込もう、Cloud Nativeの世界", CloudNative Days Fukuoka 2019




  • Life is too short to waste

  • I don't want to write a lot of code to get things done



Master Kelsey says...

image.png

https://twitter.com/kelseyhightower/status/935252923721793536



Platform?


  • Cloud?

  • Serverless?

  • PaaS?



Platform?


  • PaaS is abbrev of Platform as a Service

  • Google Cloud Platform ?

  • Cloud Foundry - Cloud Application Platform

  • OpenShift - Container Application Platform

  • Knative is often called as Serverless Platform ?

Is Platform something to do with Cloud, Serverless , Application?


image.png


Cloud Programming Simplified: A Berkeley View on Serverless Computing

February 10, 2019



image.png




  • Cloud: Hardware abstracted away


  • Serverless: = Cloud abstracted away



    • Serverless Compute: Cloud Computing abstracted away(Cloud Run, Lambda


    • Serverless Storage: Cloud Storage abstracted away(Spanner, Aurora-serverless




  • Container: Industry-Standard Application Packaging Format




  • Application Platform: Serverless abstracted away

And there are a bunch of Applications...



Kinds of Applications



  • General Apps: Project Mangement, Issue Tracking, Chat, E-mail, Spreadsheet, Web Form, Workflow Management


  • Developer Apps: CI, CD, Observability, Security, (PaaS)

This situation is damaging our lives! (I think)

Why? How?



Ex. Enterprise Application Development & Deployment Workflow 2019



  • Create Issue: GitHub, Jira


  • Discuss: GitHub, Jira, Slack


  • Write Design Doc: GitHub + Markdown, Google Doc, MS Word


  • Write Code (Editor on local machine or cloud IDE


  • Pull Request (CLI, GitHub


  • Rewiew (GitHub


  • Approve (GitHub


  • Request Approval for Production Op: Google Form/Your Int. Workflow System


  • Merge to Trigger Automation: GitOps!


    • Automation happens on Kubernetes(Containerized scripts, workflows, services)




  • Browse Web UI of your CI: CircleCI, TravisCI, Jenkins


  • Browse Web UI of your Observability Stack: Kibana, Grafana, Datadog, Prometheus


  • Debug: Terminal, Debugger, kubectl, See Logs


  • :beers:

  • Repeat



WHY

Are we forced to switch apps SO many times?


  • GitHub

  • Slack

  • JIRA

  • Google Form

  • CI

  • Observability Stack

  • Terminal



Can't we automate it?



Kinds of Automation


image.png

"Kubernetes Operators" - https://coreos.com/operators/



Is "Operator" the only solution?

Do we need to build an "Enterprise Application Development & Deployment Workflow" operator? In golang?



Can't we start small, then think BIG?



History

Back to the good old days of Linux

You've been used to write shell scripts to automate all the daunting tasks

image.png



Shell scripts remain an optimum choice for enhancing software leverage.

"Linux and the Unix Philosophy", Mike Gancarz, 2003




The world got more complicated...



Modern Environment(at glance)

image.png


Microservice Architecture — Learn, Build, and Deploy Applications - DZone Microservices




Modern Environment(Reality)

image.png


"Amazon"

Practicing Microservices, Lachlan Evenson




  • Distributed microservices...

  • Inside and outside of your company...

  • Connected with various protocols, transports, and network topology

  • With varying assumptions on latencies, durability, etc...

Today, it isn't simple as scripting:

#!/usr/bin/env bash

inputs | map | reduce | present > output



Enter Cloud Native



Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.

https://github.com/cncf/toc/blob/master/DEFINITION.md




Cloud Native

Does it help scripting, too!?

Let's see...



OOT: Baseline

You have no difficulty deploying things to Kubernetes:



  • kubectl apply -f manifests/ to install a set of manifsts


  • kustomize build -k env/ | kubectl apply -f to compose manifests to be applied


  • helm upgrade --install chart/myapp myapp to upgrade/install a single app


  • helmfile apply to deploy all the apps

This talk is not about those, but rather about scripting.



OOT: Another Stories



  • Continuous Integration: Use of CircleCI, Concourse CI, Argo CI, Drone, Jenkins, ...


  • Continuous Delivery: Argo CD, Weaveworks Flux, or whatever to automate builds and deployments

  • Observability: Datadog, Prometheus, Grafana, Zipkin, Jaeger, ...

This talk is not about those, but rather about scripting.



Hello World, Traditional Style

In old days we've been writing:


Script

echo hello world



Output

hello world




Hello World, Modern Style

kubectl run to run any command on Kubernetes:


Script

kubectl run \

#
Interactive
-i \
#
Run only once
--restart=Never \
#
Run the command in an Alpine Linux container
--image alpine:3.9 \
#
Gimme an unique ID of my command runtime...
alpine-cmd-$(date +%s)-$(LC_CTYPE=C tr -dc a-z0-9 < /dev/urandom | fold -w 3 | head -n 1) \
# This is the command
-- /bin/sh -c "echo hello world"


Output

hello world




Hello World, Modern Style(Abstracted)


Script

cat <<'EOS' > ~/bin/kuberun && chmod +x ~/bin/kuberun

#!/usr/bin/env bash
opts="-i"
if tty >/dev/null; then opts="${opts}t"; fi
kubectl run "$opts" --restart=Never --image alpine:3.9 alpine-cmd-$(date +%s)-$(LC_CTYPE=C tr -dc a-z0-9 < /dev/urandom | fold -w 3 | head -n 1) -- /bin/sh -c "$*"
EOS


Script

kuberun echo hello world



Output

hello world




kubectl run - Curious How It Works?

See "What happens when ... Kubernetes edition!" for more info:

github.com/jamiehannaford/what-happens-when-k8s



Kubernetes-as-Shell solutions



  • kubectl-repl runs shell with short aliases to various kubectl commands. no filc sync


  • kubed-sh runs ruby, node, python in containers, with periodic local-to-remote file sync


Good.

We can now run any command on Kubernetes.


But how to run multiple commands in parallel?



Run commands on Kubernetes(Parallel)


Script

kuberun echo hello & kuberun echo world & wait



You can even go further...


Script

i=0; for c in h e l l o w o r l d; do

((i+=1)); bash -c "sleep $i; kuberun echo $c" &
done


Output

h

e
l
l
o
w
o
r
l
d



Gotcha


  • Your containers don't see your local files(of course!)



Problem of "Syncing Files"

Use-cases:



  • Local-to-Container: Run local code in container(e.g. Webapp dev.)


  • Container-to-Local: Build binary in container, download the result


  • Container-to-Container Use-case: Run workflow across containers(e.g. CI/CD pipeline)



"Sync Files" Solutions


  • kubectl cp


  • rsync, kubectl-warp

  • Cloud Object Storage(e.g. AWS S3)


  • ReadWriteOnce volumes (Block Storage incl. AWS EBS, Ceph Block Device/RBD)


  • ReadWriteMany volumes (NFS, AzureFS, CephFS, AWS EFS)



Introducing kubectl-warp


  • Basically kubectl run + initial/periodic rsync(but no final rsync)


  • Runs container with local-to-remote file sync(no vice versa though)


  • rsync for point-to-point file sync before and while running the command




kubewarp

cat <<'EOS' > ~/bin/kubewarp && chmod +x ~/bin/kubewarp

#!/usr/bin/env bash
inc="$1"
shift
opts="-i"
if tty >/dev/null; then opts="${opts}t"; fi
kubectl warp "$opts" --include "$inc" --image alpine:3.9 alpine-cmd-$(date +%s)-$(LC_CTYPE=C tr -dc a-z0-9 < /dev/urandom | fold -w 3 | head -n 1) -- /bin/sh -c "$*"
EOS


Local

$ ls

hello.txt


Kubernetes

$ kubewarp ls

hello.txt



Gotcha


  • Your local machine doesn't see your remote files!



kubectl-biwarp

kube-warp + reverse rsync before stopping container

https://github.com/mumoshu/kubectl-biwarp



Concurrency + Aggregate Results

$ seq 1 5 | xargs -P5 -n1 -I{} kubebiwarp "hello.{}.txt" "echo hello {} > hello.{}.txt" ; wait

$ cat hello.*.txt

hello 1
hello 2
hello 3
hello 4
hello 5


Good.

We can now run simple stateful workflow on Kubernetes.



Shell Scripting Gotchas



  • No defacto package manager (I like bpkg. But did you know..?


  • Not so portable(Windows?


  • Testing(bats?



Can't we "graduate" to a real programming language?



JavaScript?

Package Manager: npm or yarn

Portability: Better than bash! :)

Testing: Standard node.js tools



There is...

A CNCF Sandbox Project that helps Scripting on Kubernetes



Introducing Brigade

image.png



Brigade

Use-cases:


  • Watch for incoming requests from services like GitHub, Docker, and Trello

  • Run unit tests, process data, and store results

  • Send notifications through services like Slack and Twitter

Anything that can be scripted in node and shell



Automating unit tests on git push with Brigade


  1. Write some script:

// Run unit tests for a Github push

const { events, Job , Group} = require("brigadier");
const dest = "$GOPATH/src/github.com/technosophos/ulid";

events.on("push", (e, p) => {
console.log(e.payload)
var gh = JSON.parse(e.payload)
var test = new Job("test", "golang:1.9")
test.tasks = [
"mkdir -p " + dest,
"cp -a /src/* " + dest,
"cd " + dest,
"go get -u github.com/golang/dep/cmd/dep",
"dep ensure",
"make test"
];
test.run()
});



  1. Brigade turns the script into...

image.png

Workflow composed of containers!



Brigade's Abstraction

Brigade Script: Dynamic Workflow of Jobs

Job: Containerized Shell Scripts



Testing

technosophos/brigtest for testing brigade scripts:

npm run brigtest -c event.json



Back to the topic...


  • Life is too short to waste. I don't want to write a lot of code to get things done


  • Does kubectl, shell and brigade solve our problem as whole?


No (yet)



Re: Enterprise Application Development & Deployment Workflow 2019



  • Create Issue: GitHub, Jira


  • Discuss: GitHub, Jira, Slack


  • Write Design Doc: GitHub + Markdown, Google Doc, MS Word


  • Write Code (Editor on local machine or cloud IDE


  • Pull Request (CLI, GitHub


  • Rewiew (GitHub


  • Approve (GitHub


  • Request Approval for Production Op: Google Form/Your Int. Workflow System


  • Merge to Trigger Automation: GitOps!


    • Automation happens on Kubernetes(Containerized scripts, workflows, services)




  • Browse Web UI of your CI: CircleCI, TravisCI, Jenkins


  • Browse Web UI of your Observability Stack: Kibana, Grafana, Datadog, Prometheus


  • Debug: Terminal, Debugger, kubectl, See Logs


  • :beers:

  • Repeat



Why are we forced to switch apps so many times?


  • GitHub

  • Slack

  • JIRA

  • Google Form

  • CI

  • Observability Stack

  • Terminal



Can't we script the WHOLE workflow?

read -p "title? " title

create_issue --title=$title > issue.json

echo discussion in the following communication channels...
for url in $(myctl create discussions --issue $(jq .issue.channels[].url issue.json)); do
open ${url}
done

for ch in $(myctl get channels --discussion $discussion_id -i json | jq .channelId); do
wait_for_settlement $ch
done

echo "All the discussions have been settled! Let's write some code!"

pushd $PROJECT
branch=$(escape-title "$title")
git checkout -b $branch origin/master
code .

read -p "type enter once you finish coding"

git add .
git commit -m "$title"
git push origin $branch
hub pull-request

wait_for_approval $pr_number

verify_ci_builds

# fail if (1)deplyoment failed (2)alert triggered in 10 min.
wait_for_deployment $pr_number

status=$?

if [ $status -ne 0 ]; then
echo "deployment failed!" 1>&2
open $datadog_url
fi



Going Production...

$ HISTTIMEFORMAT=$'\r\e[K' history > myscript

$ cat <(echo '#!/usr/bin/env bash') myscript > myscript
$ chmod +x myscript
$ $EDITOR $_

And finally deploy it as a Brigade script?

$ helm install brigade/brigade-project \

--set-file defaultScript=brigade.js ...



Is that all you need?

No!



Remaining Gotchas



Multiple Events Involved in Single Workflow


  • Webhook (Do you own a webhook-to-job-queue server?

  • Slack Interaction Message (Internet-facing HTTP server required in order to respond e.g. button clicks!

  • And do it without compromising security?


1-to-1

image.png

PubSub

image.png

Ref: https://www.knative.dev/docs/eventing/


image.png

Compromiesd "Channel Receiver" is horror!


Instead of building and running (internet-facing) webhook server per automation,

Use Event Gateway



Event Gateway Implementations



CloudNative Wishlist

Scripting Platform that allows me to script the whole workflow:

if pr-not-created; then

hub pull-request
fi

# Subscribe to Event Gateway!
await-events \
-f github-pr-approvals.yaml \
-f slack-thread-comments.yaml \
-f circleci-builds-pass.yaml \
-o result.json

status=$?

if [ $status -ne 0 ];
exit 1
fi

./do-merge-pr $(./extract_pr_number result.json)

await-events \
-f argo-cd-deployment-success.yaml \
-t 300s

status=$?

if [ $status -ne 0 ];
exit 1
fi



Important Point

All those works on top of Kubernetes!

Kubernetes: API and Runtime for Various Automations



Ending



Recap



  • WHY: Life is too short to waste. I don't want to write a lot of code to get things done


  • HOW: I need a Scripting Platform that automates all the things(Cloud, Kubernetes, Service Mesh, Dev/Business Process, ...)


  • WHAT: Shell Script + Brigade + Event Gateway. Kubernetes is just API and Runtime


Script on Kubernetes for automating everything.

Bring back your productivity!


git add progress/

git commit -m 'Finish the talk. Start coding.'

Thanks! :bow: :wave:



Any question?

@mumoshu / Yusuke KUOKA at Z Lab, Developing Kubernetes as a Service for YJ


AWS EC2上にKubernetesクラスタをベンチャー2社で構築・運用し続けて3年、AWS EKSの登場を機にこれから何をしていくべきかを考えてます。


AWS Container Hero

OSSメンテナ: kube-aws, helmfile, helm-diff, eksctl, ...

Kubernetesのお仕事 at freee, Quipper, ChatWork, ...



Democratizing Workflow Automation

WHY: 人生は短いので、普段のお仕事を自動化するのに沢山コードを書きたくない!

HOW: クラウドもサービスメッシュもPaaSもSaaSも社内システムも、何もかもを自動化するためにScripting Platformがほしい

WHAT: Shell Script + Brigade + Event Gatewayで作ってます。Kubernetesは実装詳細

](qi)