０．はじめに
k8s超初心者な筆者が、k8sとistioを勉強してみるためにEKSでIstioサンプルアプリを動かして、ついでにGrafanaやらJaegerやらKialiやらの良い感じなツールを触ってみたものです。
基本はこちらのワークショップとIstio公式チュートリアルをベースに進め、気になる点や汲み取れない部分は個人的に補完するように努めましたが、もし間違った理解をしている部分があればご指摘いただけると幸いです。
１．クラスター＆ワーカーノードの準備
eksctlを使って、EKSを立てていきます。
eksctlの導入
$ curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
$ sudo mv /tmp/eksctl /usr/local/bin
$ eksctl version
[?] version.Info{BuiltAt:"", GitCommit:"", GitTag:"0.1.23"}
eksctlでクラスター＆ワーカーノードを作成
今回は既存のVPC/サブネットを使いたかったので、”--vpc-public-subnets”オプションを指定しています。
（サブネット指定のみでOK）
$ eksctl create cluster \
> --name=lets-istio \
> --region=us-west-2 \
> --nodes=2 \
> --nodes-min=1 \
> --nodes-max=2 \
> --node-type=t3.medium \
> --version=1.11 \
> --vpc-public-subnets=subnet-02444fd81a76e520f,subnet-04478d97b94927e41
[?] using region us-west-2
[?] using existing VPC (vpc-035904bf2c34af1e1) and subnets (private:[] public:[subnet-04478d97b94927e41 subnet-02444fd81a76e520f])
[!] custom VPC/subnets will be used; if resulting cluster doesn't function as expected, make sure to review the configuration of VPC/subnets
[?] nodegroup "ng-cae1ee27" will use "ami-0c28139856aaf9c3b" [AmazonLinux2/1.11]
[?] creating EKS cluster "lets-istio" in "us-west-2" region
[?] will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
[?] if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-west-2 --name=lets-istio'
[?] creating cluster stack "eksctl-lets-istio-cluster"
[?] creating nodegroup stack "eksctl-lets-istio-nodegroup-ng-cae1ee27"
[?] all EKS cluster resource for "lets-istio" had been created
[?] saved kubeconfig as "/home/ec2-user/.kube/config"
[?] nodegroup "ng-cae1ee27" has 0 node(s)
[?] waiting for at least 1 node(s) to become ready in "ng-cae1ee27"
[?] nodegroup "ng-cae1ee27" has 2 node(s)
[?] node "ip-10-0-0-4.us-west-2.compute.internal" is ready
[?] node "ip-10-0-3-130.us-west-2.compute.internal" is ready
[?] kubectl command should work with "/home/ec2-user/.kube/config", try 'kubectl get nodes'
[?] EKS cluster "lets-istio" in "us-west-2" region is ready
作成結果を確認
$ eksctl get cluster
NAME REGION
lets-istio us-west-2
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-0-4.us-west-2.compute.internal Ready <none> 54s v1.11.5
ip-10-0-3-130.us-west-2.compute.internal Ready <none> 57s v1.11.5
２．Istioの導入
Istioをダウンロード
$ curl -L https://git.io/getLatestIstio | sh -
$ cd istio-*
$ sudo mv -v bin/istioctl /usr/local/bin/
$ istioctl version
Version: 1.0.6
GitRevision: 98598f88f6ee9c1e6b3f03b652d8e0e3cd114fa2
User: root@464fc845-2bf8-11e9-b805-0a580a2c0506
Hub: docker.io/istio
GolangVersion: go1.10.4
BuildStatus: Clean
Kiali用のsecretを作成
Kialiを有効化するには、事前にSecretを作成しておく必要がある。（無いとapply時にエラーとなってpodが起動しない）
$ NAMESPACE=istio-system
$ echo -n 'admin' | base64
YWRtaW4=
$ echo -n '1f2d1e2e67df' | base64
MWYyZDFlMmU2N2Rm
$ cat <<EOF | kubectl apply -f -
> apiVersion: v1
> kind: Secret
> metadata:
> name: kiali
> namespace: $NAMESPACE
> labels:
> app: kiali
> type: Opaque
> data:
> username: YWRtaW4=
> passphrase: MWYyZDFlMmU2N2Rm
> EOF
secret "kiali" created
Istioをインストール
JaegerとKialiはデフォルトでは無効なので、”--set tracing.enabled=true””--set kiali.enabled=true”オプションで有効化する。
$ pwd
/home/ec2-user/istio-1.0.6
#CustomResourceDefinitionのデプロイ
$ kubectl apply -f install/kubernetes/helm/istio/templates/crds.yaml
#service account & namespaceを作成
$ kubectl create -f install/kubernetes/helm/helm-service-account.yaml
$ helm init --service-account tiller
$ kubectl create namespace istio-system
#helmテンプレートでistioをデプロイ
$ helm template install/kubernetes/helm/istio \
>--name istio \
>--namespace istio-system \
>--set global.configValidation=false \
>--set sidecarInjectorWebhook.enabled=true \
>--set grafana.enabled=true \
>--set servicegraph.enabled=true \
>--set tracing.enabled=true \
>--set kiali.enabled=true> istio.yaml
$ kubectl apply -f istio.yaml
$ kubectl get pod -n istio-system
NAME READY STATUS RESTARTS AGE
grafana-7f6cd4bf56-tq5tp 1/1 Running 0 39s
istio-citadel-796c94878b-s7zc4 1/1 Running 0 38s
istio-cleanup-secrets-p94tx 0/1 Completed 0 40s
istio-egressgateway-864444d6ff-cgjh4 1/1 Running 0 39s
istio-galley-6c68c5dbcf-hg9rx 1/1 Running 0 39s
istio-grafana-post-install-2pcdx 0/1 Completed 0 40s
istio-ingressgateway-694576c7bb-kwsk4 1/1 Running 0 39s
istio-pilot-79f5f46dd5-ngcm6 2/2 Running 0 38s
istio-policy-5bd5578b94-w9sq4 2/2 Running 0 38s
istio-security-post-install-chzxk 0/1 Completed 0 40s
istio-sidecar-injector-6d8f88c98f-gbhtq 1/1 Running 0 37s
istio-telemetry-5598f86cd8-7c7bc 2/2 Running 0 38s
istio-tracing-7596597bd7-t5qjg 1/1 Running 0 37s
kiali-84cd87cd5d-pl29c 1/1 Running 0 39s
prometheus-76db5fddd5-w2477 1/1 Running 0 38s
servicegraph-df4d5f678-prxhn 1/1 Running 0 37s
３．サンプルアプリケーションをデプロイ
アプリケーションのデプロイ
ワークショップのIstio編で用意されているサンプルアプリをデプロイします。
※Istioと一緒にダウンロードされています（/home/ec2-user/istio-1.0.6/samples/bookinfo/platform/kube/bookinfo.yaml）
#istioctl kube-injectでサイドカー(istio-proxy)を組み込んだManifestを作成し、そのManifestを使ってデプロイする
kubectl apply -f <(istioctl kube-inject -f samples/bookinfo/platform/kube/bookinfo.yaml)
#デプロイ結果を確認
$ kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/details-v1-589c676b6d-pdrwc 2/2 Running 0 17s
pod/productpage-v1-8fc45744b-sxvdq 2/2 Running 0 17s
pod/ratings-v1-6f7f894cd4-hkpfr 2/2 Running 0 17s
pod/reviews-v1-767c6f8477-5x8fs 2/2 Running 0 17s
pod/reviews-v2-5f5d6b9dbd-t6z5s 2/2 Running 0 17s
pod/reviews-v3-865cd5bcc5-g4gwm 2/2 Running 0 17s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/details ClusterIP 172.20.136.178 <none> 9080/TCP 17s
service/kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 18h
service/productpage ClusterIP 172.20.64.76 <none> 9080/TCP 17s
service/ratings ClusterIP 172.20.90.58 <none> 9080/TCP 17s
service/reviews ClusterIP 172.20.3.26 <none> 9080/TCP 17s
各podが2/2となっているので、それぞれpod毎にサイドカーコンテナが併せて起動しているのがわかります。
この段階で、アプリケーションはこんな感じでデプロイされた状態になっており、ポイントは以下2点かと思います。
①各podでistio-initによるiptablesの書き換えが行われ、podへのinbound/outboundが全てistio-proxyコンテナへリダイレクトされるようになっている
②istio-ingressgatewayをデプロイした段階でCLBが作成されているが、現時点ではアプリケーションとの接続はされていないためアクセスできない
※ちなみに、組み込み前後ではManifestはこんな感じに違います。（組み込み後はかなり肥大化してます）
もともとのbookinfo.yaml
# Copyright 2017 Istio Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##################################################################################################
# Details service
apiVersion: v1
kind: Service
metadata:
name: details
labels:
app: details
spec:
ports:
- port: 9080
app: details
kind: Deployment
metadata:
name: details-v1
spec:
replicas: 1
template:
metadata:
labels:
app: details
version: v1
spec:
containers:
- name: details
image: istio/examples-bookinfo-details-v1:1.8.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
# Ratings service
apiVersion: v1
kind: Service
metadata:
name: ratings
labels:
app: ratings
spec:
ports:
- port: 9080
app: ratings
kind: Deployment
metadata:
name: ratings-v1
spec:
replicas: 1
template:
metadata:
labels:
app: ratings
version: v1
spec:
containers:
- name: ratings
image: istio/examples-bookinfo-ratings-v1:1.8.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
# Reviews service
apiVersion: v1
kind: Service
metadata:
name: reviews
labels:
app: reviews
spec:
ports:
- port: 9080
name: http
selector:
app: reviews
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: reviews-v1
spec:
replicas: 1
template:
metadata:
labels:
app: reviews
version: v1
spec:
containers:
- name: reviews
image: istio/examples-bookinfo-reviews-v1:1.8.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: reviews-v2
spec:
replicas: 1
template:
metadata:
labels:
app: reviews
version: v2
spec:
containers:
- name: reviews
image: istio/examples-bookinfo-reviews-v2:1.8.0
- containerPort: 9080
kind: Deployment
metadata:
name: reviews-v3
spec:
replicas: 1
template:
metadata:
labels:
app: reviews
version: v3
spec:
containers:
- name: reviews
image: istio/examples-bookinfo-reviews-v3:1.8.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
# Productpage services
apiVersion: v1
kind: Service
metadata:
name: productpage
labels:
app: productpage
spec:
ports:
- port: 9080
name: http
selector:
app: productpage
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: productpage-v1
spec:
replicas: 1
template:
metadata:
labels:
app: productpage
version: v1
spec:
containers:
- name: productpage
image: istio/examples-bookinfo-productpage-v1:1.8.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
組み込み後のbookinfo.yaml
# Copyright 2017 Istio Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##################################################################################################
# Details service
apiVersion: v1
labels:
app: details
spec:
ports:
- port: 9080
name: http
selector:
app: details
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: details-v1
spec:
replicas: 1
strategy: {}
template:
metadata:
annotations:
creationTimestamp: null
labels:
app: details
version: v1
spec:
containers:
- image: istio/examples-bookinfo-details-v1:1.8.0
imagePullPolicy: IfNotPresent
name: details
ports:
- containerPort: 9080
resources: {}
- args:
- proxy
- sidecar
- --configPath
- /etc/istio/proxy
- --binaryPath
- /usr/local/bin/envoy
- --serviceCluster
- details
- --drainDuration
- 45s
- --parentShutdownDuration
- 1m0s
- --discoveryAddress
- istio-pilot.istio-system:15007
- --discoveryRefreshDelay
- 1s
- --zipkinAddress
- zipkin.istio-system:9411
- --connectTimeout
- 10s
- --proxyAdminPort
- "15000"
- --controlPlaneAuthPolicy
- NONE
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: ISTIO_META_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ISTIO_META_INTERCEPTION_MODE
value: REDIRECT
- name: ISTIO_METAJSON_LABELS
value: |
{"app":"details","version":"v1"}
image: docker.io/istio/proxyv2:1.0.6
imagePullPolicy: IfNotPresent
name: istio-proxy
ports:
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
resources:
requests:
cpu: 10m
securityContext:
readOnlyRootFilesystem: true
runAsUser: 1337
volumeMounts:
- mountPath: /etc/istio/proxy
name: istio-envoy
- mountPath: /etc/certs/
name: istio-certs
readOnly: true
initContainers:
- args:
- -p
- "15001"
- -u
- "1337"
- -m
- REDIRECT
- -i
- '*'
- -x
- ""
- -d
imagePullPolicy: IfNotPresent
name: istio-init
resources: {}
securityContext:
capabilities:
add:
- NET_ADMIN
privileged: true
volumes:
- emptyDir:
medium: Memory
name: istio-envoy
- name: istio-certs
secret:
optional: true
secretName: istio.default
status: {}
---
# Ratings service
apiVersion: v1
labels:
app: ratings
spec:
ports:
- port: 9080
name: http
selector:
app: ratings
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: ratings-v1
spec:
replicas: 1
strategy: {}
template:
metadata:
annotations:
creationTimestamp: null
labels:
app: ratings
version: v1
spec:
containers:
- image: istio/examples-bookinfo-ratings-v1:1.8.0
imagePullPolicy: IfNotPresent
name: ratings
ports:
- containerPort: 9080
resources: {}
- args:
- proxy
- sidecar
- --configPath
- /etc/istio/proxy
- --binaryPath
- /usr/local/bin/envoy
- --serviceCluster
- ratings
- --drainDuration
- 45s
- --parentShutdownDuration
- 1m0s
- --discoveryAddress
- istio-pilot.istio-system:15007
- --discoveryRefreshDelay
- 1s
- --zipkinAddress
- zipkin.istio-system:9411
- --connectTimeout
- 10s
- --proxyAdminPort
- "15000"
- --controlPlaneAuthPolicy
- NONE
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: ISTIO_META_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ISTIO_META_INTERCEPTION_MODE
value: REDIRECT
- name: ISTIO_METAJSON_LABELS
value: |
{"app":"ratings","version":"v1"}
image: docker.io/istio/proxyv2:1.0.6
imagePullPolicy: IfNotPresent
name: istio-proxy
ports:
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
resources:
requests:
cpu: 10m
securityContext:
readOnlyRootFilesystem: true
runAsUser: 1337
volumeMounts:
- mountPath: /etc/istio/proxy
name: istio-envoy
- mountPath: /etc/certs/
name: istio-certs
readOnly: true
initContainers:
- args:
- -p
- "15001"
- -u
- "1337"
- -m
- REDIRECT
- -i
- '*'
- -x
- ""
- -d
imagePullPolicy: IfNotPresent
name: istio-init
resources: {}
securityContext:
capabilities:
add:
- NET_ADMIN
privileged: true
volumes:
- emptyDir:
medium: Memory
name: istio-envoy
- name: istio-certs
secret:
optional: true
secretName: istio.default
status: {}
---
# Reviews service
apiVersion: v1
labels:
app: reviews
spec:
ports:
- port: 9080
name: http
selector:
app: reviews
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: reviews-v1
spec:
replicas: 1
strategy: {}
template:
metadata:
annotations:
creationTimestamp: null
labels:
app: reviews
version: v1
spec:
containers:
- image: istio/examples-bookinfo-reviews-v1:1.8.0
imagePullPolicy: IfNotPresent
name: reviews
ports:
- containerPort: 9080
resources: {}
- args:
- proxy
- sidecar
- --configPath
- /etc/istio/proxy
- --binaryPath
- /usr/local/bin/envoy
- --serviceCluster
- reviews
- --drainDuration
- 45s
- --parentShutdownDuration
- 1m0s
- --discoveryAddress
- istio-pilot.istio-system:15007
- --discoveryRefreshDelay
- 1s
- --zipkinAddress
- zipkin.istio-system:9411
- --connectTimeout
- 10s
- --proxyAdminPort
- "15000"
- --controlPlaneAuthPolicy
- NONE
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: ISTIO_META_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ISTIO_META_INTERCEPTION_MODE
value: REDIRECT
- name: ISTIO_METAJSON_LABELS
value: |
{"app":"reviews","version":"v1"}
image: docker.io/istio/proxyv2:1.0.6
imagePullPolicy: IfNotPresent
name: istio-proxy
ports:
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
resources:
requests:
cpu: 10m
securityContext:
readOnlyRootFilesystem: true
runAsUser: 1337
volumeMounts:
- mountPath: /etc/istio/proxy
name: istio-envoy
- mountPath: /etc/certs/
name: istio-certs
readOnly: true
initContainers:
- args:
- -p
- "15001"
- -u
- "1337"
- -m
- REDIRECT
- -i
- '*'
- -x
- ""
- -b
- "9080"
- -d
- ""
image: docker.io/istio/proxy_init:1.0.6
imagePullPolicy: IfNotPresent
name: istio-init
resources: {}
securityContext:
capabilities:
add:
- emptyDir:
medium: Memory
name: istio-envoy
- name: istio-certs
secret:
optional: true
secretName: istio.default
status: {}
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: reviews-v2
spec:
replicas: 1
strategy: {}
template:
metadata:
annotations:
creationTimestamp: null
labels:
app: reviews
version: v2
spec:
containers:
- image: istio/examples-bookinfo-reviews-v2:1.8.0
imagePullPolicy: IfNotPresent
name: reviews
ports:
- containerPort: 9080
resources: {}
- args:
- proxy
- sidecar
- --configPath
- /etc/istio/proxy
- --binaryPath
- /usr/local/bin/envoy
- --serviceCluster
- reviews
- --drainDuration
- 45s
- --parentShutdownDuration
- 1m0s
- --discoveryAddress
- istio-pilot.istio-system:15007
- --discoveryRefreshDelay
- 1s
- --zipkinAddress
- zipkin.istio-system:9411
- --connectTimeout
- 10s
- --proxyAdminPort
- "15000"
- --controlPlaneAuthPolicy
- NONE
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: ISTIO_META_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ISTIO_META_INTERCEPTION_MODE
value: REDIRECT
- name: ISTIO_METAJSON_LABELS
value: |
{"app":"reviews","version":"v2"}
image: docker.io/istio/proxyv2:1.0.6
imagePullPolicy: IfNotPresent
name: istio-proxy
ports:
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
resources:
requests:
cpu: 10m
securityContext:
readOnlyRootFilesystem: true
runAsUser: 1337
volumeMounts:
- mountPath: /etc/istio/proxy
name: istio-envoy
- mountPath: /etc/certs/
name: istio-certs
readOnly: true
initContainers:
- args:
- -p
- "15001"
- -u
- "1337"
- -m
- REDIRECT
- -i
- '*'
- -x
- ""
- -b
- "9080"
- -d
- ""
image: docker.io/istio/proxy_init:1.0.6
imagePullPolicy: IfNotPresent
name: istio-init
resources: {}
securityContext:
capabilities:
add:
- emptyDir:
medium: Memory
name: istio-envoy
- name: istio-certs
secret:
optional: true
secretName: istio.default
status: {}
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: reviews-v3
spec:
replicas: 1
strategy: {}
template:
metadata:
annotations:
creationTimestamp: null
labels:
app: reviews
version: v3
spec:
containers:
- image: istio/examples-bookinfo-reviews-v3:1.8.0
imagePullPolicy: IfNotPresent
name: reviews
ports:
- containerPort: 9080
resources: {}
- args:
- proxy
- sidecar
- --configPath
- /etc/istio/proxy
- --binaryPath
- /usr/local/bin/envoy
- --serviceCluster
- reviews
- --drainDuration
- 45s
- --parentShutdownDuration
- 1m0s
- --discoveryAddress
- istio-pilot.istio-system:15007
- --discoveryRefreshDelay
- 1s
- --zipkinAddress
- zipkin.istio-system:9411
- --connectTimeout
- 10s
- --proxyAdminPort
- "15000"
- --controlPlaneAuthPolicy
- NONE
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: ISTIO_META_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ISTIO_META_INTERCEPTION_MODE
value: REDIRECT
- name: ISTIO_METAJSON_LABELS
value: |
{"app":"reviews","version":"v3"}
image: docker.io/istio/proxyv2:1.0.6
imagePullPolicy: IfNotPresent
name: istio-proxy
ports:
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
resources:
requests:
cpu: 10m
securityContext:
readOnlyRootFilesystem: true
runAsUser: 1337
volumeMounts:
- mountPath: /etc/istio/proxy
name: istio-envoy
- mountPath: /etc/certs/
name: istio-certs
readOnly: true
initContainers:
- args:
- -p
- "15001"
- -u
- "1337"
- -m
- REDIRECT
- -i
- '*'
- -x
- ""
- -d
imagePullPolicy: IfNotPresent
name: istio-init
resources: {}
securityContext:
capabilities:
add:
- NET_ADMIN
privileged: true
volumes:
- emptyDir:
medium: Memory
name: istio-envoy
- name: istio-certs
secret:
optional: true
secretName: istio.default
status: {}
---
# Productpage services
apiVersion: v1
labels:
app: productpage
spec:
ports:
- port: 9080
name: http
selector:
app: productpage
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: productpage-v1
spec:
replicas: 1
strategy: {}
template:
metadata:
annotations:
creationTimestamp: null
labels:
app: productpage
version: v1
spec:
containers:
- image: istio/examples-bookinfo-productpage-v1:1.8.0
imagePullPolicy: IfNotPresent
name: productpage
ports:
- containerPort: 9080
resources: {}
- args:
- proxy
- sidecar
- --configPath
- /etc/istio/proxy
- --binaryPath
- /usr/local/bin/envoy
- --serviceCluster
- productpage
- --drainDuration
- 45s
- --parentShutdownDuration
- 1m0s
- --discoveryAddress
- istio-pilot.istio-system:15007
- --discoveryRefreshDelay
- 1s
- --zipkinAddress
- zipkin.istio-system:9411
- --connectTimeout
- 10s
- --proxyAdminPort
- "15000"
- --controlPlaneAuthPolicy
- NONE
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: ISTIO_META_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ISTIO_META_INTERCEPTION_MODE
value: REDIRECT
- name: ISTIO_METAJSON_LABELS
value: |
{"app":"productpage","version":"v1"}
image: docker.io/istio/proxyv2:1.0.6
imagePullPolicy: IfNotPresent
name: istio-proxy
ports:
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
resources:
requests:
cpu: 10m
securityContext:
readOnlyRootFilesystem: true
runAsUser: 1337
volumeMounts:
- mountPath: /etc/istio/proxy
name: istio-envoy
- mountPath: /etc/certs/
name: istio-certs
readOnly: true
initContainers:
- args:
- -p
- "15001"
- -u
- "1337"
- -m
- REDIRECT
- -i
- '*'
- -x
- ""
- -b
- "9080"
- -d
- ""
image: docker.io/istio/proxy_init:1.0.6
imagePullPolicy: IfNotPresent
name: istio-init
resources: {}
securityContext:
capabilities:
add:
- NET_ADMIN
privileged: true
volumes:
- emptyDir:
medium: Memory
name: istio-envoy
- name: istio-certs
secret:
optional: true
secretName: istio.default
status: {}
---
GatewayとVirtualServiceの作成
ここで、アプリケーションを外部に公開するために、IstioのGateway & VirtualServiceを使って、istio-ingressgatewayとアプリケーションを繋げます。
$ kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml
$ kubectl get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' -n istio-system ; echo
a81256f2b46c011e9bec6068f6b115a8-1231799013.us-west-2.elb.amazonaws.com
IstioのGateway & VirtualServiceを作成するManifestを見る限り、
①Gatewayとしてistio-ingressgatewayの80ポートを構え
②トラフィックをproductpageのService⇒productpageの9080番ポートへ流す
ということをしているんだと理解しました。
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: bookinfo-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: bookinfo
spec:
hosts:
- "*"
gateways:
- bookinfo-gateway
http:
- match:
- uri:
exact: /productpage
- uri:
exact: /login
- uri:
exact: /logout
- uri:
prefix: /api/v1/products
route:
- destination:
host: productpage
port:
number: 9080
この段階でこんな感じに接続されて、アプリケーションへアクセスすることができるようになりました。
CLBの80番ポートへのアクセスがproductpageのpodへトラフィック⇒istio-proxyへリダイレクト⇒9080番ポートで構えているproductpageのアプリケーションコンテナへ到達、という流れを辿っている（はず）
http://＜CLBのDNS＞/productpage へアクセス
（今回はhttp://a81256f2b46c011e9bec6068f6b115a8-1231799013.us-west-2.elb.amazonaws.com/productpage）
なお、現時点ではreviewsサービスはreviews-v1～3のすべてのpodにラウンドロビンしてトラフィックを流しているため、画面をリロードする度にBook Reviewsの表示が切り替わります。
Version1
Version2
Version3
４．IstioでRouting制御
いよいよ、Istioの機能を使ってRoutingしてみます。
まずは、Routing制御のために必要なDestinationRuleを作成。
⇒ここで各versionを定義しており、それらをVirtualServiceからsubsetとして指定することでどのversionにトラフィックを流すかをRoutingします。
kubectl apply -f samples/bookinfo/networking/destination-rule-all.yaml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: reviews
spec:
host: reviews
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
- name: v3
labels:
version: v3
---
特定versionにのみRouting
versionを一つ指定し、ラウンドロビンさせずに特定versionのみにRoutingするパターン
⇒subsetに先ほど定義したDestinationRuleのversionを指定
kubectl apply -f samples/bookinfo/networking/virtual-service-all-v1.yaml
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v1
---
重みを付けて、複数versionにラウンドロビンさせてRouting
versionを2つで50:50の重みを付け、version1とversion3に50%ずつRoutingさせるパターン
⇒比率を変えてBlue/Greenデプロイをしたり、カナリアリリースしたり、A/Bテストしたり、いろいろなケースで利用できます。
kubectl apply -f samples/bookinfo/networking/virtual-service-all-v1.yaml
kubectl apply -f samples/bookinfo/networking/virtual-service-reviews-50-v3.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v1
weight: 50
- destination:
host: reviews
subset: v3
weight: 50
---
特定ユーザからのアクセス時のみ別versionへRouting
header情報をもとにRoutingを切り替えることも可能です。
ここでは、headersのuser情報を見て、jasonユーザがログインしてアクセスしてきた場合のみversion2へRoutingし、それ以外のユーザの場合にはversion1へRoutingしています。
kubectl apply -f samples/bookinfo/networking/virtual-service-reviews-test-v2.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
spec:
hosts:
- reviews
http:
- match:
- headers:
end-user:
exact: jason
route:
- destination:
host: reviews
subset: v2
- route:
- destination:
host: reviews
subset: v1
---
Fault Injection
アクセスに対して、秒指定でレスポンスを遅延させたり、エラーを発生させたりすることもできます。
例えばjasonユーザからのアクセスのみレスポンスを7秒間遅延させるパターン
⇒ratingsアプリケーションは6秒間レスポンスがない場合にはエラーとなる仕様なので、Book Reviews内のraiting部分のみエラーが表示されます。
kubectl apply -f samples/bookinfo/networking/virtual-service-ratings-test-delay.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: ratings
spec:
hosts:
- ratings
http:
- match:
- headers:
end-user:
exact: jason
fault:
delay:
percent: 100
fixedDelay: 7s
route:
- destination:
host: ratings
subset: v1
- route:
- destination:
host: ratings
subset: v1
---
次は、jasonユーザからのアクセスのみ500エラーをレスポンスさせるパターン
⇒先ほどと同じように、Book Reviews内のraiting部分にエラーメッセージが表示されます。
kubectl apply -f samples/bookinfo/networking/virtual-service-ratings-test-abort.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: ratings
spec:
hosts:
- ratings
http:
- match:
- headers:
end-user:
exact: jason
fault:
abort:
percent: 100
httpStatus: 500
route:
- destination:
host: ratings
subset: v1
- route:
- destination:
host: ratings
subset: v1
---
５．可視化ツールを使ってみる
ここで、Istioと一緒にデプロイしたGrafana、Jaeger、Kialiの画面を見てみます。
ワークショップの手順ではkubectl port-forwardを使用してlocalhostへのアクセスを行っているのですが、外部からのアクセスができるように設定してみます。
外部アクセス可能なType:NodePortなServiceを作成
Grafana、Jaeger、KialiへアクセスするServiceはそれぞれgrafana、jaeger-query、kialiですが、現状はServiceのTypeがClusterIPのため外部アクセスできません。
そのため、TypeをNordPortに指定したサービスを追加で作成します。
$ kubectl expose service grafana --type=NodePort --name=grafana -svc -n istio-system
$ kubectl expose service jaeger-query --type=NodePort --name=jaeger-query-svc -n istio-system
$ kubectl expose service kiali --type=NodePort --name=kiali-svc -n istio-system
kubectl get pod,svc -n istio-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE
pod/grafana-7f6cd4bf56-tq5tp 1/1 Running 0 3d 10.0.0.21 ip-10-0-0-4.us-west-2.compute.internal
pod/istio-citadel-796c94878b-s7zc4 1/1 Running 0 3d 10.0.3.132 ip-10-0-3-130.us-west-2.compute.internal
pod/istio-cleanup-secrets-p94tx 0/1 Completed 0 3d 10.0.3.112 ip-10-0-3-130.us-west-2.compute.internal
pod/istio-egressgateway-864444d6ff-cgjh4 1/1 Running 0 3d 10.0.0.43 ip-10-0-0-4.us-west-2.compute.internal
pod/istio-galley-6c68c5dbcf-hg9rx 1/1 Running 0 3d 10.0.3.58 ip-10-0-3-130.us-west-2.compute.internal
pod/istio-grafana-post-install-2pcdx 0/1 Completed 0 3d 10.0.0.49 ip-10-0-0-4.us-west-2.compute.internal
pod/istio-ingressgateway-694576c7bb-kwsk4 1/1 Running 0 3d 10.0.3.143 ip-10-0-3-130.us-west-2.compute.internal
pod/istio-pilot-79f5f46dd5-ngcm6 2/2 Running 0 3d 10.0.0.28 ip-10-0-0-4.us-west-2.compute.internal
pod/istio-policy-5bd5578b94-w9sq4 2/2 Running 0 3d 10.0.0.204 ip-10-0-0-4.us-west-2.compute.internal
pod/istio-security-post-install-chzxk 0/1 Completed 0 3d 10.0.0.233 ip-10-0-0-4.us-west-2.compute.internal
pod/istio-sidecar-injector-6d8f88c98f-gbhtq 1/1 Running 1 3d 10.0.3.8 ip-10-0-3-130.us-west-2.compute.internal
pod/istio-telemetry-5598f86cd8-7c7bc 2/2 Running 0 3d 10.0.3.107 ip-10-0-3-130.us-west-2.compute.internal
pod/istio-tracing-7596597bd7-t5qjg 1/1 Running 0 3d 10.0.3.97 ip-10-0-3-130.us-west-2.compute.internal
pod/kiali-84cd87cd5d-pl29c 1/1 Running 0 3d 10.0.3.75 ip-10-0-3-130.us-west-2.compute.internal
pod/prometheus-76db5fddd5-w2477 1/1 Running 0 3d 10.0.3.184 ip-10-0-3-130.us-west-2.compute.internal
pod/servicegraph-df4d5f678-prxhn 1/1 Running 0 3d 10.0.3.144 ip-10-0-3-130.us-west-2.compute.internal
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/grafana ClusterIP 172.20.250.52 <none> 3000/TCP 3d app=grafana
service/grafana-svc NodePort 172.20.169.16 <none> 3000:31179/TCP 18m app=grafana
service/istio-citadel ClusterIP 172.20.5.227 <none> 8060/TCP,9093/TCP 3d istio=citadel
service/istio-egressgateway ClusterIP 172.20.157.133 <none> 80/TCP,443/TCP 3d app=istio-egressgateway,istio=egressgateway
service/istio-galley ClusterIP 172.20.253.156 <none> 443/TCP,9093/TCP 3d istio=galley
service/istio-ingressgateway LoadBalancer 172.20.14.123 a81256f2b46c011e9bec6068f6b115a8-1231799013.us-west-2.elb.amazonaws.com 80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:31060/TCP,8060:30178/TCP,853:31935/TCP,15030:30486/TCP,15031:31441/TCP 3d app=istio-ingressgateway,istio=ingressgateway
service/istio-pilot ClusterIP 172.20.217.14 <none> 15010/TCP,15011/TCP,8080/TCP,9093/TCP 3d istio=pilot
service/istio-policy ClusterIP 172.20.145.140 <none> 9091/TCP,15004/TCP,9093/TCP 3d istio-mixer-type=policy,istio=mixer
service/istio-sidecar-injector ClusterIP 172.20.84.237 <none> 443/TCP 3d istio=sidecar-injector
service/istio-telemetry ClusterIP 172.20.105.202 <none> 9091/TCP,15004/TCP,9093/TCP,42422/TCP 3d istio-mixer-type=telemetry,istio=mixer
service/jaeger-agent ClusterIP None <none> 5775/UDP,6831/UDP,6832/UDP 3d app=jaeger
service/jaeger-collector ClusterIP 172.20.154.91 <none> 14267/TCP,14268/TCP 3d app=jaeger
service/jaeger-query ClusterIP 172.20.132.103 <none> 16686/TCP 3d app=jaeger
service/jaeger-query-svc NodePort 172.20.87.213 <none> 16686:31270/TCP 3h app=jaeger
service/kiali ClusterIP 172.20.57.195 <none> 20001/TCP 3d app=kiali
service/kiali-svc NodePort 172.20.107.2 <none> 20001:32342/TCP 1h app=kiali
service/prometheus ClusterIP 172.20.97.85 <none> 9090/TCP 3d app=prometheus
service/servicegraph ClusterIP 172.20.23.26 <none> 8088/TCP 3d app=servicegraph
service/tracing ClusterIP 172.20.145.78 <none> 80/TCP 3d app=jaeger
service/zipkin ClusterIP 172.20.6.139 <none> 9411/TCP
これで各サービスでポート31179,31270,32342が新たに割り振られましたが、現時点でノードに割り当てられたセキュリティグループが通信を許可していないため、許可をしてあげます。
これでGrafana、Jaeger、Kialiが外部公開されました。
Grafana
例えばGrafanaの場合、get pod,svcした情報から、
①DNS：grafana-7f6cd4bf56-tq5tpのノードIP（ip-10-0-0-4.us-west-2.compute.internal）に対応するEC2インスタンスのパブリックDNS
②PORT：追加作成したServiceに割り当てられた31179番ポート
を使用して、以下のアドレスでGrafanaのダッシュボードへアクセスできます。
すでにIstio用のダッシュボードが自動で作られています。
コマンドでリクエストを投げまくってみてから、Istio Mesh Dashboardを見てみると、各Serviceへのアクセス状況が確認できます。
（reviewsへのRouting設定をv1:v3=50:50で設定中なので、v2へのアクセスがNoneとなっています）
export SMHOST=$(kubectl get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname} ' -n istio-system)
SMHOST="$(echo -e "${SMHOST}" | tr -d '[:space:]')"
while true; do curl -o /dev/null -s "${SMHOST}/productpage"; done
リソース使用状況
Jaeger
各サービスのトレース情報を表示できます
Kiali
ログイン情報にはkiali用secret作成時に指定したUsername＆passを使います。
（↓ここの'admin'、'1f2d1e2e67df'）
$ NAMESPACE=istio-system
$ echo -n 'admin' | base64
YWRtaW4=
$ echo -n '1f2d1e2e67df' | base64
MWYyZDFlMmU2N2Rm
Graphからはサービスメッシュを良い感じにグラフィカルに見ることができ、かつトラフィック状況をグラフで確認することができます。
各設定ファイルの検査もしてくれたりします。Destinationに一部無駄な定義がある（v2がないのにv2の定義がある）のでエラーになってます。
本当はGrafana&Jaegerとの統合ができるようで、KialiのGithubの通りにConfigmapのURL追加をやってみたもののうまくいかず、Kiali画面右上にアラームが出っぱなしになってます。（Error fetching Grafana Info., Error: [ Grafana URL is not set in Kiali configuration ]）
別途情報探してみようと思います。（Configmap修正以外に必要なことがわかる方いれば教えて下さい）
６．さいごに
基礎の基礎だけながらも記事を書きながらワークショップを進めたことで、なんとなく難しくてずっと避けてきたk8sも少し肌間隔を掴めてきたような気がします。
Istioも今回は一先ず基本的なところだけ触りましたが、まだまだCircuitBreakerやHealthCheck等々いろんな機能があるので、引き続き触ってみようと思います。
（でもやっぱりIstio難しいので、もうすぐGAされるらしいAppMeshがうまいことこのへんラッピングしてくれて、イイ感じなマネージドサービスとして君臨してくれるのを期待したいのが本音）