Istio, Prometheus, Zipkinを使ってKubernetes Cluster上のマイクロサービスの監視と分散トレースをテストしてみました。Kubernetes ClusterはIBM Cloudを利用しています。

Istioとは、Google、IBM、Lyftが開発し、2017年5月にオープンソース化したソフトウェアです。マイクロサービス間の通信を統一的な仕組みで制御する「サービスメッシュ」と呼ばれる機能を果たします。Istioによって、きめ細かなセキュリティの確保、流量制御、フェイルオーバー、ブルー／グリーンデプロイメント、カナリアデプロイメントなどが可能になります。

Kubernetesクラスターの作成および構成

IBM Cloudのアカウント作成後、以下のリンクより[クラスターの作成]をクリックします。
https://console.bluemix.net/containers-kubernetes/overview

ここでは以下のようなスペックで作成しました。IBM Cloudの場合、ワーカー・ノードは仮想サーバーだけでなくベア・メタルも選択できます。

領域：東京
クラスター・タイプ：標準
ロケーション：tok02
Kubernetesのバージョン：1.9.7
ハードウェアの分離：仮想-共有 u2c.2x4 (2 CPU, 4 GB RAM)
ローカルディスクの暗号化：Yes
ワーカーノード：3
プライベートVLAN：任意のVLANを選択
パブリックVLAN：任意のVLANを選択
クラスター名：mycluster

ワーカーノード（2コア4GB）×3台で月額課金は￥1,631となりました。

5分くらいでクラスターがデプロイ済みになります。

作成したmyclusterのワーカーノードを確認します。

[メトリック]をクリックするとIBM Cloudモニタリングを利用したクラスターのモニタリングができます。ダッシュボードにはGrafanaが使われており、自由にカスタマイズできます。

[ロギングの使用可能化]をクリックすると、IBM Cloudロギングが利用できます。Kibanaが使われています。

各ワーカーノードには、プライベートIPとパブリックIPが付与されていることが分かります。

[Kubernetesダッシュボード]をクリックすれば、標準のKubernetesコンソールも使えます。

IBM Cloud CLIをクライアントにインストールすれば、Kubernetes CLIによる操作もできます。詳細は、kubectl Cheat Sheetを参照。

$ bx login -sso -a https://api.au-syd.bluemix.net
$ bx cs region-set ap-north
$ bx cs cluster-config mycluster
$ export KUBECONFIG=/Users/ibm/.bluemix/plugins/container-service/clusters/mycluster/kube-config-tok02-mycluster.yml

$ kubectl get nodes
NAME            STATUS    AGE       VERSION
10.129.50.227   Ready     30m       v1.9.7-2+231cc32d0a1119
10.129.50.230   Ready     32m       v1.9.7-2+231cc32d0a1119
10.129.50.231   Ready     31m       v1.9.7-2+231cc32d0a1119

$ kubectl get services
NAME         CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   172.21.0.1   <none>        443/TCP   42m

Istioのインストール

Istio: Traffic Management for your Microservicesを参考にサンプルアプリにbookinfoにIstioをインストールして、トラフィックを管理します。全体像は下図のとおり。

クライアントにリポジトリを複製し、Istioをダウンロードする作業ディレクトリを作成します。

$ mkdir ibm
$ cd ibm
$ git clone https://github.com/IBM/traffic-management-for-your-microservices-using-istio.git demo

クライアントに最新のIstio（istio-0.7.1-osx.tar.gz）をダウンロードします。

$ curl -L https://git.io/getLatestIstio | sh -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  1448  100  1448    0     0   1290      0  0:00:01  0:00:01 --:--:--  1290
Downloading istio-0.7.1 from https://github.com/istio/istio/releases/download/0.7.1/istio-0.7.1-osx.tar.gz ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   612    0   612    0     0    696      0 --:--:-- --:--:-- --:--:--   754
100 11.2M  100 11.2M    0     0   417k      0  0:00:27  0:00:27 --:--:--  374k
Downloaded into istio-0.7.1:
LICENSE     bin     istio.VERSION   tools
README.md   install     samples
Add /Users/asasaki/ibm/istio-0.7.1/bin to your path; e.g copy paste in your shell and/or ~/.profile:
export PATH="$PATH:/Users/asasaki/ibm/istio-0.7.1/bin"
$ mv istio-0.7.1 istio
$ export PATH=$PWD/istio/bin:$PATH

前ステップでIBM Cloud上に作成したKubernetesクラスターにIstioをデプロイします。

$ kubectl apply -f istio/install/kubernetes/istio.yaml
namespace "istio-system" created
clusterrole "istio-pilot-istio-system" created
clusterrole "istio-sidecar-injector-istio-system" created
clusterrole "istio-mixer-istio-system" created
clusterrole "istio-mixer-validator-istio-system" created
clusterrole "istio-ca-istio-system" created
clusterrole "istio-sidecar-istio-system" created
clusterrolebinding "istio-pilot-admin-role-binding-istio-system" created
clusterrolebinding "istio-sidecar-injector-admin-role-binding-istio-system" created
clusterrolebinding "istio-ca-role-binding-istio-system" created
clusterrolebinding "istio-ingress-admin-role-binding-istio-system" created
clusterrolebinding "istio-sidecar-role-binding-istio-system" created
clusterrolebinding "istio-mixer-admin-role-binding-istio-system" created
clusterrolebinding "istio-mixer-validator-admin-role-binding-istio-system" created
configmap "istio-mixer" created
service "istio-mixer" created
serviceaccount "istio-mixer-service-account" created
deployment "istio-mixer" created
customresourcedefinition "rules.config.istio.io" created
customresourcedefinition "attributemanifests.config.istio.io" created
customresourcedefinition "circonuses.config.istio.io" created
customresourcedefinition "deniers.config.istio.io" created
customresourcedefinition "fluentds.config.istio.io" created
customresourcedefinition "kubernetesenvs.config.istio.io" created
customresourcedefinition "listcheckers.config.istio.io" created
customresourcedefinition "memquotas.config.istio.io" created
customresourcedefinition "noops.config.istio.io" created
customresourcedefinition "opas.config.istio.io" created
customresourcedefinition "prometheuses.config.istio.io" created
customresourcedefinition "rbacs.config.istio.io" created
customresourcedefinition "servicecontrols.config.istio.io" created
customresourcedefinition "solarwindses.config.istio.io" created
customresourcedefinition "stackdrivers.config.istio.io" created
customresourcedefinition "statsds.config.istio.io" created
customresourcedefinition "stdios.config.istio.io" created
customresourcedefinition "apikeys.config.istio.io" created
customresourcedefinition "authorizations.config.istio.io" created
customresourcedefinition "checknothings.config.istio.io" created
customresourcedefinition "kuberneteses.config.istio.io" created
customresourcedefinition "listentries.config.istio.io" created
customresourcedefinition "logentries.config.istio.io" created
customresourcedefinition "metrics.config.istio.io" created
customresourcedefinition "quotas.config.istio.io" created
customresourcedefinition "reportnothings.config.istio.io" created
customresourcedefinition "servicecontrolreports.config.istio.io" created
customresourcedefinition "tracespans.config.istio.io" created
customresourcedefinition "serviceroles.config.istio.io" created
customresourcedefinition "servicerolebindings.config.istio.io" created
attributemanifest "istioproxy" created
attributemanifest "kubernetes" created
stdio "handler" created
logentry "accesslog" created
rule "stdio" created
metric "requestcount" created
metric "requestduration" created
metric "requestsize" created
metric "responsesize" created
metric "tcpbytesent" created
metric "tcpbytereceived" created
prometheus "handler" created
rule "promhttp" created
rule "promtcp" created
kubernetesenv "handler" created
rule "kubeattrgenrulerule" created
rule "tcpkubeattrgenrulerule" created
kubernetes "attributes" created
configmap "istio" created
customresourcedefinition "destinationpolicies.config.istio.io" created
customresourcedefinition "egressrules.config.istio.io" created
customresourcedefinition "routerules.config.istio.io" created
customresourcedefinition "virtualservices.networking.istio.io" created
customresourcedefinition "destinationrules.networking.istio.io" created
customresourcedefinition "externalservices.networking.istio.io" created
service "istio-pilot" created
serviceaccount "istio-pilot-service-account" created
deployment "istio-pilot" created
service "istio-ingress" created
serviceaccount "istio-ingress-service-account" created
deployment "istio-ingress" created
serviceaccount "istio-ca-service-account" created
deployment "istio-ca" created

istilo.yamlファイルは以下のとおり。

istio.yaml

$ cat istio/install/kubernetes/istio.yaml
# GENERATED FILE. Use with Kubernetes 1.5+
# TO UPDATE, modify files in install/kubernetes/templates and run install/updateVersion.sh
# Mixer
apiVersion: v1
kind: Service
metadata:
  name: istio-mixer
  labels:
    istio: mixer
spec:
  ports:
  - name: tcp
    port: 9091
  - name: configapi
    port: 9094
  - name: prometheus
    port: 42422
  selector:
    istio: mixer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-mixer
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        alpha.istio.io/sidecar: ignore
      labels:
        istio: mixer
    spec:
      containers:
      - name: mixer
        image: docker.io/istio/mixer:0.1.6
        imagePullPolicy: Always
        ports:
        - containerPort: 9091
        - containerPort: 9094
        - containerPort: 42422
        args:
          - --configStoreURL=fs:///etc/opt/mixer/configroot
          - --logtostderr
          - -v
          - "3"
---
# Pilot service for discovery
apiVersion: v1
kind: ConfigMap
metadata:
  name: istio
data:
  mesh: |-
    # Uncomment the following line to enable mutual TLS between proxies
    # authPolicy: MUTUAL_TLS
    mixerAddress: istio-mixer:9091
    discoveryAddress: istio-pilot:8080
    ingressService: istio-ingress
    zipkinAddress: zipkin:9411
---
apiVersion: v1
kind: Service
metadata:
  name: istio-pilot
  labels:
    istio: pilot
spec:
  ports:
  - port: 8080
    name: http-discovery
  - port: 8081
    name: http-apiserver
  selector:
    istio: pilot
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-pilot-service-account
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-pilot
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        alpha.istio.io/sidecar: ignore
      labels:
        istio: pilot
    spec:
      serviceAccountName: istio-pilot-service-account
      containers:
      - name: discovery
        image: docker.io/istio/pilot:0.1.6
        imagePullPolicy: Always
        args: ["discovery", "-v", "2"]
        ports:
        - containerPort: 8080
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
      - name: apiserver
        image: docker.io/istio/pilot:0.1.6
        imagePullPolicy: Always
        args: ["apiserver", "-v", "2"]
        ports:
        - containerPort: 8081
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
---
################################
# Istio ingress controller
################################
apiVersion: v1
kind: Service
metadata:
  name: istio-ingress
  labels:
    istio: ingress
spec:
  type: LoadBalancer
  ports:
  - port: 80
#   nodePort: 32000
    name: http
  - port: 443
    name: https
  selector:
    istio: ingress
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-ingress-service-account
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-ingress
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        alpha.istio.io/sidecar: ignore
      labels:
        istio: ingress
    spec:
      serviceAccountName: istio-ingress-service-account
      containers:
      - name: istio-ingress
        image: docker.io/istio/proxy_debug:0.1.6
        args: ["proxy", "ingress", "-v", "2"]
        imagePullPolicy: Always
        ports:
        - containerPort: 80
        - containerPort: 443
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
---

################################
# Istio egress envoy
################################
apiVersion: v1
kind: Service
metadata:
  name: istio-egress
spec:
  ports:
  - port: 80
  selector:
    istio: egress
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-egress
spec:
  replicas: 1
  template:
    metadata:
      labels:
        istio: egress
    spec:
      containers:
      - name: proxy
        image: docker.io/istio/proxy_debug:0.1.6
        imagePullPolicy: Always
        args: ["proxy", "egress", "-v", "2"]
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
---

Istioサイドカー付きのサンプルアプリBookInfoのデプロイ

サンプルアプリBookInfoをデプロイする

$ kubectl apply -f <(istioctl kube-inject -f istio/samples/bookinfo/kube/bookinfo.yaml)
service "details" created
deployment "details-v1" created
service "ratings" created
deployment "ratings-v1" created
service "reviews" created
deployment "reviews-v1" created
deployment "reviews-v2" created
deployment "reviews-v3" created
service "productpage" created
deployment "productpage-v1" created
ingress "gateway" created

サンプルアプリBookInfoはPython、Java、Ruby、Node.jsの4種類の異なる言語からなるマイクロサービスです。bookinfo.yamlの中身は以下のとおり。

bookinfo.yaml

$ cat istio/samples/bookinfo/kube/bookinfo.yaml
# Copyright 2017 Istio Authors
#
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

##################################################################################################
# Details service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: details
  labels:
    app: details
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: details
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: details-v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: details
        version: v1
    spec:
      containers:
      - name: details
        image: istio/examples-bookinfo-details-v1:1.5.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
---
##################################################################################################
# Ratings service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: ratings
  labels:
    app: ratings
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: ratings
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ratings-v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: ratings
        version: v1
    spec:
      containers:
      - name: ratings
        image: istio/examples-bookinfo-ratings-v1:1.5.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
---
##################################################################################################
# Reviews service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: reviews
  labels:
    app: reviews
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: reviews
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: reviews-v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: reviews
        version: v1
    spec:
      containers:
      - name: reviews
        image: istio/examples-bookinfo-reviews-v1:1.5.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: reviews-v2
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: reviews
        version: v2
    spec:
      containers:
      - name: reviews
        image: istio/examples-bookinfo-reviews-v2:1.5.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: reviews-v3
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: reviews
        version: v3
    spec:
      containers:
      - name: reviews
        image: istio/examples-bookinfo-reviews-v3:1.5.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
---
##################################################################################################
# Productpage services
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: productpage
  labels:
    app: productpage
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: productpage
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: productpage-v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: productpage
        version: v1
    spec:
      containers:
      - name: productpage
        image: istio/examples-bookinfo-productpage-v1:1.5.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
---
###########################################################################
# Ingress resource (gateway)
##########################################################################
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gateway
  annotations:
    kubernetes.io/ingress.class: "istio"
spec:
  rules:
  - http:
      paths:
      - path: /productpage
        backend:
          serviceName: productpage
          servicePort: 9080
      - path: /login
        backend:
          serviceName: productpage
          servicePort: 9080
      - path: /logout
        backend:
          serviceName: productpage
          servicePort: 9080
      - path: /api/v1/products.*
        backend:
          serviceName: productpage
          servicePort: 9080
---

稼働しているサンプルアプリBookInfoのPodを確認します。

$ kubectl get pods
NAME                              READY     STATUS    RESTARTS   AGE
details-v1-55496dcd64-72hhw       2/2       Running   0          3m
productpage-v1-586897968d-7xlmx   2/2       Running   0          3m
ratings-v1-6d9f5df564-zrbdb       2/2       Running   0          3m
reviews-v1-5985df7dd4-4nmch       2/2       Running   0          3m
reviews-v2-856d5b976-bn8vp        2/2       Running   0          3m
reviews-v3-c4fbb98d8-28lrh        2/2       Running   0          3m

アプリケーションにアクセスするために、アプリケーションのパブリックIPアドレスを確認します。ここでは、ゲートウェイは以下のIPアドレスになっています。

$ kubectl get ingress -o wide
NAME      HOSTS     ADDRESS        PORTS     AGE
gateway   *         169.56.28.18   80        4m

クライアントで環境変数をセットします。

$ export GATEWAY_URL=169.56.28.18:80

ブラウザで、http://169.56.28.18:80/productpageにアクセスします。ブラウザを更新するとサンプルアプリBookinfoのレビューセクションが以下の3つのバージョン（v1、v2、v3）にラウンドロビンで切り替わることを確認します。

PrometheusとGrafanaによるメトリックとログの収集

次に、クラスタ内のサービスのテレメトリーを収集するようにIstio Mixerを設定します。KubernetesクラスターにIstioのアドオンをインストールします。

Prometheusをデプロイします。

$ kubectl apply -f istio/install/kubernetes/addons/prometheus.yaml
configmap "prometheus" created
service "prometheus" created
deployment "prometheus" created
serviceaccount "prometheus" created
clusterrole "prometheus" created
clusterrolebinding "prometheus" created

GrafanaをKubernetesクラスターにデプロイします。

$ kubectl apply -f istio/install/kubernetes/addons/grafana.yaml
service "grafana" created
deployment "grafana" created
serviceaccount "grafana" created

Grafanaのダッシュボードをhttp://localhost:3000にポートフォワーディングします。

$ kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000

ブラウザでIstioダッシュボードが表示されることを確認します。

新しいテレメトリデータを収集するために、ミキサー・ルールを適用します。ここでは、レビュー・サービスのレスポンス・サイズのログを生成するようにします。設定のためのYAMLファイルnew_telemetry.yamlを利用します（参考：https://istio.io/docs/tasks/telemetry/metrics-logs.html）。

$ istioctl create -f demo/new_telemetry.yaml
Created config metric/istio-system/doublerequestcount at revision 7063
Created config prometheus/istio-system/doublehandler at revision 7064
Created config rule/istio-system/doubleprom at revision 7065
Created config logentry/istio-system/newlog at revision 7066
Created config stdio/istio-system/newhandler at revision 7067
Created config rule/istio-system/newlogstdio at revision 7068

new_telemetry.yamlファイルの中身は以下のとおりです。

new_telemetry.yaml

# Configuration for metric instances
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
  name: doublerequestcount
  namespace: istio-system
spec:
  value: "2" # count each request twice
  dimensions:
    source: source.service | "unknown"
    destination: destination.service | "unknown"
    message: '"twice the fun!"'
  monitored_resource_type: '"UNSPECIFIED"'
---
# Configuration for a Prometheus handler
apiVersion: "config.istio.io/v1alpha2"
kind: prometheus
metadata:
  name: doublehandler
  namespace: istio-system
spec:
  metrics:
  - name: double_request_count # Prometheus metric name
    instance_name: doublerequestcount.metric.istio-system # Mixer instance name (fully-qualified)
    kind: COUNTER
    label_names:
    - source
    - destination
    - message
---
# Rule to send metric instances to a Prometheus handler
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: doubleprom
  namespace: istio-system
spec:
  actions:
  - handler: doublehandler.prometheus
    instances:
    - doublerequestcount.metric
---
# Configuration for logentry instances
apiVersion: "config.istio.io/v1alpha2"
kind: logentry
metadata:
  name: newlog
  namespace: istio-system
spec:
  severity: '"warning"'
  timestamp: request.time
  variables:
    source: source.labels["app"] | source.service | "unknown"
    user: source.user | "unknown"
    destination: destination.labels["app"] | destination.service | "unknown"
    responseCode: response.code | 0
    responseSize: response.size | 0
    latency: response.duration | "0ms"
  monitored_resource_type: '"UNSPECIFIED"'
---
# Configuration for a stdio handler
apiVersion: "config.istio.io/v1alpha2"
kind: stdio
metadata:
  name: newhandler
  namespace: istio-system
spec:
 severity_levels:
   warning: 1 # Params.Level.WARNING
 outputAsJson: true
---
# Rule to send logentry instances to a stdio handler
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: newlogstdio
  namespace: istio-system
spec:
  match: "true" # match for all requests
  actions:
   - handler: newhandler.stdio
     instances:
     - newlog.logentry
---

forループでサンプルアプリBookInfo (http://169.56.28.18/productpage)にトラフィックを送信します。

$ for i in {1..5}; do echo -n .; curl -s http://${GATEWAY_URL}/productpage > /dev/null; done

Grafanaダッシュボードに再度アクセスして、新しいメトリックが収集されていることを確認します。

ログ・ストリームが作成されていることと、リクエストが生成されていることが確認できます。

$ kubectl -n istio-system logs $(kubectl -n istio-system get pods -l istio=mixer -o jsonpath='{.items[0].metadata.name}') mixer | grep \"instance\":\"newlog.logentry.istio-system\"

{"level":"warn","time":"2018-05-18T08:03:45.072460Z","instance":"newlog.logentry.istio-system","destination":"reviews","latency":"8.812174ms","responseCode":200,"responseSize":379,"source":"productpage","user":"unknown"}
{"level":"warn","time":"2018-05-18T08:03:45.163212Z","instance":"newlog.logentry.istio-system","destination":"reviews","latency":"18.7925ms","responseCode":200,"responseSize":375,"source":"productpage","user":"unknown"}
{"level":"warn","time":"2018-05-18T08:03:45.151312Z","instance":"newlog.logentry.istio-system","destination":"istio-ingress.istio-system.svc.cluster.local","latency":"34.802885ms","responseCode":200,"responseSize":5719,"source":"unknown","user":"unknown"}

Zipkinによる分散トレース

Zipkinをインストールします。

$ kubectl apply -f istio/install/kubernetes/addons/zipkin.yaml
deployment "zipkin" created
service "zipkin" created

Zipkinダッシュボードをポート・フォワーディングします。

$ kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=zipkin -o jsonpath='{.items[0].metadata.name}') 9411:9411

forループでサンプルアプリBookInfoにアクセスし、ブラウザでhttp://localhost:9411/zipkin/を確認します。BookInfoに送信したトラフィックの詳細が表示されます。これは、ここでは、http://169.56.28.18/productpageへのリクエストに要した時間を表示させています。

以上

Istio, Prometheus, Zipkinによるマイクロサービスの監視と分散トレース

Kubernetesクラスターの作成および構成

Istioのインストール

Istioサイドカー付きのサンプルアプリBookInfoのデプロイ

PrometheusとGrafanaによるメトリックとログの収集

Zipkinによる分散トレース