Istio, Prometheus, Zipkinを使ってKubernetes Cluster上のマイクロサービスの監視と分散トレースをテストしてみました。Kubernetes ClusterはIBM Cloudを利用しています。
Istioとは、Google、IBM、Lyftが開発し、2017年5月にオープンソース化したソフトウェアです。マイクロサービス間の通信を統一的な仕組みで制御する「サービスメッシュ」と呼ばれる機能を果たします。Istioによって、きめ細かなセキュリティの確保、流量制御、フェイルオーバー、ブルー／グリーンデプロイメント、カナリアデプロイメントなどが可能になります。
Kubernetesクラスターの作成および構成
IBM Cloudのアカウント作成後、以下のリンクより[クラスターの作成]をクリックします。
https://console.bluemix.net/containers-kubernetes/overview
ここでは以下のようなスペックで作成しました。IBM Cloudの場合、ワーカー・ノードは仮想サーバーだけでなくベア・メタルも選択できます。
- 領域：東京
- クラスター・タイプ：標準
- ロケーション：tok02
- Kubernetesのバージョン：1.9.7
- ハードウェアの分離：仮想-共有 u2c.2x4 (2 CPU, 4 GB RAM)
- ローカルディスクの暗号化：Yes
- ワーカーノード：3
- プライベートVLAN：任意のVLANを選択
- パブリックVLAN：任意のVLANを選択
- クラスター名：mycluster
ワーカーノード（2コア4GB）×3台で月額課金は￥1,631となりました。
5分くらいでクラスターがデプロイ済みになります。
作成したmyclusterのワーカーノードを確認します。
[メトリック]をクリックするとIBM Cloudモニタリングを利用したクラスターのモニタリングができます。ダッシュボードにはGrafanaが使われており、自由にカスタマイズできます。
[ロギングの使用可能化]をクリックすると、IBM Cloudロギングが利用できます。Kibanaが使われています。
各ワーカーノードには、プライベートIPとパブリックIPが付与されていることが分かります。
[Kubernetesダッシュボード]をクリックすれば、標準のKubernetesコンソールも使えます。
IBM Cloud CLIをクライアントにインストールすれば、Kubernetes CLIによる操作もできます。詳細は、kubectl Cheat Sheetを参照。
$ bx login -sso -a https://api.au-syd.bluemix.net
$ bx cs region-set ap-north
$ bx cs cluster-config mycluster
$ export KUBECONFIG=/Users/ibm/.bluemix/plugins/container-service/clusters/mycluster/kube-config-tok02-mycluster.yml
$ kubectl get nodes
NAME STATUS AGE VERSION
10.129.50.227 Ready 30m v1.9.7-2+231cc32d0a1119
10.129.50.230 Ready 32m v1.9.7-2+231cc32d0a1119
10.129.50.231 Ready 31m v1.9.7-2+231cc32d0a1119
$ kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 172.21.0.1 <none> 443/TCP 42m
Istioのインストール
Istio: Traffic Management for your Microservicesを参考にサンプルアプリにbookinfoにIstioをインストールして、トラフィックを管理します。全体像は下図のとおり。
クライアントにリポジトリを複製し、Istioをダウンロードする作業ディレクトリを作成します。
$ mkdir ibm
$ cd ibm
$ git clone https://github.com/IBM/traffic-management-for-your-microservices-using-istio.git demo
クライアントに最新のIstio（istio-0.7.1-osx.tar.gz）をダウンロードします。
$ curl -L https://git.io/getLatestIstio | sh -
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 1448 100 1448 0 0 1290 0 0:00:01 0:00:01 --:--:-- 1290
Downloading istio-0.7.1 from https://github.com/istio/istio/releases/download/0.7.1/istio-0.7.1-osx.tar.gz ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 612 0 612 0 0 696 0 --:--:-- --:--:-- --:--:-- 754
100 11.2M 100 11.2M 0 0 417k 0 0:00:27 0:00:27 --:--:-- 374k
Downloaded into istio-0.7.1:
LICENSE bin istio.VERSION tools
README.md install samples
Add /Users/asasaki/ibm/istio-0.7.1/bin to your path; e.g copy paste in your shell and/or ~/.profile:
export PATH="$PATH:/Users/asasaki/ibm/istio-0.7.1/bin"
$ mv istio-0.7.1 istio
$ export PATH=$PWD/istio/bin:$PATH
前ステップでIBM Cloud上に作成したKubernetesクラスターにIstioをデプロイします。
$ kubectl apply -f istio/install/kubernetes/istio.yaml
namespace "istio-system" created
clusterrole "istio-pilot-istio-system" created
clusterrole "istio-sidecar-injector-istio-system" created
clusterrole "istio-mixer-istio-system" created
clusterrole "istio-mixer-validator-istio-system" created
clusterrole "istio-ca-istio-system" created
clusterrole "istio-sidecar-istio-system" created
clusterrolebinding "istio-pilot-admin-role-binding-istio-system" created
clusterrolebinding "istio-sidecar-injector-admin-role-binding-istio-system" created
clusterrolebinding "istio-ca-role-binding-istio-system" created
clusterrolebinding "istio-ingress-admin-role-binding-istio-system" created
clusterrolebinding "istio-sidecar-role-binding-istio-system" created
clusterrolebinding "istio-mixer-admin-role-binding-istio-system" created
clusterrolebinding "istio-mixer-validator-admin-role-binding-istio-system" created
configmap "istio-mixer" created
service "istio-mixer" created
serviceaccount "istio-mixer-service-account" created
deployment "istio-mixer" created
customresourcedefinition "rules.config.istio.io" created
customresourcedefinition "attributemanifests.config.istio.io" created
customresourcedefinition "circonuses.config.istio.io" created
customresourcedefinition "deniers.config.istio.io" created
customresourcedefinition "fluentds.config.istio.io" created
customresourcedefinition "kubernetesenvs.config.istio.io" created
customresourcedefinition "listcheckers.config.istio.io" created
customresourcedefinition "memquotas.config.istio.io" created
customresourcedefinition "noops.config.istio.io" created
customresourcedefinition "opas.config.istio.io" created
customresourcedefinition "prometheuses.config.istio.io" created
customresourcedefinition "rbacs.config.istio.io" created
customresourcedefinition "servicecontrols.config.istio.io" created
customresourcedefinition "solarwindses.config.istio.io" created
customresourcedefinition "stackdrivers.config.istio.io" created
customresourcedefinition "statsds.config.istio.io" created
customresourcedefinition "stdios.config.istio.io" created
customresourcedefinition "apikeys.config.istio.io" created
customresourcedefinition "authorizations.config.istio.io" created
customresourcedefinition "checknothings.config.istio.io" created
customresourcedefinition "kuberneteses.config.istio.io" created
customresourcedefinition "listentries.config.istio.io" created
customresourcedefinition "logentries.config.istio.io" created
customresourcedefinition "metrics.config.istio.io" created
customresourcedefinition "quotas.config.istio.io" created
customresourcedefinition "reportnothings.config.istio.io" created
customresourcedefinition "servicecontrolreports.config.istio.io" created
customresourcedefinition "tracespans.config.istio.io" created
customresourcedefinition "serviceroles.config.istio.io" created
customresourcedefinition "servicerolebindings.config.istio.io" created
attributemanifest "istioproxy" created
attributemanifest "kubernetes" created
stdio "handler" created
logentry "accesslog" created
rule "stdio" created
metric "requestcount" created
metric "requestduration" created
metric "requestsize" created
metric "responsesize" created
metric "tcpbytesent" created
metric "tcpbytereceived" created
prometheus "handler" created
rule "promhttp" created
rule "promtcp" created
kubernetesenv "handler" created
rule "kubeattrgenrulerule" created
rule "tcpkubeattrgenrulerule" created
kubernetes "attributes" created
configmap "istio" created
customresourcedefinition "destinationpolicies.config.istio.io" created
customresourcedefinition "egressrules.config.istio.io" created
customresourcedefinition "routerules.config.istio.io" created
customresourcedefinition "virtualservices.networking.istio.io" created
customresourcedefinition "destinationrules.networking.istio.io" created
customresourcedefinition "externalservices.networking.istio.io" created
service "istio-pilot" created
serviceaccount "istio-pilot-service-account" created
deployment "istio-pilot" created
service "istio-ingress" created
serviceaccount "istio-ingress-service-account" created
deployment "istio-ingress" created
serviceaccount "istio-ca-service-account" created
deployment "istio-ca" created
istilo.yamlファイルは以下のとおり。
istio.yaml
$ cat istio/install/kubernetes/istio.yaml
# GENERATED FILE. Use with Kubernetes 1.5+
# TO UPDATE, modify files in install/kubernetes/templates and run install/updateVersion.sh
# Mixer
apiVersion: v1
kind: Service
metadata:
name: istio-mixer
labels:
istio: mixer
spec:
ports:
- name: tcp
port: 9091
- name: configapi
port: 9094
- name: prometheus
port: 42422
selector:
istio: mixer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: istio-mixer
spec:
replicas: 1
template:
metadata:
annotations:
alpha.istio.io/sidecar: ignore
labels:
istio: mixer
spec:
containers:
- name: mixer
image: docker.io/istio/mixer:0.1.6
imagePullPolicy: Always
ports:
- containerPort: 9091
- containerPort: 9094
- containerPort: 42422
args:
- --configStoreURL=fs:///etc/opt/mixer/configroot
- --logtostderr
- -v
- "3"
---
# Pilot service for discovery
apiVersion: v1
kind: ConfigMap
metadata:
name: istio
data:
mesh: |-
# Uncomment the following line to enable mutual TLS between proxies
# authPolicy: MUTUAL_TLS
mixerAddress: istio-mixer:9091
discoveryAddress: istio-pilot:8080
ingressService: istio-ingress
zipkinAddress: zipkin:9411
---
apiVersion: v1
kind: Service
metadata:
name: istio-pilot
labels:
istio: pilot
spec:
ports:
- port: 8080
name: http-discovery
- port: 8081
name: http-apiserver
selector:
istio: pilot
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: istio-pilot-service-account
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: istio-pilot
spec:
replicas: 1
template:
metadata:
annotations:
alpha.istio.io/sidecar: ignore
labels:
istio: pilot
spec:
serviceAccountName: istio-pilot-service-account
containers:
- name: discovery
image: docker.io/istio/pilot:0.1.6
imagePullPolicy: Always
args: ["discovery", "-v", "2"]
ports:
- containerPort: 8080
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: apiserver
image: docker.io/istio/pilot:0.1.6
imagePullPolicy: Always
args: ["apiserver", "-v", "2"]
ports:
- containerPort: 8081
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
---
################################
# Istio ingress controller
################################
apiVersion: v1
kind: Service
metadata:
name: istio-ingress
labels:
istio: ingress
spec:
type: LoadBalancer
ports:
- port: 80
# nodePort: 32000
name: http
- port: 443
name: https
selector:
istio: ingress
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: istio-ingress-service-account
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: istio-ingress
spec:
replicas: 1
template:
metadata:
annotations:
alpha.istio.io/sidecar: ignore
labels:
istio: ingress
spec:
serviceAccountName: istio-ingress-service-account
containers:
- name: istio-ingress
image: docker.io/istio/proxy_debug:0.1.6
args: ["proxy", "ingress", "-v", "2"]
imagePullPolicy: Always
ports:
- containerPort: 80
- containerPort: 443
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
---
################################
# Istio egress envoy
################################
apiVersion: v1
kind: Service
metadata:
name: istio-egress
spec:
ports:
- port: 80
selector:
istio: egress
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: istio-egress
spec:
replicas: 1
template:
metadata:
labels:
istio: egress
spec:
containers:
- name: proxy
image: docker.io/istio/proxy_debug:0.1.6
imagePullPolicy: Always
args: ["proxy", "egress", "-v", "2"]
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
---
Istioサイドカー付きのサンプルアプリBookInfoのデプロイ
サンプルアプリBookInfoをデプロイする
$ kubectl apply -f <(istioctl kube-inject -f istio/samples/bookinfo/kube/bookinfo.yaml)
service "details" created
deployment "details-v1" created
service "ratings" created
deployment "ratings-v1" created
service "reviews" created
deployment "reviews-v1" created
deployment "reviews-v2" created
deployment "reviews-v3" created
service "productpage" created
deployment "productpage-v1" created
ingress "gateway" created
サンプルアプリBookInfoはPython、Java、Ruby、Node.jsの4種類の異なる言語からなるマイクロサービスです。bookinfo.yamlの中身は以下のとおり。
bookinfo.yaml
$ cat istio/samples/bookinfo/kube/bookinfo.yaml
# Copyright 2017 Istio Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##################################################################################################
# Details service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: details
labels:
app: details
spec:
ports:
- port: 9080
name: http
selector:
app: details
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: details-v1
spec:
replicas: 1
template:
metadata:
labels:
app: details
version: v1
spec:
containers:
- name: details
image: istio/examples-bookinfo-details-v1:1.5.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
##################################################################################################
# Ratings service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: ratings
labels:
app: ratings
spec:
ports:
- port: 9080
name: http
selector:
app: ratings
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ratings-v1
spec:
replicas: 1
template:
metadata:
labels:
app: ratings
version: v1
spec:
containers:
- name: ratings
image: istio/examples-bookinfo-ratings-v1:1.5.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
##################################################################################################
# Reviews service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: reviews
labels:
app: reviews
spec:
ports:
- port: 9080
name: http
selector:
app: reviews
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: reviews-v1
spec:
replicas: 1
template:
metadata:
labels:
app: reviews
version: v1
spec:
containers:
- name: reviews
image: istio/examples-bookinfo-reviews-v1:1.5.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: reviews-v2
spec:
replicas: 1
template:
metadata:
labels:
app: reviews
version: v2
spec:
containers:
- name: reviews
image: istio/examples-bookinfo-reviews-v2:1.5.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: reviews-v3
spec:
replicas: 1
template:
metadata:
labels:
app: reviews
version: v3
spec:
containers:
- name: reviews
image: istio/examples-bookinfo-reviews-v3:1.5.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
##################################################################################################
# Productpage services
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: productpage
labels:
app: productpage
spec:
ports:
- port: 9080
name: http
selector:
app: productpage
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: productpage-v1
spec:
replicas: 1
template:
metadata:
labels:
app: productpage
version: v1
spec:
containers:
- name: productpage
image: istio/examples-bookinfo-productpage-v1:1.5.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
###########################################################################
# Ingress resource (gateway)
##########################################################################
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gateway
annotations:
kubernetes.io/ingress.class: "istio"
spec:
rules:
- http:
paths:
- path: /productpage
backend:
serviceName: productpage
servicePort: 9080
- path: /login
backend:
serviceName: productpage
servicePort: 9080
- path: /logout
backend:
serviceName: productpage
servicePort: 9080
- path: /api/v1/products.*
backend:
serviceName: productpage
servicePort: 9080
---
稼働しているサンプルアプリBookInfoのPodを確認します。
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
details-v1-55496dcd64-72hhw 2/2 Running 0 3m
productpage-v1-586897968d-7xlmx 2/2 Running 0 3m
ratings-v1-6d9f5df564-zrbdb 2/2 Running 0 3m
reviews-v1-5985df7dd4-4nmch 2/2 Running 0 3m
reviews-v2-856d5b976-bn8vp 2/2 Running 0 3m
reviews-v3-c4fbb98d8-28lrh 2/2 Running 0 3m
アプリケーションにアクセスするために、アプリケーションのパブリックIPアドレスを確認します。 ここでは、ゲートウェイは以下のIPアドレスになっています。
$ kubectl get ingress -o wide
NAME HOSTS ADDRESS PORTS AGE
gateway * 169.56.28.18 80 4m
クライアントで環境変数をセットします。
$ export GATEWAY_URL=169.56.28.18:80
ブラウザで、http://169.56.28.18:80/productpageにアクセスします。ブラウザを更新するとサンプルアプリBookinfoのレビューセクションが以下の3つのバージョン（v1、v2、v3）にラウンドロビンで切り替わることを確認します。
PrometheusとGrafanaによるメトリックとログの収集
次に、クラスタ内のサービスのテレメトリーを収集するようにIstio Mixerを設定します。KubernetesクラスターにIstioのアドオンをインストールします。
Prometheusをデプロイします。
$ kubectl apply -f istio/install/kubernetes/addons/prometheus.yaml
configmap "prometheus" created
service "prometheus" created
deployment "prometheus" created
serviceaccount "prometheus" created
clusterrole "prometheus" created
clusterrolebinding "prometheus" created
GrafanaをKubernetesクラスターにデプロイします。
$ kubectl apply -f istio/install/kubernetes/addons/grafana.yaml
service "grafana" created
deployment "grafana" created
serviceaccount "grafana" created
Grafanaのダッシュボードをhttp://localhost:3000にポートフォワーディングします。
$ kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000
ブラウザでIstioダッシュボードが表示されることを確認します。
新しいテレメトリデータを収集するために、ミキサー・ルールを適用します。ここでは、レビュー・サービスのレスポンス・サイズのログを生成するようにします。設定のためのYAMLファイルnew_telemetry.yamlを利用します（参考：https://istio.io/docs/tasks/telemetry/metrics-logs.html）。
$ istioctl create -f demo/new_telemetry.yaml
Created config metric/istio-system/doublerequestcount at revision 7063
Created config prometheus/istio-system/doublehandler at revision 7064
Created config rule/istio-system/doubleprom at revision 7065
Created config logentry/istio-system/newlog at revision 7066
Created config stdio/istio-system/newhandler at revision 7067
Created config rule/istio-system/newlogstdio at revision 7068
new_telemetry.yamlファイルの中身は以下のとおりです。
new_telemetry.yaml
# Configuration for metric instances
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
name: doublerequestcount
namespace: istio-system
spec:
value: "2" # count each request twice
dimensions:
source: source.service | "unknown"
destination: destination.service | "unknown"
message: '"twice the fun!"'
monitored_resource_type: '"UNSPECIFIED"'
---
# Configuration for a Prometheus handler
apiVersion: "config.istio.io/v1alpha2"
kind: prometheus
metadata:
name: doublehandler
namespace: istio-system
spec:
metrics:
- name: double_request_count # Prometheus metric name
instance_name: doublerequestcount.metric.istio-system # Mixer instance name (fully-qualified)
kind: COUNTER
label_names:
- source
- destination
- message
---
# Rule to send metric instances to a Prometheus handler
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: doubleprom
namespace: istio-system
spec:
actions:
- handler: doublehandler.prometheus
instances:
- doublerequestcount.metric
---
# Configuration for logentry instances
apiVersion: "config.istio.io/v1alpha2"
kind: logentry
metadata:
name: newlog
namespace: istio-system
spec:
severity: '"warning"'
timestamp: request.time
variables:
source: source.labels["app"] | source.service | "unknown"
user: source.user | "unknown"
destination: destination.labels["app"] | destination.service | "unknown"
responseCode: response.code | 0
responseSize: response.size | 0
latency: response.duration | "0ms"
monitored_resource_type: '"UNSPECIFIED"'
---
# Configuration for a stdio handler
apiVersion: "config.istio.io/v1alpha2"
kind: stdio
metadata:
name: newhandler
namespace: istio-system
spec:
severity_levels:
warning: 1 # Params.Level.WARNING
outputAsJson: true
---
# Rule to send logentry instances to a stdio handler
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: newlogstdio
namespace: istio-system
spec:
match: "true" # match for all requests
actions:
- handler: newhandler.stdio
instances:
- newlog.logentry
---
forループでサンプルアプリBookInfo (http://169.56.28.18/productpage)にトラフィックを送信します。
$ for i in {1..5}; do echo -n .; curl -s http://${GATEWAY_URL}/productpage > /dev/null; done
Grafanaダッシュボードに再度アクセスして、新しいメトリックが収集されていることを確認します。
ログ・ストリームが作成されていることと、リクエストが生成されていることが確認できます。
$ kubectl -n istio-system logs $(kubectl -n istio-system get pods -l istio=mixer -o jsonpath='{.items[0].metadata.name}') mixer | grep \"instance\":\"newlog.logentry.istio-system\"
{"level":"warn","time":"2018-05-18T08:03:45.072460Z","instance":"newlog.logentry.istio-system","destination":"reviews","latency":"8.812174ms","responseCode":200,"responseSize":379,"source":"productpage","user":"unknown"}
{"level":"warn","time":"2018-05-18T08:03:45.163212Z","instance":"newlog.logentry.istio-system","destination":"reviews","latency":"18.7925ms","responseCode":200,"responseSize":375,"source":"productpage","user":"unknown"}
{"level":"warn","time":"2018-05-18T08:03:45.151312Z","instance":"newlog.logentry.istio-system","destination":"istio-ingress.istio-system.svc.cluster.local","latency":"34.802885ms","responseCode":200,"responseSize":5719,"source":"unknown","user":"unknown"}
Zipkinによる分散トレース
Zipkinをインストールします。
$ kubectl apply -f istio/install/kubernetes/addons/zipkin.yaml
deployment "zipkin" created
service "zipkin" created
Zipkinダッシュボードをポート・フォワーディングします。
$ kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=zipkin -o jsonpath='{.items[0].metadata.name}') 9411:9411
forループでサンプルアプリBookInfoにアクセスし、ブラウザでhttp://localhost:9411/zipkin/を確認します。BookInfoに送信したトラフィックの詳細が表示されます。これは、ここでは、http://169.56.28.18/productpageへのリクエストに要した時間を表示させています。
以上