背景
Kubernetes Clusterが外部ネットワークに出れない環境において
istioをインストールする必要があったため、今回その手順を共有できればと思います。
imageの準備
必要なimageをRegistoryにアップロードします。
今回、自分はHarborというものを使用しているため、その手順で進めていきます。
先に以下の準備をする。
# docker save [image名: tag] > [ファイル名].tar
対象image
docker.io/istio/operator
docker.io/istio/proxyv2:1.5.0
docker.io/istio/pilot:1.5.0
docker.io/prom/prometheus:v2.15.1
docker.io/jaegertracing/all-in-one:1.16
quay.io/kiali/kiali
上記のものをdocker pushできる環境に持っていき、以下を実行します。
# docker load < [ファイル名].tar
次にtag付けし、RegistoryにPushします。
# docker tag ce6f52800e2a harbor.pks.local/istio/operator:1.5.0
# docker tag 1e8369af71b4 harbor.pks.local/istio/proxyv2:1.5.0
# docker tag 105e95c7f9f1 harbor.pks.local/istio/pilot:1.5.0
# docker tag 30572ec34ec8 harbor.pks.local/kiali/kiali:v1.14
# docker tag 61bf337f2956 harbor.pks.local/prom/prometheus:v2.15.1
# docker tag fea586ade9d0 harbor.pks.local/jaegertracing/all-in-one:1.16
# docker push harbor.pks.local/istio/operator
# docker push harbor.pks.local/istio/proxyv2:1.5.0
# docker push harbor.pks.local/istio/pilot:1.5.0
# docker push harbor.pks.local/kiali/kiali:v1.14
# docker push harbor.pks.local/prom/prometheus:v2.15.1
# docker push harbor.pks.local/jaegertracing/all-in-one:1.16
istioctl のインストール
以下より、tarファイルを落としてきます。
https://github.com/istio/istio/releases/tag/1.5.0
このファイルをアップロードし、以下を実行します。
# tar xvf istioctl-1.5.0-linux.tar.gz
istioctl
# mv istioctl /usr/local/bin/
istio operatorのインストール
istioctl operator initを用いて、istio operatorのmanifestを適用すします。
# istioctl operator init
Using operator Deployment image: docker.io/istio/operator:1.5.0
- Applying manifest for component Operator...
✔ Finished applying manifest for component Operator.
Component Operator installed successfully.
*** Success. ***
状態の確認
# kubectl get all -n istio-operator
NAME READY STATUS RESTARTS AGE
pod/istio-operator-5bf589cd8-s46kn 0/1 ErrImagePull 0 23s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/istio-operator ClusterIP 10.100.200.12 <none> 8383/TCP 23s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/istio-operator 0/1 1 0 23s
NAME DESIRED CURRENT READY AGE
replicaset.apps/istio-operator-5bf589cd8 1 1 0 23s
manifestを書き換えていきます。
# kubectl edit deploy istio-operator
(変更前)
image: docker.io/istio/operator
(変更後)
image: harbor.pks.local/istio/operator
再度確認します。
# kubectl get all -n istio-operator
NAME READY STATUS RESTARTS AGE
pod/istio-operator-5bf589cd8-s46kn 1/1 Running 0 23s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/istio-operator ClusterIP 10.100.200.12 <none> 8383/TCP 23s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/istio-operator 1/1 1 1 23s
NAME DESIRED CURRENT READY AGE
replicaset.apps/istio-operator-5bf589cd8 1 1 1 23s
istioをインストール
まず、Namespaceを作成します。
# kubectl create ns istio-system
namespace/istio-system created
次に以下を適用します。
# kubectl apply -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: example-istiocontrolplane
spec:
profile: default
hub: harbor.pks.local/istio
addonComponents:
kiali:
enabled: true
k8s:
replicaCount: 1
prometheus:
enabled: true
k8s:
replicaCount: 1
tracing:
enabled: true
EOF
istiooperator.install.istio.io/example-istiocontrolplane created
ここのhubは、今後isio proxyを入れる時にimageを引っ張ってくる参照先となるので、今回のlocal private Registryを登録してください。
状態を確認していきます。
# kubectl get all -n istio-system
NAME READY STATUS RESTARTS AGE
pod/istio-ingressgateway-757f454bff-7x84z 0/1 ErrImagePull 0 29s
pod/istio-tracing-c7b59f68f-f2846 0/1 ErrImagePull 0 30s
pod/istiod-b56f454c6-k2ph4 0/1 ErrImagePull 0 28s
pod/kiali-57bd8ccd7-nl87l 0/1 ErrImagePull 0 65s
pod/prometheus-78f785fc6b-k66v7 0/2 ErrImagePull 0 31s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/istio-ingressgateway LoadBalancer 10.100.200.92 <pending> 15020:30296/TCP,80:30514/TCP,443:31564/TCP,15029:32485/TCP,15030:31468/TCP,15031:31656/TCP,15032:30444/TCP,15443:31122/TCP 31s
service/istio-pilot ClusterIP 10.100.200.167 <none> 15010/TCP,15011/TCP,15012/TCP,8080/TCP,15014/TCP,443/TCP 29s
service/istiod ClusterIP 10.100.200.35 <none> 15012/TCP,443/TCP 29s
service/jaeger-agent ClusterIP None <none> 5775/UDP,6831/UDP,6832/UDP 29s
service/jaeger-collector ClusterIP 10.100.200.190 <none> 14267/TCP,14268/TCP,14250/TCP 30s
service/jaeger-collector-headless ClusterIP None <none> 14250/TCP 30s
service/jaeger-query ClusterIP 10.100.200.166 <none> 16686/TCP 30s
service/kiali ClusterIP 10.100.200.90 <none> 20001/TCP 48s
service/prometheus ClusterIP 10.100.200.243 <none> 9090/TCP 31s
service/tracing ClusterIP 10.100.200.196 <none> 80/TCP 29s
service/zipkin ClusterIP 10.100.200.41 <none> 9411/TCP 29s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/istio-ingressgateway 0/1 1 0 32s
deployment.apps/istio-tracing 0/1 1 0 30s
deployment.apps/istiod 0/1 1 0 30s
deployment.apps/kiali 0/1 1 0 48s
deployment.apps/prometheus 0/1 1 0 31s
NAME DESIRED CURRENT READY AGE
replicaset.apps/istio-ingressgateway-757f454bff 1 1 0 32s
replicaset.apps/istio-tracing-c7b59f68f 1 1 0 30s
replicaset.apps/istiod-b56f454c6 1 1 0 30s
replicaset.apps/kiali-57bd8ccd7 1 1 0 66s
replicaset.apps/prometheus-78f785fc6b 1 1 0 31s
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/istio-ingressgateway Deployment/istio-ingressgateway <unknown>/80% 1 5 0 32s
horizontalpodautoscaler.autoscaling/istiod Deployment/istiod <unknown>/80% 1 5 0 32s
同様に、imageの部分を書き換えていきます。
# kubectl edit deploy istio-ingressgateway
(変更前)
image: docker.io/istio/proxyv2:1.5.0
(変更後)
image: harbor.pks.local/istio/proxyv2:1.5.0
# kubectl edit deploy istio-tracing
(変更前)
image: docker.io/jaegertracing/all-in-one:1.16
(変更後)
image: harbor.pks.local/jaegertracing/all-in-one:1.16
# kubectl edit deploy istiod
(変更前)
image: docker.io/istio/pilot:1.5.0
(変更後)
image: harbor.pks.local/istio/pilot:1.5.0
# kubectl edit deploy kiali
(変更前)
image: quay.io/kiali/kiali:v1.14
(変更後)
image: harbor.pks.local/kiali/kiali:v1.14
# kubectl edit deploy prometheus
(変更前)
image: docker.io/prom/prometheus:v2.15.1
image: docker.io/istio/proxyv2:1.5.0
(変更後)
image: harbor.pks.local/prom/prometheus:v2.15.1
image: harbor.pks.local/istio/proxyv2:1.5.0
再度確認をします。
# kubectl get all -n istio-system
NAME READY STATUS RESTARTS AGE
pod/istio-ingressgateway-6b86497c9b-zpmzm 1/1 Running 0 45m
pod/istio-tracing-8456948944-slnkt 1/1 Running 0 44m
pod/istiod-7bcf4b8c95-xdhgc 1/1 Running 0 43m
pod/kiali-57bd8ccd7-ghvj2 1/1 Running 0 42m
pod/prometheus-7f4b6d8c64-lnkjc 2/2 Running 0 40m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/istio-ingressgateway LoadBalancer 10.100.200.92 <pending> 15020:31399/TCP,80:30800/TCP,443:31880/TCP,15029:30769/TCP,15030:31945/TCP,15031:32435/TCP,15032:31443/TCP,15443:30316/TCP 46m
service/istio-pilot ClusterIP 10.100.200.167 <none> 15010/TCP,15011/TCP,15012/TCP,8080/TCP,15014/TCP,443/TCP 46m
service/istiod ClusterIP 10.100.200.35 <none> 15012/TCP,443/TCP 46m
service/jaeger-agent ClusterIP None <none> 5775/UDP,6831/UDP,6832/UDP 46m
service/jaeger-collector ClusterIP 10.100.200.190 <none> 14267/TCP,14268/TCP,14250/TCP 46m
service/jaeger-collector-headless ClusterIP None <none> 14250/TCP 46m
service/jaeger-query ClusterIP 10.100.200.166 <none> 16686/TCP 46m
service/kiali ClusterIP 10.100.200.90 <none> 20001/TCP 46m
service/prometheus ClusterIP 10.100.200.243 <none> 9090/TCP 46m
service/tracing ClusterIP 10.100.200.196 <none> 80/TCP 46m
service/zipkin ClusterIP 10.100.200.41 <none> 9411/TCP 46m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/istio-ingressgateway 1/1 1 1 46m
deployment.apps/istio-tracing 1/1 1 1 46m
deployment.apps/istiod 1/1 1 1 46m
deployment.apps/kiali 1/1 1 1 46m
deployment.apps/prometheus 1/1 1 1 46m
NAME DESIRED CURRENT READY AGE
replicaset.apps/istio-ingressgateway-6b86497c9b 1 1 1 45m
replicaset.apps/istio-ingressgateway-757f454bff 0 0 0 46m
replicaset.apps/istio-tracing-8456948944 1 1 1 44m
replicaset.apps/istio-tracing-c7b59f68f 0 0 0 46m
replicaset.apps/istiod-7bcf4b8c95 1 1 1 43m
replicaset.apps/istiod-b56f454c6 0 0 0 46m
replicaset.apps/kiali-57bd8ccd7 1 1 1 42m
replicaset.apps/kiali-57fb5bb5c6 0 0 0 46m
replicaset.apps/prometheus-69465d5bd7 0 0 0 41m
replicaset.apps/prometheus-78f785fc6b 0 0 0 46m
replicaset.apps/prometheus-7f4b6d8c64 1 1 1 40m
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/istio-ingressgateway Deployment/istio-ingressgateway 3%/80% 1 5 1 46m
horizontalpodautoscaler.autoscaling/istiod Deployment/istiod 0%/80% 1 5 1 46m
以上で、install完了です。
補足
istioctl manifest generateを使ってyamlファイルを生成した後、
imageの箇所を書き換え、kubectl apply -fで適用したところ以下のような状態となり
Podが上がってきませんでした。
istio-ingressgateway-6bc66454d9-s2ll9
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 29m default-scheduler Successfully assigned istio-system/istio-ingressgateway-6bc66454d9-s2ll9 to 6936cb3e-4414-4e9c-a22f-d9b335886036
Warning FailedMount 23m (x11 over 29m) kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036 MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the server could not find the requested resource
Warning FailedMount 9m28s (x9 over 27m) kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036 Unable to mount volumes for pod "istio-ingressgateway-6bc66454d9-s2ll9_istio-system(c34e3024-01c6-4c89-aeb0-78d1c1bc387a)": timeout expired waiting for volumes to attach or mount for pod "istio-system"/"istio-ingressgateway-6bc66454d9-s2ll9". list of unmounted volumes=[istiod-ca-cert istio-token]. list of unattached volumes=[istiod-ca-cert podinfo ingressgatewaysdsudspath istio-token ingressgateway-certs ingressgateway-ca-certs istio-ingressgateway-service-account-token-szxww]
Warning FailedMount <invalid> (x23 over 29m) kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036 MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found
istiod-7d6cbf97c8-gfqvv
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 30m default-scheduler Successfully assigned istio-system/istiod-7d6cbf97c8-gfqvv to 6936cb3e-4414-4e9c-a22f-d9b335886036
Warning FailedMount 7m57s (x10 over 28m) kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036 Unable to mount volumes for pod "istiod-7d6cbf97c8-gfqvv_istio-system(cc17e972-42bc-4bc3-adb1-880cb03f5056)": timeout expired waiting for volumes to attach or mount for pod "istio-system"/"istiod-7d6cbf97c8-gfqvv". list of unmounted volumes=[istio-token]. list of unattached volumes=[local-certs istio-token istiod cacerts inject config-volume pilot-envoy-config istiod-service-account-token-hmkh6]
Warning FailedMount <invalid> (x23 over 30m) kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036 MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the server could not find the requested resource
prometheus-7fc945d578-gspnn
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 31m default-scheduler Successfully assigned istio-system/prometheus-7fc945d578-gspnn to 6936cb3e-4414-4e9c-a22f-d9b335886036
Warning FailedMount 25m (x11 over 31m) kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036 MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the server could not find the requested resource
Warning FailedMount 11m (x9 over 29m) kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036 Unable to mount volumes for pod "prometheus-7fc945d578-gspnn_istio-system(c0c9df68-4353-40c9-adad-95742a357599)": timeout expired waiting for volumes to attach or mount for pod "istio-system"/"prometheus-7fc945d578-gspnn". list of unmounted volumes=[istio-token istiod-ca-cert]. list of unattached volumes=[config-volume istio-certs istio-envoy istio-token istiod-ca-cert prometheus-token-6mx7s]
Warning FailedMount 67s (x23 over 31m) kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036 MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found