背景

Kubernetes Clusterが外部ネットワークに出れない環境において
istioをインストールする必要があったため、今回その手順を共有できればと思います。

imageの準備

必要なimageをRegistoryにアップロードします。

今回、自分はHarborというものを使用しているため、その手順で進めていきます。

先に以下の準備をする。

# docker save [image名: tag] > [ファイル名].tar

対象image

docker.io/istio/operator
docker.io/istio/proxyv2:1.5.0
docker.io/istio/pilot:1.5.0
docker.io/prom/prometheus:v2.15.1
docker.io/jaegertracing/all-in-one:1.16
quay.io/kiali/kiali

上記のものをdocker pushできる環境に持っていき、以下を実行します。

# docker load < [ファイル名].tar

次にtag付けし、RegistoryにPushします。

# docker tag ce6f52800e2a harbor.pks.local/istio/operator:1.5.0
# docker tag 1e8369af71b4 harbor.pks.local/istio/proxyv2:1.5.0
# docker tag 105e95c7f9f1 harbor.pks.local/istio/pilot:1.5.0
# docker tag 30572ec34ec8 harbor.pks.local/kiali/kiali:v1.14
# docker tag 61bf337f2956 harbor.pks.local/prom/prometheus:v2.15.1
# docker tag fea586ade9d0 harbor.pks.local/jaegertracing/all-in-one:1.16

# docker push harbor.pks.local/istio/operator
# docker push harbor.pks.local/istio/proxyv2:1.5.0
# docker push harbor.pks.local/istio/pilot:1.5.0
# docker push harbor.pks.local/kiali/kiali:v1.14
# docker push harbor.pks.local/prom/prometheus:v2.15.1
# docker push harbor.pks.local/jaegertracing/all-in-one:1.16

istioctl のインストール

以下より、tarファイルを落としてきます。
https://github.com/istio/istio/releases/tag/1.5.0

このファイルをアップロードし、以下を実行します。

# tar xvf istioctl-1.5.0-linux.tar.gz
istioctl

# mv istioctl /usr/local/bin/

istio operatorのインストール

istioctl operator initを用いて、istio operatorのmanifestを適用すします。

# istioctl operator init
Using operator Deployment image: docker.io/istio/operator:1.5.0

- Applying manifest for component Operator...
✔ Finished applying manifest for component Operator.
Component Operator installed successfully.

*** Success. ***

状態の確認

# kubectl get all -n istio-operator
NAME                                 READY   STATUS         RESTARTS   AGE
pod/istio-operator-5bf589cd8-s46kn   0/1     ErrImagePull   0          23s


NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/istio-operator   ClusterIP   10.100.200.12   <none>        8383/TCP   23s


NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/istio-operator   0/1     1            0           23s

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/istio-operator-5bf589cd8   1         1         0       23s

manifestを書き換えていきます。

# kubectl edit deploy istio-operator

(変更前)
image: docker.io/istio/operator

(変更後)
image: harbor.pks.local/istio/operator

再度確認します。

# kubectl get all -n istio-operator
NAME                                 READY   STATUS    RESTARTS   AGE
pod/istio-operator-5bf589cd8-s46kn   1/1     Running   0          23s


NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/istio-operator   ClusterIP   10.100.200.12   <none>        8383/TCP   23s


NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/istio-operator   1/1     1            1           23s

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/istio-operator-5bf589cd8   1         1         1       23s

istioをインストール

まず、Namespaceを作成します。

# kubectl create ns istio-system
namespace/istio-system created

次に以下を適用します。

# kubectl apply -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  profile: default
  hub: harbor.pks.local/istio
  addonComponents:
    kiali:
      enabled: true
      k8s:
        replicaCount: 1
    prometheus:
      enabled: true
      k8s:
        replicaCount: 1
    tracing:
      enabled: true
EOF
istiooperator.install.istio.io/example-istiocontrolplane created

ここのhubは、今後isio proxyを入れる時にimageを引っ張ってくる参照先となるので、今回のlocal private Registryを登録してください。

状態を確認していきます。

# kubectl get all -n istio-system
NAME                                        READY   STATUS         RESTARTS   AGE
pod/istio-ingressgateway-757f454bff-7x84z   0/1     ErrImagePull   0          29s
pod/istio-tracing-c7b59f68f-f2846           0/1     ErrImagePull   0          30s
pod/istiod-b56f454c6-k2ph4                  0/1     ErrImagePull   0          28s
pod/kiali-57bd8ccd7-nl87l                   0/1     ErrImagePull   0          65s
pod/prometheus-78f785fc6b-k66v7             0/2     ErrImagePull   0          31s


NAME                                TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                                                                      AGE
service/istio-ingressgateway        LoadBalancer   10.100.200.92    <pending>     15020:30296/TCP,80:30514/TCP,443:31564/TCP,15029:32485/TCP,15030:31468/TCP,15031:31656/TCP,15032:30444/TCP,15443:31122/TCP   31s
service/istio-pilot                 ClusterIP      10.100.200.167   <none>        15010/TCP,15011/TCP,15012/TCP,8080/TCP,15014/TCP,443/TCP                                                                     29s
service/istiod                      ClusterIP      10.100.200.35    <none>        15012/TCP,443/TCP                                                                                                            29s
service/jaeger-agent                ClusterIP      None             <none>        5775/UDP,6831/UDP,6832/UDP                                                                                                   29s
service/jaeger-collector            ClusterIP      10.100.200.190   <none>        14267/TCP,14268/TCP,14250/TCP                                                                                                30s
service/jaeger-collector-headless   ClusterIP      None             <none>        14250/TCP                                                                                                                    30s
service/jaeger-query                ClusterIP      10.100.200.166   <none>        16686/TCP                                                                                                                    30s
service/kiali                       ClusterIP      10.100.200.90    <none>        20001/TCP                                                                                                                    48s
service/prometheus                  ClusterIP      10.100.200.243   <none>        9090/TCP                                                                                                                     31s
service/tracing                     ClusterIP      10.100.200.196   <none>        80/TCP                                                                                                                       29s
service/zipkin                      ClusterIP      10.100.200.41    <none>        9411/TCP                                                                                                                     29s


NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/istio-ingressgateway   0/1     1            0           32s
deployment.apps/istio-tracing          0/1     1            0           30s
deployment.apps/istiod                 0/1     1            0           30s
deployment.apps/kiali                  0/1     1            0           48s
deployment.apps/prometheus             0/1     1            0           31s

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/istio-ingressgateway-757f454bff   1         1         0       32s
replicaset.apps/istio-tracing-c7b59f68f           1         1         0       30s
replicaset.apps/istiod-b56f454c6                  1         1         0       30s
replicaset.apps/kiali-57bd8ccd7                   1         1         0       66s
replicaset.apps/prometheus-78f785fc6b             1         1         0       31s


NAME                                                       REFERENCE                         TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/istio-ingressgateway   Deployment/istio-ingressgateway   <unknown>/80%    1         5         0          32s
horizontalpodautoscaler.autoscaling/istiod                 Deployment/istiod                 <unknown>/80%    1         5         0          32s

同様に、imageの部分を書き換えていきます。

# kubectl edit deploy istio-ingressgateway

(変更前)
image: docker.io/istio/proxyv2:1.5.0

(変更後)
image: harbor.pks.local/istio/proxyv2:1.5.0



# kubectl edit deploy istio-tracing

(変更前)
image: docker.io/jaegertracing/all-in-one:1.16

(変更後)
image: harbor.pks.local/jaegertracing/all-in-one:1.16



# kubectl edit deploy istiod

(変更前)
image: docker.io/istio/pilot:1.5.0

(変更後)
image: harbor.pks.local/istio/pilot:1.5.0



# kubectl edit deploy kiali

(変更前)
 image: quay.io/kiali/kiali:v1.14

(変更後)
 image: harbor.pks.local/kiali/kiali:v1.14



# kubectl edit deploy prometheus

(変更前)
image: docker.io/prom/prometheus:v2.15.1
image: docker.io/istio/proxyv2:1.5.0

(変更後)
image: harbor.pks.local/prom/prometheus:v2.15.1
image: harbor.pks.local/istio/proxyv2:1.5.0

再度確認をします。

# kubectl get all -n istio-system
NAME                                        READY   STATUS    RESTARTS   AGE
pod/istio-ingressgateway-6b86497c9b-zpmzm   1/1     Running   0          45m
pod/istio-tracing-8456948944-slnkt          1/1     Running   0          44m
pod/istiod-7bcf4b8c95-xdhgc                 1/1     Running   0          43m
pod/kiali-57bd8ccd7-ghvj2                   1/1     Running   0          42m
pod/prometheus-7f4b6d8c64-lnkjc             2/2     Running   0          40m


NAME                                TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                                                                      AGE
service/istio-ingressgateway        LoadBalancer   10.100.200.92    <pending>     15020:31399/TCP,80:30800/TCP,443:31880/TCP,15029:30769/TCP,15030:31945/TCP,15031:32435/TCP,15032:31443/TCP,15443:30316/TCP   46m
service/istio-pilot                 ClusterIP      10.100.200.167   <none>        15010/TCP,15011/TCP,15012/TCP,8080/TCP,15014/TCP,443/TCP                                                                     46m
service/istiod                      ClusterIP      10.100.200.35    <none>        15012/TCP,443/TCP                                                                                                            46m
service/jaeger-agent                ClusterIP      None             <none>        5775/UDP,6831/UDP,6832/UDP                                                                                                   46m
service/jaeger-collector            ClusterIP      10.100.200.190   <none>        14267/TCP,14268/TCP,14250/TCP                                                                                                46m
service/jaeger-collector-headless   ClusterIP      None             <none>        14250/TCP                                                                                                                    46m
service/jaeger-query                ClusterIP      10.100.200.166   <none>        16686/TCP                                                                                                                    46m
service/kiali                       ClusterIP      10.100.200.90    <none>        20001/TCP                                                                                                                    46m
service/prometheus                  ClusterIP      10.100.200.243   <none>        9090/TCP                                                                                                                     46m
service/tracing                     ClusterIP      10.100.200.196   <none>        80/TCP                                                                                                                       46m
service/zipkin                      ClusterIP      10.100.200.41    <none>        9411/TCP                                                                                                                     46m


NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/istio-ingressgateway   1/1     1            1           46m
deployment.apps/istio-tracing          1/1     1            1           46m
deployment.apps/istiod                 1/1     1            1           46m
deployment.apps/kiali                  1/1     1            1           46m
deployment.apps/prometheus             1/1     1            1           46m

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/istio-ingressgateway-6b86497c9b   1         1         1       45m
replicaset.apps/istio-ingressgateway-757f454bff   0         0         0       46m
replicaset.apps/istio-tracing-8456948944          1         1         1       44m
replicaset.apps/istio-tracing-c7b59f68f           0         0         0       46m
replicaset.apps/istiod-7bcf4b8c95                 1         1         1       43m
replicaset.apps/istiod-b56f454c6                  0         0         0       46m
replicaset.apps/kiali-57bd8ccd7                   1         1         1       42m
replicaset.apps/kiali-57fb5bb5c6                  0         0         0       46m
replicaset.apps/prometheus-69465d5bd7             0         0         0       41m
replicaset.apps/prometheus-78f785fc6b             0         0         0       46m
replicaset.apps/prometheus-7f4b6d8c64             1         1         1       40m


NAME                                                       REFERENCE                         TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/istio-ingressgateway   Deployment/istio-ingressgateway   3%/80%    1         5         1          46m
horizontalpodautoscaler.autoscaling/istiod                 Deployment/istiod                 0%/80%    1         5         1          46m

以上で、install完了です。

補足

istioctl manifest generateを使ってyamlファイルを生成した後、
imageの箇所を書き換え、kubectl apply -fで適用したところ以下のような状態となり
Podが上がってきませんでした。

istio-ingressgateway-6bc66454d9-s2ll9

Events:
  Type     Reason       Age                       From                                           Message
  ----     ------       ----                      ----                                           -------
  Normal   Scheduled    29m                       default-scheduler                              Successfully assigned istio-system/istio-ingressgateway-6bc66454d9-s2ll9 to 6936cb3e-4414-4e9c-a22f-d9b335886036
  Warning  FailedMount  23m (x11 over 29m)        kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036  MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the server could not find the requested resource
  Warning  FailedMount  9m28s (x9 over 27m)       kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036  Unable to mount volumes for pod "istio-ingressgateway-6bc66454d9-s2ll9_istio-system(c34e3024-01c6-4c89-aeb0-78d1c1bc387a)": timeout expired waiting for volumes to attach or mount for pod "istio-system"/"istio-ingressgateway-6bc66454d9-s2ll9". list of unmounted volumes=[istiod-ca-cert istio-token]. list of unattached volumes=[istiod-ca-cert podinfo ingressgatewaysdsudspath istio-token ingressgateway-certs ingressgateway-ca-certs istio-ingressgateway-service-account-token-szxww]
  Warning  FailedMount  <invalid> (x23 over 29m)  kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036  MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found


istiod-7d6cbf97c8-gfqvv

Events:
  Type     Reason       Age                       From                                           Message
  ----     ------       ----                      ----                                           -------
  Normal   Scheduled    30m                       default-scheduler                              Successfully assigned istio-system/istiod-7d6cbf97c8-gfqvv to 6936cb3e-4414-4e9c-a22f-d9b335886036
  Warning  FailedMount  7m57s (x10 over 28m)      kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036  Unable to mount volumes for pod "istiod-7d6cbf97c8-gfqvv_istio-system(cc17e972-42bc-4bc3-adb1-880cb03f5056)": timeout expired waiting for volumes to attach or mount for pod "istio-system"/"istiod-7d6cbf97c8-gfqvv". list of unmounted volumes=[istio-token]. list of unattached volumes=[local-certs istio-token istiod cacerts inject config-volume pilot-envoy-config istiod-service-account-token-hmkh6]
  Warning  FailedMount  <invalid> (x23 over 30m)  kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036  MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the server could not find the requested resource


prometheus-7fc945d578-gspnn

Events:
  Type     Reason       Age                 From                                           Message
  ----     ------       ----                ----                                           -------
  Normal   Scheduled    31m                 default-scheduler                              Successfully assigned istio-system/prometheus-7fc945d578-gspnn to 6936cb3e-4414-4e9c-a22f-d9b335886036
  Warning  FailedMount  25m (x11 over 31m)  kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036  MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the server could not find the requested resource
  Warning  FailedMount  11m (x9 over 29m)   kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036  Unable to mount volumes for pod "prometheus-7fc945d578-gspnn_istio-system(c0c9df68-4353-40c9-adad-95742a357599)": timeout expired waiting for volumes to attach or mount for pod "istio-system"/"prometheus-7fc945d578-gspnn". list of unmounted volumes=[istio-token istiod-ca-cert]. list of unattached volumes=[config-volume istio-certs istio-envoy istio-token istiod-ca-cert prometheus-token-6mx7s]
  Warning  FailedMount  67s (x23 over 31m)  kubelet, 6936cb3e-4414-4e9c-a22f-d9b335886036  MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found