LoginSignup
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

@testnin2(Shinobu Yasuda)

Red Hat OpenShift on IBM Cloud(VPC):Service(type:LoadBalancer)で private/zoneアノテーションを試してみる

Last updated at Posted at 2020-07-04

はじめに

Red Hat OpenShift on IBM CloudのVPC環境にて、type:LoadBalancerのサービスを作成する際のアノテーションを試してみた。利用環境としては、マルチゾーンで構成されており、それぞれのゾーンごとに2台ずつのWorker nodeが配置されています。

ノード
$ ibmcloud oc worker ls --cluster myroksclustervpc --show-pools
OK
ID                                                       Primary IP     Flavor     State    Status   Zone         Version                 Worker Pool
kube-bru1t0nd075uqsfusee0-myroksclust-default-00000147   10.240.0.5     bx2.4x16   normal   Ready    us-south-1   4.3.25_1527_openshift   default
kube-bru1t0nd075uqsfusee0-myroksclust-default-0000026a   10.240.0.4     bx2.4x16   normal   Ready    us-south-1   4.3.25_1527_openshift   default
kube-bru1t0nd075uqsfusee0-myroksclust-default-00000399   10.240.128.5   bx2.4x16   normal   Ready    us-south-3   4.3.25_1527_openshift   default
kube-bru1t0nd075uqsfusee0-myroksclust-default-00000410   10.240.128.4   bx2.4x16   normal   Ready    us-south-3   4.3.25_1527_openshift   default
kube-bru1t0nd075uqsfusee0-myroksclust-default-00000586   10.240.64.5    bx2.4x16   normal   Ready    us-south-2   4.3.25_1527_openshift   default
kube-bru1t0nd075uqsfusee0-myroksclust-default-000006a5   10.240.64.4    bx2.4x16   normal   Ready    us-south-2   4.3.25_1527_openshift   default

※2020/07/04現在、service.kubernetes.io/ibm-load-balancer-cloud-provider-enable-features: "proxy-protocol"は利用できません。これは現在開発中の機能であり、ドキュメントのミスのようです。

Private指定

  • アノテーションを指定しない場合はデフォルトでPublic NW側に公開される。
  • service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: <public_or_private>はPublic NW側に公開するLoadBalancerなのか、Private NW側に公開するLoadBalancerなのかを指定できる。今回は明示的にPrivate指定をしてみる。
hello-world-private.yaml
apiVersion: v1
kind: Service
metadata:
 name: hello-world-private
 namespace: syasuda
 annotations:
   service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: private
spec:
 type: LoadBalancer
 selector:
   deploymentconfig: hello-world
 ports:
  - name: http
    protocol: TCP
    port: 8080
    targetPort: 8080


$ oc apply -f hello-world-private.yaml

$ oc get service hello-world-private
NAME                  TYPE           CLUSTER-IP     EXTERNAL-IP                            PORT(S)          AGE
hello-world-private   LoadBalancer   172.21.34.34   xxxxxxxx-us-south.lb.appdomain.cloud   8080:31711/TCP   6m9s

$ oc describe service hello-world-private
Name:                     hello-world-private
Namespace:                syasuda
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type":"private"}...
                          service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: private
Selector:                 deploymentconfig=hello-world
Type:                     LoadBalancer
IP:                       172.21.34.34
LoadBalancer Ingress:     xxxxxxxx-us-south.lb.appdomain.cloud
Port:                     http  8080/TCP
TargetPort:               8080/TCP
NodePort:                 http  31711/TCP
Endpoints:                172.17.111.9:8080,172.17.115.137:8080,172.17.123.76:8080 + 2 more...
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                           Age                    From                Message
  ----     ------                           ----                   ----                -------
  Warning  CreatingCloudLoadBalancerFailed  4m35s (x5 over 5m48s)  ibm-cloud-provider  Error on cloud load balancer kube-bru1t0nd075uqsfusee0-408ea2409e574f6fbe066a0120aa6efc for service syasuda/hello-world-private with UID 408ea240-9e57-4f6f-be06-6a0120aa6efc: LoadBalancer is busy: offline/create_pending
  Warning  SyncLoadBalancerFailed           4m35s (x5 over 5m48s)  service-controller  Error syncing load balancer: failed to ensure load balancer: Error on cloud load balancer kube-bru1t0nd075uqsfusee0-408ea2409e574f6fbe066a0120aa6efc for service syasuda/hello-world-private with UID 408ea240-9e57-4f6f-be06-6a0120aa6efc: LoadBalancer is busy: offline/create_pending
  Normal   EnsuringLoadBalancer             3m15s (x7 over 6m12s)  service-controller  Ensuring load balancer
  Normal   EnsuredLoadBalancer              3m5s (x2 over 5m57s)   service-controller  Ensured load balancer

確かに割り当てられたこのFQDNを名前解決するとPrivate IPになる。
また、VPC Load Balancerインスタンスは異なるZoneに配置されていることが、そのIPアドレス情報からわかる。

$ dig b3160ca4-us-south.lb.appdomain.cloud
dig A +noall +answer @1.1.1.1 xxxxxxxx-us-south.lb.appdomain.cloud
xxxxxxxx-us-south.lb.appdomain.cloud. 120 IN A	10.240.64.10
xxxxxxxx-us-south.lb.appdomain.cloud. 120 IN A	10.240.128.9
image.png

Private指定 + Zone指定

  • 前節の結果からも分かるように、アノテーションを指定しない場合はデフォルトでVPC Load Balancerは複数Zoneにまたがって配置される。また、このVPC Load Balancerは任意のZoneにあるWorker Nodeに割り振る。
  • service.kubernetes.io/ibm-load-balancer-cloud-provider-zone: "<zone>"を指定することで、特定のZoneにのみVPC Load Balancerを配置することが可能になる。また、このVPC Load Balancerの割り振り先対象もこの指定したWorker Nodeにのみになる。ただし、Worker NodeのNodePortは、他のZoneに存在するPodにも割り振りを行うことができる。
  • IBM Cloud docsより引用
    • The VPC load balancer is deployed to the same subnet in that zone that your worker nodes are connected to.
    • Only worker nodes in your cluster in this zone are configured to receive traffic from the VPC load balancer.
hello-world-us-south-1.yaml
apiVersion: v1
kind: Service
metadata:
 name: hello-world-us-south-1
 namespace: syasuda
 annotations:
   service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: private
   service.kubernetes.io/ibm-load-balancer-cloud-provider-zone: "us-south-1"
spec:
 type: LoadBalancer
 selector:
   deploymentconfig: hello-world
 ports:
  - name: http
    protocol: TCP
    port: 8080
    targetPort: 8080

$ oc apply -f hello-world-us-south-1.yaml

$ oc get service hello-world-us-south-1
NAME                     TYPE           CLUSTER-IP       EXTERNAL-IP                            PORT(S)          AGE
hello-world-us-south-1   LoadBalancer   172.21.131.127   yyyyyyyy-us-south.lb.appdomain.cloud   8080:31333/TCP   6m35s

oc describe service hello-world-us-south-1
Name:                     hello-world-us-south-1
Namespace:                syasuda
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type":"private",...
                          service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: private
                          service.kubernetes.io/ibm-load-balancer-cloud-provider-zone: us-south-1
Selector:                 deploymentconfig=hello-world
Type:                     LoadBalancer
IP:                       172.21.131.127
LoadBalancer Ingress:     yyyyyyyy-us-south.lb.appdomain.cloud
Port:                     http  8080/TCP
TargetPort:               8080/TCP
NodePort:                 http  31333/TCP
Endpoints:                172.17.111.9:8080,172.17.115.137:8080,172.17.123.76:8080 + 2 more...
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                           Age    From                Message
  ----    ------                           ----   ----                -------
  Normal  EnsuringLoadBalancer             7m     service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer              6m49s  service-controller  Ensured load balancer
  Normal  CloudVPCLoadBalancerNormalEvent  51s    ibm-cloud-provider  Event on cloud load balancer hello-world-us-south-1 for service syasuda/hello-world-us-south-1 with UID ea8fd792-5d72-475a-b972-e836654aedc2: The VPC load balancer that routes requests to this Kubernetes LoadBalancer service is currently online/active.

$ oc get pods -o wide|grep -v Completed
NAME                   READY   STATUS      RESTARTS   AGE    IP               NODE           NOMINATED NODE   READINESS GATES
hello-world-1-45m9j    1/1     Running     0          3d2h   172.17.111.9     10.240.128.4   <none>           <none>
hello-world-1-b75zr    1/1     Running     0          3d2h   172.17.115.137   10.240.128.5   <none>           <none>
hello-world-1-gl5sd    1/1     Running     0          3d2h   172.17.74.18     10.240.64.5    <none>           <none>
hello-world-1-j8w4f    1/1     Running     0          3d2h   172.17.67.14     10.240.64.4    <none>           <none>
hello-world-1-rb8nt    1/1     Running     0          3d2h   172.17.123.76    10.240.0.4     <none>           <none>

上記のEndpoints情報の結果より、別Zoneに存在する複数のPodにも割り振りは行われている。
Private指定のアノテーションを付けているので、確かに割り当てられたこのFQDNを名前解決するとPrivate IPになっているが、その一方でその連続したIPアドレスからVPC Load Balancerインスタンスは同一Zoneに配置されていることが分かる。

$ dig A +noall +answer @1.1.1.1 yyyyyyyy-us-south.lb.appdomain.cloud
yyyyyyyy-us-south.lb.appdomain.cloud. 120 IN A	10.240.0.8
yyyyyyyy-us-south.lb.appdomain.cloud. 120 IN A	10.240.0.9

また、以下のように全6台（各Zoneに2台ずつ）Worker Nodeがあるにも関わらず、2台しか割り振り先対象になっていない。
image.png

0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?