Go to Qiita Advent Calendar 2024 Top
LoginSignup
46
25

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

@murata-tomohide(村田 智英)in株式会社セレス

Ingressでクロスネームスペースでのロードバランシング

Last updated at Posted at 2019-06-07

TL;DR

複数のnamespaceのpodsたちにむけてを1つのロードバランサで振り分けるのはめんどくさいって話です。
ExternalNameを使えばできる。

ことの発端

そんなにトラフィックはないけど機能ごとにnamespaceを分けておこうかなと思って、バッチやAPPのPodsやServiceのnamespaceを分けてデプロイしようかなと思った。
いざ、Ingressの設定をしようと思ったら、backendにnamespace書くところないじゃんとなった。

このspec.rules.http.pathごとにnamespaceを切っていた。


apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: lb
  annotations:
    #nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS"
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"
spec:
  rules:
    - host: api.192.168.99.107.nip.io
      http:
        paths:
          - path: /dapps
            backend:
              serviceName: dapps-svc
              servicePort: 80
          - path: /gas
            backend:
              serviceName: gas-svc
              servicePort: 80
          - path: /miner
            backend:
              serviceName: miner-svc
              servicePort: 80

もしかしてkubens的な感じで書けばいけるのかなと思ったけど、serviceNameに「.」を入れるとエラーを吐いてくれてデプロイすらできない。（dapps-svc.namespace1とかってしても無駄。）

* spec.rules[0].http.backend.serviceName: Invalid value: "dapps-svc.namespace1": a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name',  or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?')

ググるマン

とりあえず公式のドキュメントを見ると、
https://kubernetes.io/docs/concepts/services-networking/ingress/
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/#ingress-v1beta1-networking-k8s-io

その辺の記載はない。
とりあえず今回は切ったnamespaceを全部Ingress側に寄せて(default)デプロイして回避することにした。

そんな馬鹿な納得できない。

ちゃんと検証してみようと思ってこんな感じに環境を構築した。http HeaderのServerとかで振り分け確認をすればいいや。
（minikubeでやりました。）

image.png

conf

とりあえず全部同一namespaceで動くものを。


# namespace
apiVersion: v1
kind: Namespace
metadata:
  name: ns-test
---
# ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: lb
  namespace: ns-test
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
    - host: ns-test.192.168.99.107.nip.io
      http:
        paths:
          - path: /apache
            backend:
              serviceName: apache-svc
              servicePort: 80
          - path: /nginx
            backend:
              serviceName: nginx-svc
              servicePort: 80
          - path: /
            backend:
              serviceName: blackhole
              servicePort: 80
---
# apache
apiVersion: v1
kind: Service
metadata:
  name: apache-svc
  namespace: ns-test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: httpd
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd
  namespace: ns-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpd
  template:
    metadata:
      labels:
        app: httpd
    spec:
      containers:
      - image: httpd:alpine
        name: httpd
        ports:
        - containerPort: 80
---
# nginx
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  namespace: ns-test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: ns-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:alpine
        name: nginx
        ports:
        - containerPort: 80

デフォルトnamespaceを切り替えて

❯ kubens ns-test
Context "vm" modified.
Active namespace is "ns-test".

こんな感じ

❯ kubectl get all,ingress -o wide
NAME                         READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
pod/httpd-55c79f4cb-zrzdc    1/1     Running   0          29s   172.17.0.2   minikube   <none>           <none>
pod/nginx-66d89c74cb-l7gdv   1/1     Running   0          29s   172.17.0.3   minikube   <none>           <none>

NAME                 TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/apache-svc   NodePort   10.101.246.95   <none>        80:32141/TCP   29s   app=httpd
service/nginx-svc    NodePort   10.97.137.34    <none>        80:30783/TCP   29s   app=nginx

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES         SELECTOR
deployment.apps/httpd   1/1     1            1           29s   httpd        httpd:alpine   app=httpd
deployment.apps/nginx   1/1     1            1           29s   nginx        nginx:alpine   app=nginx

NAME                               DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES         SELECTOR
replicaset.apps/httpd-55c79f4cb    1         1         1       29s   httpd        httpd:alpine   app=httpd,pod-template-hash=55c79f4cb
replicaset.apps/nginx-66d89c74cb   1         1         1       29s   nginx        nginx:alpine   app=nginx,pod-template-hash=66d89c74cb

NAME                    HOSTS                           ADDRESS   PORTS   AGE
ingress.extensions/lb   ns-test.192.168.99.107.nip.io             80      29s

動いてるかな？

❯ curl http://ns-test.192.168.99.107.nip.io/apache
<html><body><h1>It works!</h1></body></html>

~
❯ curl http://ns-test.192.168.99.107.nip.io/nginx
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

問題なさそう。本当はヘッダーで判別したかったけど、Ingressのnginxで上書きされてしまっていたのでbodyで。

namespace変えてみる。

podsのapacheのnamespaceをns-test-podsとしてみよう。
IngressのTargetは変えていないので404になるはず。
kubectl delete all --all とかで一回きれいにしてからやらないと前のnamespaceのpodsが残るよ。

namespace追加とServiceとPod変更


6a7,12
> # namespace
> apiVersion: v1
> kind: Namespace
> metadata:
>   name: ns-test-pods
> ---
39c45
<   namespace: ns-test
---
>   namespace: ns-test-pods
53c59
<   namespace: ns-test
---
>   namespace: ns-test-pods

❯ curl http://ns-test.192.168.99.107.nip.io/nginx
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {

~snip;

❯ curl http://ns-test.192.168.99.107.nip.io/apache
<html>
<head><title>503 Service Temporarily Unavailable</title></head>

~snip;

❯ curl http://ns-test.192.168.99.107.nip.io/apache -I
HTTP/1.1 503 Service Temporarily Unavailable
Server: nginx/1.15.9
Date: Fri, 07 Jun 2019 04:18:20 GMT
Content-Type: text/html
Content-Length: 197
Connection: keep-alive

503になった、リバプロ(Ingress)が吐いてるから正しいね。

serviceName変えてみる。

29c29
<               serviceName: apache-svc
---
>               serviceName: apache-svc.ns-test-pods

違うだろボケナス！って言われる。

The Ingress "lb" is invalid: spec.rules[0].http.backend.serviceName: Invalid value: "apache-svc.ns-test-pods": a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic c
haracter, and end with an alphanumeric character (e.g. 'my-name',  or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?')

ExternalNameの登場

ExternalNameっていうtypeのServiceがあって、別名をつけることができる。

そもそも、k8sのpodsやらserviceやらは名前がついていて、<名前>.<ネームスペース>.svc.cluster.local で名前解決できるようになっている。
ここで別のnamespaceのapache-svcをこのServiceで紐づけてやる。

わかりやすい？ように、Ingress側のターゲットを serviceName: apache-svc-koitu として下のServiceを挟む。

apiVersion: v1
kind: Service
metadata:
  name: apache-svc-koitu
  namespace: ns-test
spec:
  type: ExternalName
  externalName: apache-svc.ns-test-pods.svc.cluster.local

こうすると、

❯ curl http://ns-test.192.168.99.107.nip.io/apache
<html><body><h1>It works!</h1></body></html>

ちゃんとアクセスできたまん。

最終的にこんな感じになった。

# namespace
apiVersion: v1
kind: Namespace
metadata:
  name: ns-test
---
# namespace
apiVersion: v1
kind: Namespace
metadata:
  name: ns-test-pods
---
# ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: lb
  namespace: ns-test
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
    - host: ns-test.192.168.99.107.nip.io
      http:
        paths:
          - path: /apache
            backend:
              serviceName: apache-svc-koitu
              servicePort: 80
          - path: /nginx
            backend:
              serviceName: nginx-svc
              servicePort: 80
          - path: /
            backend:
              serviceName: blackhole
              servicePort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: apache-svc-koitu
  namespace: ns-test
spec:
  type: ExternalName
  externalName: apache-svc.ns-test-pods.svc.cluster.local
---
# apache
apiVersion: v1
kind: Service
metadata:
  name: apache-svc
  namespace: ns-test-pods
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: httpd
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd
  namespace: ns-test-pods
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpd
  template:
    metadata:
      labels:
        app: httpd
    spec:
      containers:
      - image: httpd:alpine
        name: httpd
        ports:
        - containerPort: 80
---
# nginx
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  namespace: ns-test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: ns-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:alpine
        name: nginx
        ports:
        - containerPort: 80

こんなイメージ

image.png

目的は果たせた。

まとめ

ALBを1個増やすとさんぜんえんだからね！ExternalName使っていって良いと思います。
ただし、namespaceを分けすぎるのは良くない。kubectl get xxxで見るときにツライことになる。
本当はIngressまで分けたほうがいいけど、そこまでしなくても・・・みたいなときに使うようにしたいと思います。

参考: )
https://github.com/kubernetes/ingress-nginx/issues/2971
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
https://qiita.com/ryicoh/items/24e5cf4ffdb6cc8dbbdf （この記事そのまんまであった）

追記

EKSでは試せていませんが、オンプレのv1.19.2で↑のマニフェストのままでExternalNameの動作が確認できました。

murata:~ $ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

下記のような構成になりました。

murata:~ $ kubectl get pod,svc,ing -A
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAMESPACE        NAME                                                    READY   STATUS      RESTARTS   AGE
ingress-nginx    pod/ingress-nginx-admission-create-jkrr6                0/1     Completed   0          9m6s
ingress-nginx    pod/ingress-nginx-admission-patch-hr6z4                 0/1     Completed   1          9m6s
ingress-nginx    pod/ingress-nginx-controller-59859f77c7-26tfl           1/1     Running     0          9m6s
kube-system      pod/calico-kube-controllers-c9784d67d-cztwl             1/1     Running     0          19m
kube-system      pod/canal-788gd                                         2/2     Running     0          19m
kube-system      pod/coredns-f9fd979d6-4zzmr                             1/1     Running     0          19m
kube-system      pod/coredns-f9fd979d6-sd68h                             1/1     Running     0          19m
kube-system      pod/etcd-k8s-tmp.dev.deroris.local                      1/1     Running     0          19m
kube-system      pod/kube-apiserver-k8s-tmp.dev.deroris.local            1/1     Running     0          19m
kube-system      pod/kube-controller-manager-k8s-tmp.dev.deroris.local   1/1     Running     0          19m
kube-system      pod/kube-proxy-r7qtv                                    1/1     Running     0          19m
kube-system      pod/kube-scheduler-k8s-tmp.dev.deroris.local            1/1     Running     0          19m
metallb-system   pod/controller-fb659dc8-dbxrw                           1/1     Running     0          18m
metallb-system   pod/speaker-shz4n                                       1/1     Running     0          18m
ns-test-pods     pod/httpd-6cd65b68c6-7phmm                              1/1     Running     0          9m37s
ns-test          pod/nginx-7fb7fd49b4-c5df7                              1/1     Running     0          9m37s

NAMESPACE       NAME                                         TYPE           CLUSTER-IP       EXTERNAL-IP                                 PORT(S)                      AGE
default         service/kubernetes                           ClusterIP      10.96.0.1        <none>                                      443/TCP                      19m
ingress-nginx   service/ingress-nginx-controller             LoadBalancer   10.97.180.87     172.30.203.204                              80:32003/TCP,443:32626/TCP   9m6s
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP      10.110.169.132   <none>                                      443/TCP                      9m6s
kube-system     service/kube-dns                             ClusterIP      10.96.0.10       <none>                                      53/UDP,53/TCP,9153/TCP       19m
ns-test-pods    service/apache-svc                           NodePort       10.96.121.175    <none>                                      80:31181/TCP                 9m37s
ns-test         service/apache-svc-koitu                     ExternalName   <none>           apache-svc.ns-test-pods.svc.cluster.local   <none>                       9m37s
ns-test         service/nginx-svc                            NodePort       10.111.149.9     <none>                                      80:31453/TCP                 9m37s

NAMESPACE   NAME                    CLASS    HOSTS                           ADDRESS          PORTS   AGE
ns-test     ingress.extensions/lb   <none>   ns-test.172.30.203.204.nip.io   172.30.203.204   80      9m37s
46
25
3

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
46
25

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?