概要

下表の構成で、オンプレ上に k8s を構築したときの記録である.
本記事では「L2 MetalLB の導入」を記す.

No	用途	ノード名	形態	公開IP	内部IP	OS	備考
1	k8sマスタ	master01	VM	192.168.1.91	172.24.20.11	Ubuntu18.04
2	k8sノード	node01	VM	192.168.1.92	172.24.20.12	Ubuntu18.04

参考にしたサイトおよび書籍

URL	備考
実践 Vagrant
15Stepで習得 Dockerから入るKubernetes	K8s だけでなく、Ansible, Vagrant, GlusterFS のことなども学べる.
https://github.com/takara9/vagrant-k8s	『15Stepで習得 Dockerから入るKubernetes』の著者が公開されている GitHub. Vagrant や Ansible コードを公開してくださっている.
https://github.com/takara9/vagrant-kubernetes	同上
https://github.com/takara9/codes_for_lessons	同上
https://nextpublishing.jp/book/12197.html	『解体kubeadm　フェーズから読み解くKubernetesクラスタ構築ツールの全貌』を参考にして 1マスタ・1ノードを構築した.
https://www.kaitoy.xyz/2020/10/31/metallb/

環境

物理PC および仮想PC の OS

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"

物理PC

下記ソフトを導入済みであること.

Vagrant
VirtualBox

手順

1. master01 に ssh ログインする

$ vagrant ssh master01

2. MetalLB マニフェストを入手する

現時点 (2021.05.26) で最新バージョンである 0.9.6 の MetalLB マニフェストを入手する

vagrant@master01:~$ wget https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/namespace.yaml
vagrant@master01:~$ wget https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/metallb.yaml

3. MetalLB マニフェストを適用する

ここの手順を実行しても、まだ MetalLB は動かない.

vagrant@master01:~$ kubectl apply -f namespace.yaml
namespace/metallb-system created

vagrant@master01:~$ kubectl apply -f metallb.yaml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/controller created
podsecuritypolicy.policy/speaker created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/config-watcher created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/config-watcher created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created

vagrant@master01:~$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
secret/memberlist created

4. MetalLB の L2 用マニフェストを作成する

MetalLB 公式サイトの Layer 2 Configuration から次のようにマニフェストを作成する

metallb-config.yml

今回は、172.24.20.245 - 250 までの 6個の LB が導入できるようにしておく.

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 172.24.20.245-172.24.20.250

5. MetalLB の L2 用マニフェストを適用する

Pod「MetalLB」を作成する

vagrant@master01:~$ kubectl apply -f metallb-config.yml
configmap/config created

Pod「MetalLB」の状態を見る (下記🛑)

vagrant@master01:~$ kubectl get pod -A
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
kube-system            coredns-558bd4d5db-86gdp                     1/1     Running   0          94m
kube-system            coredns-558bd4d5db-9wj98                     1/1     Running   0          94m
kube-system            etcd-master01                                1/1     Running   0          94m
kube-system            kube-apiserver-master01                      1/1     Running   0          94m
kube-system            kube-controller-manager-master01             1/1     Running   1          94m
kube-system            kube-flannel-ds-mvlx9                        1/1     Running   2          90m
kube-system            kube-flannel-ds-n9zbw                        1/1     Running   0          94m
kube-system            kube-proxy-h9hwt                             1/1     Running   0          90m
kube-system            kube-proxy-qrng2                             1/1     Running   0          94m
kube-system            kube-scheduler-master01                      1/1     Running   1          94m
kube-system            metrics-server-755675c897-h86pc              1/1     Running   0          94m
kubernetes-dashboard   cluster-admin-68f7c6c68f-l8hr4               1/1     Running   0          94m
kubernetes-dashboard   dashboard-metrics-scraper-778b77d469-9xzl9   1/1     Running   0          94m
metallb-system 🛑      controller-64f86798cc-8f6s4                  1/1     Running   0          9m6s
metallb-system 🛑      speaker-tshfh                                1/1     Running   0          9m6s
metallb-system 🛑      speaker-xv26j                                1/1     Running   0          9m6s

6. MetalLB を使ってみる(1/2) ... Pod「Nginx」を起動させる

外部からアクセスしたい Pod「Nginx」を立てる
　

6-1. test-metallb-1.yml を作成する

https://www.kaitoy.xyz/2020/10/31/metallb/ からの転記である.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: psp:priv
rules:
- apiGroups:
  - policy
  resourceNames:
  - privileged
  resources:
  - podsecuritypolicies
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default:psp:privileged
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: psp:priv
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.19.1-alpine
        ports:
        - containerPort: 80

6-2. test-metallb-1.yml を適用する

vagrant@master01:~$ kubectl apply -f test-metallb-1.yml
role.rbac.authorization.k8s.io/psp:priv created
rolebinding.rbac.authorization.k8s.io/default:psp:privileged created
deployment.apps/nginx-deployment created

6-3. Pod「Nginx」が稼動していることを見る

vagrant@master01:~/metallb$ kubectl get pod -A
NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
default                nginx-deployment-79d8d59989-q4sff            0/1     ContainerCreating   0          9s  🆗
default                nginx-deployment-79d8d59989-t2gxk            0/1     ContainerCreating   0          9s  🆗
kube-system            coredns-558bd4d5db-86gdp                     1/1     Running             0          95m
kube-system            coredns-558bd4d5db-9wj98                     1/1     Running             0          95m
kube-system            etcd-master01                                1/1     Running             0          95m
kube-system            kube-apiserver-master01                      1/1     Running             0          95m
kube-system            kube-controller-manager-master01             1/1     Running             1          95m
kube-system            kube-flannel-ds-mvlx9                        1/1     Running             2          92m
kube-system            kube-flannel-ds-n9zbw                        1/1     Running             0          95m
kube-system            kube-proxy-h9hwt                             1/1     Running             0          92m
kube-system            kube-proxy-qrng2                             1/1     Running             0          95m
kube-system            kube-scheduler-master01                      1/1     Running             1          95m
kube-system            metrics-server-755675c897-h86pc              1/1     Running             0          95m
kubernetes-dashboard   cluster-admin-68f7c6c68f-l8hr4               1/1     Running             0          95m
kubernetes-dashboard   dashboard-metrics-scraper-778b77d469-9xzl9   1/1     Running             0          95m
metallb-system         controller-64f86798cc-8f6s4                  1/1     Running             0          10m
metallb-system         speaker-tshfh                                1/1     Running             0          10m
metallb-system         speaker-xv26j                                1/1     Running             0          10m

-o wide を指定して確認してみる.

vagrant@master01:~/metallb$ kubectl get pod -A -o wide
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
default 🆗             nginx-deployment-79d8d59989-q4sff            1/1     Running   0          55s   10.244.1.4     node01     <none>           <none>
default 🆗             nginx-deployment-79d8d59989-t2gxk            1/1     Running   0          55s   10.244.1.5     node01     <none>           <none>
kube-system            coredns-558bd4d5db-86gdp                     1/1     Running   0          96m   10.244.0.3     master01   <none>           <none>
kube-system            coredns-558bd4d5db-9wj98                     1/1     Running   0          96m   10.244.0.2     master01   <none>           <none>
kube-system            etcd-master01                                1/1     Running   0          96m   172.24.20.11   master01   <none>           <none>
kube-system            kube-apiserver-master01                      1/1     Running   0          96m   172.24.20.11   master01   <none>           <none>
kube-system            kube-controller-manager-master01             1/1     Running   1          96m   172.24.20.11   master01   <none>           <none>
kube-system            kube-flannel-ds-mvlx9                        1/1     Running   2          92m   172.24.20.12   node01     <none>           <none>
kube-system            kube-flannel-ds-n9zbw                        1/1     Running   0          96m   172.24.20.11   master01   <none>           <none>
kube-system            kube-proxy-h9hwt                             1/1     Running   0          92m   172.24.20.12   node01     <none>           <none>
kube-system            kube-proxy-qrng2                             1/1     Running   0          96m   172.24.20.11   master01   <none>           <none>
kube-system            kube-scheduler-master01                      1/1     Running   1          96m   172.24.20.11   master01   <none>           <none>
kube-system            metrics-server-755675c897-h86pc              1/1     Running   0          96m   10.244.1.2     node01     <none>           <none>
kubernetes-dashboard   cluster-admin-68f7c6c68f-l8hr4               1/1     Running   0          96m   10.244.0.4     master01   <none>           <none>
kubernetes-dashboard   dashboard-metrics-scraper-778b77d469-9xzl9   1/1     Running   0          96m   10.244.0.5     master01   <none>           <none>
metallb-system         controller-64f86798cc-8f6s4                  1/1     Running   0          11m   10.244.1.3     node01     <none>           <none>
metallb-system         speaker-tshfh                                1/1     Running   0          11m   172.24.20.12   node01     <none>           <none>
metallb-system         speaker-xv26j                                1/1     Running   0          11m   172.24.20.11   master01   <none>           <none>

7. MetalLB を使ってみる(2/2) ... LoadBalancer Service を動かす

7-1. Service「MetallLB」用のマニフェスト「metallb-service.yml」を作成する

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 80
  type: LoadBalancer

7-2. 現在の Service の状態を見ておく

vagrant@master01:~/metallb$ kubectl get svc -A -o wide 
NAMESPACE              NAME                        TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE    SELECTOR
default                kubernetes                  ClusterIP   10.32.0.1     <none>        443/TCP                  100m   <none>
kube-system            kube-dns                    ClusterIP   10.32.0.10    <none>        53/UDP,53/TCP,9153/TCP   100m   k8s-app=kube-dns
kube-system            metrics-server              ClusterIP   10.32.0.51    <none>        443/TCP                  100m   k8s-app=metrics-server
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP   10.32.0.215   <none>        8000/TCP                 100m   k8s-app=dashboard-metrics-scraper
kubernetes-dashboard   kubernetes-dashboard        ClusterIP   10.32.0.70    <none>        443/TCP                  100m   k8s-app=kubernetes-dashboard

7-3. Service「MetallLB」用のマニフェストを適用する

vagrant@master01:~$ kubectl apply -f metallb-service.yml
service/nginx created

7-4. 適用後の Service の状態を見ておく

LoadBalancer「nginx」が追加された

vagrant@master01:~/metallb$ kubectl get svc -A -o wide 
NAMESPACE              NAME                        TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                  AGE    SELECTOR
default                kubernetes                  ClusterIP      10.32.0.1     <none>          443/TCP                  101m   <none>
default 🆗             nginx                       LoadBalancer   10.32.0.188   172.24.20.245   8080:32005/TCP           3s     app=nginx
kube-system            kube-dns                    ClusterIP      10.32.0.10    <none>          53/UDP,53/TCP,9153/TCP   101m   k8s-app=kube-dns
kube-system            metrics-server              ClusterIP      10.32.0.51    <none>          443/TCP                  101m   k8s-app=metrics-server
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP      10.32.0.215   <none>          8000/TCP                 101m   k8s-app=dashboard-metrics-scraper
kubernetes-dashboard   kubernetes-dashboard        ClusterIP      10.32.0.70    <none>          443/TCP                  101m   k8s-app=kubernetes-dashboard

describe を使って詳細を見てみる.

下記 🆗 の通り、LoadBalancer Service「nginx」に対して、
EXTERNAL-IP 172.24.20.245 が割り当てられた.

vagrant@master01:~/metallb$ kubectl describe svc nginx
Name:                     nginx
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.32.0.188
IPs:                      10.32.0.188
LoadBalancer Ingress:     172.24.20.245    🆗
Port:                     <unset>  8080/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  32005/TCP
Endpoints:                10.244.1.4:80,10.244.1.5:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason        Age   From                Message
  ----    ------        ----  ----                -------
  Normal  IPAllocated   54s   metallb-controller  Assigned IP "172.24.20.245"
  Normal  nodeAssigned  53s   metallb-speaker     announcing from node "node01"

8. 動作確認をする

8-1. Service「LoadBalancer」の状態を見る

次の通りクラスタ外に対して 172.24.20.245 が公開できた

vagrant@master01:~/metallb$ kubectl get svc -A -o wide 
NAMESPACE              NAME                        TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                  AGE    SELECTOR
default                kubernetes                  ClusterIP      10.32.0.1     <none>          443/TCP                  101m   <none>
default 🆗             nginx                       LoadBalancer   10.32.0.188   172.24.20.245   8080:32005/TCP           3s     app=nginx
kube-system            kube-dns                    ClusterIP      10.32.0.10    <none>          53/UDP,53/TCP,9153/TCP   101m   k8s-app=kube-dns
kube-system            metrics-server              ClusterIP      10.32.0.51    <none>          443/TCP                  101m   k8s-app=metrics-server
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP      10.32.0.215   <none>          8000/TCP                 101m   k8s-app=dashboard-metrics-scraper
kubernetes-dashboard   kubernetes-dashboard        ClusterIP      10.32.0.70    <none>          443/TCP                  101m   k8s-app=kubernetes-dashboard

8-2. クラスタ外 (ここでは物理ホスト 192.168.10.115) から 1 の EXTERNAL-IP の 8080 にアクセスする

クラスタ外から wget を実行してみる

ここでは物理ホスト上での操作である.
L4 LoadBalancer の IP:PORT は 172.24.20.245:8080 である.

$ wget http://172.24.20.245:8080 -O -
--2021-05-29 11:30:29--  http://172.24.20.245:8080/
Connecting to 172.24.20.245:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 612 [text/html]
Saving to: ‘STDOUT’

-                                                     0%[                                                                                                                   ]       0  --.-KB/s               <!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
-                                                   100%[==================================================================================================================>]     612  --.-KB/s    in 0s      

2021-05-29 11:30:29 (103 MB/s) - written to stdout [612/612]

Web Browser からのアクセスをしてみる

・http://172.24.20.245 にアクセスし、Nginx のトップページが表示された.