概要
下表の構成で、オンプレ上に k8s を構築したときの記録である.
本記事では「L2 MetalLB の導入」を記す.
| No | 用途 | ノード名 | 形態 | 公開IP | 内部IP | OS | 備考 |
|---|---|---|---|---|---|---|---|
| 1 | k8sマスタ | master01 | VM | 192.168.1.91 | 172.24.20.11 | Ubuntu18.04 | |
| 2 | k8sノード | node01 | VM | 192.168.1.92 | 172.24.20.12 | Ubuntu18.04 |
参考にしたサイトおよび書籍
| URL | 備考 |
|---|---|
| 実践 Vagrant | |
| 15Stepで習得 Dockerから入るKubernetes | K8s だけでなく、Ansible, Vagrant, GlusterFS のことなども学べる. |
| https://github.com/takara9/vagrant-k8s |
『15Stepで習得 Dockerから入るKubernetes』の著者が公開されている GitHub. Vagrant や Ansible コードを公開してくださっている. |
| https://github.com/takara9/vagrant-kubernetes | 同上 |
| https://github.com/takara9/codes_for_lessons | 同上 |
| https://nextpublishing.jp/book/12197.html | 『解体kubeadm フェーズから読み解くKubernetesクラスタ構築ツールの全貌』を参考にして 1マスタ・1ノードを構築した. |
| https://www.kaitoy.xyz/2020/10/31/metallb/ |
環境
物理PC および 仮想PC の OS
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
物理PC
下記ソフトを導入済みであること.
- Vagrant
- VirtualBox
手順
1. master01 に ssh ログインする
$ vagrant ssh master01
2. MetalLB マニフェストを入手する
現時点 (2021.05.26) で最新バージョンである 0.9.6 の MetalLB マニフェストを入手する
vagrant@master01:~$ wget https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/namespace.yaml
vagrant@master01:~$ wget https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/metallb.yaml
3. MetalLB マニフェストを適用する
ここの手順を実行しても、まだ MetalLB は動かない.
vagrant@master01:~$ kubectl apply -f namespace.yaml
namespace/metallb-system created
vagrant@master01:~$ kubectl apply -f metallb.yaml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/controller created
podsecuritypolicy.policy/speaker created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/config-watcher created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/config-watcher created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created
vagrant@master01:~$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
secret/memberlist created
4. MetalLB の L2 用マニフェストを作成する
MetalLB 公式サイトの Layer 2 Configuration から次のように マニフェストを作成する
metallb-config.yml
今回は、172.24.20.245 - 250 までの 6個の LB が導入できるようにしておく.
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 172.24.20.245-172.24.20.250
5. MetalLB の L2 用マニフェストを適用する
Pod「MetalLB」を作成する
vagrant@master01:~$ kubectl apply -f metallb-config.yml
configmap/config created
Pod「MetalLB」の状態を見る (下記🛑)
vagrant@master01:~$ kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-558bd4d5db-86gdp 1/1 Running 0 94m
kube-system coredns-558bd4d5db-9wj98 1/1 Running 0 94m
kube-system etcd-master01 1/1 Running 0 94m
kube-system kube-apiserver-master01 1/1 Running 0 94m
kube-system kube-controller-manager-master01 1/1 Running 1 94m
kube-system kube-flannel-ds-mvlx9 1/1 Running 2 90m
kube-system kube-flannel-ds-n9zbw 1/1 Running 0 94m
kube-system kube-proxy-h9hwt 1/1 Running 0 90m
kube-system kube-proxy-qrng2 1/1 Running 0 94m
kube-system kube-scheduler-master01 1/1 Running 1 94m
kube-system metrics-server-755675c897-h86pc 1/1 Running 0 94m
kubernetes-dashboard cluster-admin-68f7c6c68f-l8hr4 1/1 Running 0 94m
kubernetes-dashboard dashboard-metrics-scraper-778b77d469-9xzl9 1/1 Running 0 94m
metallb-system 🛑 controller-64f86798cc-8f6s4 1/1 Running 0 9m6s
metallb-system 🛑 speaker-tshfh 1/1 Running 0 9m6s
metallb-system 🛑 speaker-xv26j 1/1 Running 0 9m6s
6. MetalLB を使ってみる(1/2) ... Pod「Nginx」を起動させる
外部からアクセスしたい Pod「Nginx」を立てる
6-1. test-metallb-1.yml を作成する
https://www.kaitoy.xyz/2020/10/31/metallb/ からの転記である.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: psp:priv
rules:
- apiGroups:
- policy
resourceNames:
- privileged
resources:
- podsecuritypolicies
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: default:psp:privileged
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: psp:priv
subjects:
- kind: ServiceAccount
name: default
namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.19.1-alpine
ports:
- containerPort: 80
6-2. test-metallb-1.yml を適用する
vagrant@master01:~$ kubectl apply -f test-metallb-1.yml
role.rbac.authorization.k8s.io/psp:priv created
rolebinding.rbac.authorization.k8s.io/default:psp:privileged created
deployment.apps/nginx-deployment created
6-3. Pod「Nginx」が稼動していることを見る
vagrant@master01:~/metallb$ kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-deployment-79d8d59989-q4sff 0/1 ContainerCreating 0 9s 🆗
default nginx-deployment-79d8d59989-t2gxk 0/1 ContainerCreating 0 9s 🆗
kube-system coredns-558bd4d5db-86gdp 1/1 Running 0 95m
kube-system coredns-558bd4d5db-9wj98 1/1 Running 0 95m
kube-system etcd-master01 1/1 Running 0 95m
kube-system kube-apiserver-master01 1/1 Running 0 95m
kube-system kube-controller-manager-master01 1/1 Running 1 95m
kube-system kube-flannel-ds-mvlx9 1/1 Running 2 92m
kube-system kube-flannel-ds-n9zbw 1/1 Running 0 95m
kube-system kube-proxy-h9hwt 1/1 Running 0 92m
kube-system kube-proxy-qrng2 1/1 Running 0 95m
kube-system kube-scheduler-master01 1/1 Running 1 95m
kube-system metrics-server-755675c897-h86pc 1/1 Running 0 95m
kubernetes-dashboard cluster-admin-68f7c6c68f-l8hr4 1/1 Running 0 95m
kubernetes-dashboard dashboard-metrics-scraper-778b77d469-9xzl9 1/1 Running 0 95m
metallb-system controller-64f86798cc-8f6s4 1/1 Running 0 10m
metallb-system speaker-tshfh 1/1 Running 0 10m
metallb-system speaker-xv26j 1/1 Running 0 10m
-o wide を指定して確認してみる.
vagrant@master01:~/metallb$ kubectl get pod -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default 🆗 nginx-deployment-79d8d59989-q4sff 1/1 Running 0 55s 10.244.1.4 node01 <none> <none>
default 🆗 nginx-deployment-79d8d59989-t2gxk 1/1 Running 0 55s 10.244.1.5 node01 <none> <none>
kube-system coredns-558bd4d5db-86gdp 1/1 Running 0 96m 10.244.0.3 master01 <none> <none>
kube-system coredns-558bd4d5db-9wj98 1/1 Running 0 96m 10.244.0.2 master01 <none> <none>
kube-system etcd-master01 1/1 Running 0 96m 172.24.20.11 master01 <none> <none>
kube-system kube-apiserver-master01 1/1 Running 0 96m 172.24.20.11 master01 <none> <none>
kube-system kube-controller-manager-master01 1/1 Running 1 96m 172.24.20.11 master01 <none> <none>
kube-system kube-flannel-ds-mvlx9 1/1 Running 2 92m 172.24.20.12 node01 <none> <none>
kube-system kube-flannel-ds-n9zbw 1/1 Running 0 96m 172.24.20.11 master01 <none> <none>
kube-system kube-proxy-h9hwt 1/1 Running 0 92m 172.24.20.12 node01 <none> <none>
kube-system kube-proxy-qrng2 1/1 Running 0 96m 172.24.20.11 master01 <none> <none>
kube-system kube-scheduler-master01 1/1 Running 1 96m 172.24.20.11 master01 <none> <none>
kube-system metrics-server-755675c897-h86pc 1/1 Running 0 96m 10.244.1.2 node01 <none> <none>
kubernetes-dashboard cluster-admin-68f7c6c68f-l8hr4 1/1 Running 0 96m 10.244.0.4 master01 <none> <none>
kubernetes-dashboard dashboard-metrics-scraper-778b77d469-9xzl9 1/1 Running 0 96m 10.244.0.5 master01 <none> <none>
metallb-system controller-64f86798cc-8f6s4 1/1 Running 0 11m 10.244.1.3 node01 <none> <none>
metallb-system speaker-tshfh 1/1 Running 0 11m 172.24.20.12 node01 <none> <none>
metallb-system speaker-xv26j 1/1 Running 0 11m 172.24.20.11 master01 <none> <none>
7. MetalLB を使ってみる(2/2) ... LoadBalancer Service を動かす
7-1. Service「MetallLB」用のマニフェスト「metallb-service.yml」を作成する
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 8080
targetPort: 80
type: LoadBalancer
7-2. 現在の Service の状態を見ておく
vagrant@master01:~/metallb$ kubectl get svc -A -o wide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default kubernetes ClusterIP 10.32.0.1 <none> 443/TCP 100m <none>
kube-system kube-dns ClusterIP 10.32.0.10 <none> 53/UDP,53/TCP,9153/TCP 100m k8s-app=kube-dns
kube-system metrics-server ClusterIP 10.32.0.51 <none> 443/TCP 100m k8s-app=metrics-server
kubernetes-dashboard dashboard-metrics-scraper ClusterIP 10.32.0.215 <none> 8000/TCP 100m k8s-app=dashboard-metrics-scraper
kubernetes-dashboard kubernetes-dashboard ClusterIP 10.32.0.70 <none> 443/TCP 100m k8s-app=kubernetes-dashboard
7-3. Service「MetallLB」用のマニフェストを適用する
vagrant@master01:~$ kubectl apply -f metallb-service.yml
service/nginx created
7-4. 適用後の Service の状態を見ておく
LoadBalancer「nginx」が追加された
vagrant@master01:~/metallb$ kubectl get svc -A -o wide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default kubernetes ClusterIP 10.32.0.1 <none> 443/TCP 101m <none>
default 🆗 nginx LoadBalancer 10.32.0.188 172.24.20.245 8080:32005/TCP 3s app=nginx
kube-system kube-dns ClusterIP 10.32.0.10 <none> 53/UDP,53/TCP,9153/TCP 101m k8s-app=kube-dns
kube-system metrics-server ClusterIP 10.32.0.51 <none> 443/TCP 101m k8s-app=metrics-server
kubernetes-dashboard dashboard-metrics-scraper ClusterIP 10.32.0.215 <none> 8000/TCP 101m k8s-app=dashboard-metrics-scraper
kubernetes-dashboard kubernetes-dashboard ClusterIP 10.32.0.70 <none> 443/TCP 101m k8s-app=kubernetes-dashboard
describe を使って詳細を見てみる.
下記 🆗 の通り、LoadBalancer Service「nginx」に対して、
EXTERNAL-IP 172.24.20.245 が割り当てられた.
vagrant@master01:~/metallb$ kubectl describe svc nginx
Name: nginx
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.32.0.188
IPs: 10.32.0.188
LoadBalancer Ingress: 172.24.20.245 🆗
Port: <unset> 8080/TCP
TargetPort: 80/TCP
NodePort: <unset> 32005/TCP
Endpoints: 10.244.1.4:80,10.244.1.5:80
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IPAllocated 54s metallb-controller Assigned IP "172.24.20.245"
Normal nodeAssigned 53s metallb-speaker announcing from node "node01"
8. 動作確認をする
8-1. Service「LoadBalancer」の状態を見る
次の通り クラスタ外に対して 172.24.20.245 が公開できた
vagrant@master01:~/metallb$ kubectl get svc -A -o wide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default kubernetes ClusterIP 10.32.0.1 <none> 443/TCP 101m <none>
default 🆗 nginx LoadBalancer 10.32.0.188 172.24.20.245 8080:32005/TCP 3s app=nginx
kube-system kube-dns ClusterIP 10.32.0.10 <none> 53/UDP,53/TCP,9153/TCP 101m k8s-app=kube-dns
kube-system metrics-server ClusterIP 10.32.0.51 <none> 443/TCP 101m k8s-app=metrics-server
kubernetes-dashboard dashboard-metrics-scraper ClusterIP 10.32.0.215 <none> 8000/TCP 101m k8s-app=dashboard-metrics-scraper
kubernetes-dashboard kubernetes-dashboard ClusterIP 10.32.0.70 <none> 443/TCP 101m k8s-app=kubernetes-dashboard
8-2. クラスタ外 (ここでは物理ホスト 192.168.10.115) から 1 の EXTERNAL-IP の 8080 にアクセスする
クラスタ外から wget を実行してみる
ここでは物理ホスト上での操作である.
L4 LoadBalancer の IP:PORT は 172.24.20.245:8080 である.
$ wget http://172.24.20.245:8080 -O -
--2021-05-29 11:30:29-- http://172.24.20.245:8080/
Connecting to 172.24.20.245:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 612 [text/html]
Saving to: ‘STDOUT’
- 0%[ ] 0 --.-KB/s <!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
- 100%[==================================================================================================================>] 612 --.-KB/s in 0s
2021-05-29 11:30:29 (103 MB/s) - written to stdout [612/612]
Web Browser からのアクセスをしてみる
・http://172.24.20.245 にアクセスし、Nginx のトップページが表示された.
