LoginSignup
2
4

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

@Tsu_hao_Zhang(Tsu_hao Zhang)

Kubernetes 1.15.1 バイナリからインストール(worker編)

Last updated at Posted at 2019-10-03

Master編よりも先にWorker編を公開してしまう。
worker node(以下、node)にKubernetesを入れていく。
Kubeadmは未使用。

nodeのHostファイル編集

###各nodeの/etc/hostsにMasterの/etc/hostsと同じエントリーを追加
Masterの/etc/hostsの内容をnodeにも追加します

[master3]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.1.201 master1
192.168.1.202 master2
192.168.1.203 master3

192.168.1.201 k8svip.com

127.0.0.1と::1はすでにNodeのHostファイルにも書き込まれているはずなので、
それ以下の192.168.1.201,2,3とVIPをコピーします。

つまり、追加後にNodeのHostファイルをCatすると以下の状態になっているはずです

[root@node1 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.1.201 master1
192.168.1.202 master2
192.168.1.203 master3

192.168.1.201 k8svip.com

#Master から証明書をコピー

(任意)Masterにてssh証明書を作成し、rsyncの認証をなくす(Masterから操作)

[master]#ssh-keygen
#すべてEnter

[master]#ssh-copy-id node1
#node1のパスワードを入力。node1は各自の環境、IPに変更

フォルダ作成(workerから)

mkdir -p /etc/kubernetes/

(Masterから操作)(node01は各自のノード名 or IP address)

[master]# rsync -av /etc/kubernetes/ssl/ca* node01:/etc/kubernetes/ssl/
[master]# rsync -av /etc/kubernetes/config node01:/etc/kubernetes/
[master]# rsync -av /etc/kubernetes/bootstrap.kubeconfig node01:/etc/kubernetes/

#dockerまわりのインストールと起動

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce

systemctl enable docker
systemctl daemon-reload
systemctl start docker

#Calico
##calico service登録と起動
注意点として
Environment=ETCD_ENDPOINTS=
は各自のMasterのホストネーム等に置き換えること。

cat >/usr/lib/systemd/system/calico-node.service <<"EOF"
[Unit]
Description=calico-node
After=docker.service
Requires=docker.service
[Service]
User=root
Environment=ETCD_ENDPOINTS=http://master1:2379,http://master2:2379,http://master3:2379
PermissionsStartOnly=true
ExecStart=/usr/bin/docker run --net=host --privileged --name=calico-node \
  -e ETCD_ENDPOINTS=${ETCD_ENDPOINTS} \
  -e NODENAME=${HOSTNAME} \
  -e IP= \
  -e IP6= \
  -e AS= \
  -e NO_DEFAULT_POOLS= \
  -e CALICO_NETWORKING_BACKEND=bird \
  -e FELIX_DEFAULTENDPOINTTOHOSTACTION=ACCEPT \
  -e CALICO_LIBNETWORK_ENABLED=true \
  -v /lib/modules:/lib/modules \
  -v /run/docker/plugins:/run/docker/plugins \
  -v /var/run/calico:/var/run/calico \
  -v /var/log/calico:/var/log/calico \
  -v /var/lib/calico:/var/lib/calico \
  calico/node:v3.8.0
ExecStop=/usr/bin/docker rm -f calico-node
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF


systemctl daemon-reload
systemctl start calico-node
systemctl enable calico-node
systemctl status calico-node

systemctl status calico-nodeは数回繰り返してプロセスが落ちないことを確認すること。
Proxy環境下だとDocker pullができずに落ちる。
そのためProxy環境下の場合はDockerのProxyの設定をすること。

###DockerのProxyの設定(Proxy環境下の場合のみ)

mkdir -p /etc/systemd/system/docker.service.d

cat >/etc/systemd/system/docker.service.d/http-proxy.conf <<"EOF"

[Service]
Environment="HTTP_PROXY=http://<user>:<pass>@<proxy_host>:<proxy_port>" "HTTPS_PROXY=http://<user>:<pass>@<proxy_host>:<proxy_port>" "NO_PROXY=localhost"
EOF

systemctl daemon-reload
systemctl restart docker
systemctl start calico-node
systemctl status calico-node

Environment= は私の環境の例だと

Environment="HTTP_PROXY=http://10.0.0.254:8080" "HTTPS_PROXY=http://"10.0.0.254:8080 "NO_PROXY=localhost"

##Calico諸設定

下記の "etcd_endpoints": "http://master1:2379,http://master2:2379,http://master3:2379",は各自環境に合わせること。

[ -d /opt/cni/bin ] || mkdir -p /opt/cni/bin && cd /opt/cni/bin
wget -N https://github.com/projectcalico/cni-plugin/releases/download/v3.8.0/calico-amd64
wget -N https://github.com/projectcalico/cni-plugin/releases/download/v3.8.0/calico-ipam-amd64
mv calico-ipam-amd64 calico-ipam
mv calico-amd64 calico
chmod +x /opt/cni/bin/calico /opt/cni/bin/calico-ipam


[ -d /var/src ] || mkdir /var/src && cd /var/src
wget https://github.com/containernetworking/plugins/releases/download/v0.7.1/cni-plugins-amd64-v0.7.1.tgz
mkdir cni-plugins-amd64-v0.7.1
tar -xzvf cni-plugins-amd64-v0.7.1.tgz -C cni-plugins-amd64-v0.7.1
rsync -av cni-plugins-amd64-v0.7.1/* /opt/cni/bin/
mkdir -p /etc/cni/net.d
rm -f /etc/cni/net.d/*


cat >/etc/cni/net.d/10-calico.conflist <<EOF
{
  "name": "calico-k8s-network",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "calico",
      "log_level": "info",
      "etcd_endpoints": "http://master1:2379,http://master2:2379,http://master3:2379",
      "ipam": {
        "type": "calico-ipam"
      },
      "policy": {
        "type": "k8s"
      },
      "kubernetes": {
        "kubeconfig": "/etc/kubernetes/kubelet.kubeconfig"
      }
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true}
    }
  ]
}
EOF

上記で指定されている /etc/kubernetes/kubelet.kubeconfig はMasterからコピーする。(Masterから操作)

[master]# rsync -av /root/.kube/config node1:/etc/kubernetes/kubelet.kubeconfig

#kubelet導入
##ファイル準備とプロファイル作成

cd /var/src
wget https://dl.k8s.io/v1.15.2/kubernetes-node-linux-amd64.tar.gz -O kubernetes-node-linux-amd64-v1.15.2.tar.gz
tar -xzvf kubernetes-node-linux-amd64-v1.15.2.tar.gz
mv kubernetes /usr/local/

cat > /etc/profile.d/kubernetes.sh << EOF
export PATH=/usr/local/kubernetes/node/bin:\$PATH
EOF

source /etc/profile

mkdir -p /var/lib/kubelet

##Kubelet service登録

cat > /usr/lib/systemd/system/kubelet.service <<"EOF"
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
User=root
Group=root
EnvironmentFile=/etc/kubernetes/kubelet.conf
ExecStart=/usr/local/kubernetes/node/bin/kubelet $KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

##諸フォルダ作成

mkdir /var/log/kubernetes
mkdir -p /var/lib/kubelet

##kubelet.conf作成
--hostname-override=node1 は各自のノード名に合わせる

cat > /etc/kubernetes/kubelet.conf <<"EOF"
KUBELET_OPTS="\
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4 \
  --hostname-override=node1 \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
  --config=/etc/kubernetes/kubelet.yaml \
  --cert-dir=/etc/kubernetes/ssl \
  --network-plugin=cni \
  --container-runtime=docker \
  --container-runtime-endpoint=unix:///var/run/dockershim.sock \
  --pod-infra-container-image=gcr.io/google-containers/pause:3.1"
EOF

##kubelet.yaml作成
clusterDNS: ["10.254.0.2"] は各自の環境に合わせる?

cat > /etc/kubernetes/kubelet.yaml <<"EOF"
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:  ["10.254.0.2"]
clusterDomain: cluster.local.
failSwapOn: false
hairpin-mode: promiscuous-bridge
authentication:
  anonymous:
    enabled: false
  webhook:
    enbaled: false
  x509:
    clientCAFile: "/etc/kubernetes/ssl/ca.pem"
EOF

ここでKubeletを起動しても起動できないので一旦保留。

#Kube-proxy導入
##下準備

yum install conntrack-tools -y

##Masterから証明書等をコピー(Masterから操作)
node1は各自のノード名に置き換える

[master]# rsync -av /etc/kubernetes/kube-proxy.kubeconfig node1:/etc/kubernetes/
[master]# rsync -av /etc/kubernetes/ssl/kube-proxy* node1:/etc/kubernetes/ssl/
[master]# rsync -av /etc/kubernetes/config node1:/etc/kubernetes/

##Kube-proxyのサービスを作成

cat > /usr/lib/systemd/system/kube-proxy.service <<"EOF"
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/proxy.conf
ExecStart=/usr/local/kubernetes/node/bin/kube-proxy \
        --config=/etc/kubernetes/proxy.yaml \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOGDIR \
        $KUBE_LOG_LEVEL \
        $KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

##Kube-proxy.yaml作成
hostnameOverride: node1　は各自のノード名に置き換えること。

cat > /etc/kubernetes/proxy.yaml <<"EOF"
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  burst: 200
  kubeconfig: "/etc/kubernetes/kube-proxy.kubeconfig"
  qps: 100
bindAddress: 0.0.0.0
healthzBindAddress: 0.0.0.0:10256
metricsBindAddress: 0.0.0.0:10249
clusterCIDR: 192.168.0.0/16
hostnameOverride: node1
mode: ipvs
kubeProxyIPVSConfiguration:
  syncPeriod: 5s
  minSyncPeriod: 1s
  scheduler: lc
EOF

余談ですけど、api versionのv1alpha1ってなんやねん。
ここがイマイチよくわかっていないのですが、Proxyの設定が後述のどちらを見ているのかわからないため
下記も設定します。

##proxy.conf作成
KUBE_PROXY_ARGS="--hostname-override=node1　は各自のノード名に置き換えること。

cat >/etc/kubernetes/proxy.conf <<"EOF"
KUBE_PROXY_ARGS="--hostname-override=node1 \
                 --cluster-cidr=192.168.0.0/16 \
                 --ipvs-min-sync-period=5s \
                 --ipvs-sync-period=5s \
                 --ipvs-scheduler=rr"
EOF

#ipvs(NAT,ロードバランサー)導入
コンテナ内外のネットワークをつなぐために導入します。しかもロードバランシングもしてくれる。
##ipvs.module作成

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack_ipv4"
for kernel_module in \${ipvs_modules}; do
    /sbin/modinfo -F filename \${kernel_module} > /dev/null 2>&1
    if [ $? -eq 0 ]; then
        /sbin/modprobe \${kernel_module}
    fi
done
EOF

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs

##ipvs.conf作成

cat > /etc/modprobe.d/ip_vs.conf << EOF
options ip_vs conn_tab_bits=20
EOF

modprobe -r ip_vs

#戦闘開始

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet

systemctl enable kube-proxy
systemctl start kube-proxy
systemctl status kube-proxy

Masterにて

(Master)kube-proxy-rbac.yamlの作成(Masterのうち1台で実行)

[master]# cat >kube-proxy-rbac.yaml <<"EOF"
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-proxy
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:kube-proxy
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
  - kind: ServiceAccount
    name: kube-proxy
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:node-proxier
  apiGroup: rbac.authorization.k8s.io
EOF

kube-proxy-rbacデプロイ(Master)

[master]# kubectl apply -f kube-proxy-rbac.yaml

Nginxでテスト(Master)

試しに定番のnginxを動かしてみましょう。

[master]# cat > nginx-ds.yml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: nginx-ds
  labels:
    app: nginx-ds
spec:
  type: NodePort
  selector:
    app: nginx-ds
  ports:
  - name: http
    port: 80
    targetPort: 80

---

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ds
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  template:
    metadata:
      labels:
        app: nginx-ds
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80

EOF

[master]# kubectl create -f nginx-ds.yml
[master]# kubectl get pods

[master]#  kubectl get pods
#NAME             READY   STATUS    RESTARTS   AGE
#nginx-ds-bqc6t   1/1     Running   0          1m
2
4
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
2
4

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?