はじめに

今回は、レプリカセット、デプロイメント、サービスあたり。
この辺は、ノードを意識した機能も出てくるため、minikube環境ではなく、VirtualBox上に複数ノードによるKubernetesクラスター環境を構築して試してみることにします。

参考情報

kubernetes セットアップ
 2019年版・Kubernetesクラスタ構築入門

K8s Node障害時の振る舞いについての検証記録
 Kubernetesはクラスタで障害があったとき、どういう動きをするのか
 Kubernetesのノード障害時の挙動とノード復帰後の再スケジューリング

#マルチノードのkubernetesクラスター環境構築

Windows10 + VirtualBox上に、マルチノードのKubernetes環境を構築します。
構成を支援するツールも環境によって色々提供されているっぽいですが、オンプレ環境に一から構築しようと思ったらkubeadm辺りが使われるんですかね。
https://kubernetes.io/ja/docs/setup/independent/install-kubeadm/

お勉強用に手っ取り環境作りたいだけなので、以下に提供されているものをそのまま使います。
https://github.com/takara9/vagrant-kubernetes
こいつを使うとVagrantとAnsibleを使ってKubernetesクラスター環境を構築してくれます(MasterNode x1, WorkerNode x2)。素晴らしいですねぇ。

構築手順メモ

【H/W】
ThankPad T480
CPU Intel Core i5-8350U 1.70GHz
Memory 32GB

【OS,S/W】
Windows10
VirtualBox 6.0.8
Vagrant 2.2.4
Git 2.21.0

プロジェクトをクローンして、ローカルにVagrantfileなどを持ってきます。

c:\y\Vagrant>git clone https://github.com/takara9/vagrant-kubernetes
Cloning into 'vagrant-kubernetes'...
remote: Enumerating objects: 21, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (14/14), done.
Receiving objects:  87% (201/230)
Receiving objects: 100% (230/230), 37.66 KiB | 332.00 KiB/s, done.
Resolving deltas: 100% (122/122), done.

で、起動。これだけ。楽ちん。素晴らしい。

c:\y\Vagrant>cd vagrant-kubernetes
c:\y\Vagrant\vagrant-kubernetes>vagrant up
==> vagrant: A new version of Vagrant is available: 2.2.6 (installed version: 2.2.4)!
==> vagrant: To upgrade visit: https://www.vagrantup.com/downloads.html

Bringing machine 'node1' up with 'virtualbox' provider...
Bringing machine 'node2' up with 'virtualbox' provider...
Bringing machine 'master' up with 'virtualbox' provider...
==> node1: Importing base box 'ubuntu/xenial64'...
==> node1: Matching MAC address for NAT networking...
==> node1: Checking if box 'ubuntu/xenial64' version '20190613.1.0' is up to date...
==> node1: A newer version of the box 'ubuntu/xenial64' for provider 'virtualbox' is
==> node1: available! You currently have version '20190613.1.0'. The latest is version
==> node1: '20191113.0.0'. Run `vagrant box update` to update.
==> node1: Setting the name of the VM: vagrant-kubernetes_node1_1573712027100_58519
==> node1: Clearing any previously set network interfaces...
==> node1: Preparing network interfaces based on configuration...
    node1: Adapter 1: nat
    node1: Adapter 2: hostonly
==> node1: Forwarding ports...
    node1: 22 (guest) => 2222 (host) (adapter 1)
==> node1: Running 'pre-boot' VM customizations...
==> node1: Booting VM...
==> node1: Waiting for machine to boot. This may take a few minutes...
    node1: SSH address: 127.0.0.1:2222
    node1: SSH username: vagrant
    node1: SSH auth method: private key
    node1:
    node1: Vagrant insecure key detected. Vagrant will automatically replace
    node1: this with a newly generated keypair for better security.
    node1:
    node1: Inserting generated public key within guest...
    node1: Removing insecure key from the guest if it's present...
    node1: Key inserted! Disconnecting and reconnecting using new SSH key...
==> node1: Machine booted and ready!
==> node1: Checking for guest additions in VM...
    node1: The guest additions on this VM do not match the installed version of
    node1: VirtualBox! In most cases this is fine, but in rare cases it can
    node1: prevent things such as shared folders from working properly. If you see
    node1: shared folder errors, please make sure the guest additions within the
    node1: virtual machine match the version of VirtualBox you have installed on
    node1: your host and reload your VM.
    node1:
    node1: Guest Additions Version: 5.1.38
    node1: VirtualBox Version: 6.0
==> node1: Setting hostname...
==> node1: Configuring and enabling network interfaces...
==> node1: Mounting shared folders...
    node1: /vagrant => C:/y/Vagrant/vagrant-kubernetes
==> node1: Running provisioner: ansible_local...
    node1: Installing Ansible...
Vagrant has automatically selected the compatibility mode '2.0'
according to the Ansible version installed (2.8.6).

Alternatively, the compatibility mode can be specified in your Vagrantfile:
https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode
    node1: Running ansible-playbook...

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [node1]

TASK [include vars] ************************************************************
ok: [node1]

TASK [debug] *******************************************************************
ok: [node1] => {
    "k8s_minor_ver": "14"
}

TASK [Add Docker GPG key] ******************************************************
changed: [node1]

TASK [Add Docker APT repository] ***********************************************
changed: [node1]

TASK [Install a list of packages] **********************************************
[WARNING]: Could not find aptitude. Using apt-get instead
changed: [node1]

TASK [usermod -aG docker vagrant] **********************************************
changed: [node1]

TASK [Set sysctl] **************************************************************
changed: [node1]

TASK [Add GlusterFS Repository] ************************************************
changed: [node1]

TASK [Install GlusterFS client] ************************************************
changed: [node1]

TASK [add Kubernetes apt-key] **************************************************
changed: [node1]

TASK [add Kubernetes' APT repository] ******************************************
changed: [node1]

TASK [Install kubectl] *********************************************************
changed: [node1]

TASK [Install kubelet] *********************************************************
changed: [node1]

TASK [Install kubeadm] *********************************************************
changed: [node1]

TASK [include vars] ************************************************************
ok: [node1]

TASK [getting hostonly ip address] *********************************************
changed: [node1]

TASK [debug] *******************************************************************
ok: [node1] => {
    "ip.stdout_lines[1].split(':')[1].split(' ')[0]": "172.16.20.12"
}

TASK [change 10-kubeadm.conf for v1.11 or later] *******************************
changed: [node1]

TASK [change 10-kubeadm.conf for v1.9 or v1.10] ********************************
skipping: [node1]

TASK [daemon-reload and restart kubelet] ***************************************
changed: [node1]

PLAY RECAP *********************************************************************
node1                      : ok=20   changed=15   unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

==> node2: Importing base box 'ubuntu/xenial64'...
==> node2: Matching MAC address for NAT networking...
==> node2: Checking if box 'ubuntu/xenial64' version '20190613.1.0' is up to date...
==> node2: Setting the name of the VM: vagrant-kubernetes_node2_1573712312261_68038
==> node2: Fixed port collision for 22 => 2222. Now on port 2200.
==> node2: Clearing any previously set network interfaces...
==> node2: Preparing network interfaces based on configuration...
    node2: Adapter 1: nat
    node2: Adapter 2: hostonly
==> node2: Forwarding ports...
    node2: 22 (guest) => 2200 (host) (adapter 1)
==> node2: Running 'pre-boot' VM customizations...
==> node2: Booting VM...
==> node2: Waiting for machine to boot. This may take a few minutes...
    node2: SSH address: 127.0.0.1:2200
    node2: SSH username: vagrant
    node2: SSH auth method: private key
    node2:
    node2: Vagrant insecure key detected. Vagrant will automatically replace
    node2: this with a newly generated keypair for better security.
    node2:
    node2: Inserting generated public key within guest...
    node2: Removing insecure key from the guest if it's present...
    node2: Key inserted! Disconnecting and reconnecting using new SSH key...
==> node2: Machine booted and ready!
==> node2: Checking for guest additions in VM...
    node2: The guest additions on this VM do not match the installed version of
    node2: VirtualBox! In most cases this is fine, but in rare cases it can
    node2: prevent things such as shared folders from working properly. If you see
    node2: shared folder errors, please make sure the guest additions within the
    node2: virtual machine match the version of VirtualBox you have installed on
    node2: your host and reload your VM.
    node2:
    node2: Guest Additions Version: 5.1.38
    node2: VirtualBox Version: 6.0
==> node2: Setting hostname...
==> node2: Configuring and enabling network interfaces...
==> node2: Mounting shared folders...
    node2: /vagrant => C:/y/Vagrant/vagrant-kubernetes
==> node2: Running provisioner: ansible_local...
    node2: Installing Ansible...
Vagrant has automatically selected the compatibility mode '2.0'
according to the Ansible version installed (2.8.6).

Alternatively, the compatibility mode can be specified in your Vagrantfile:
https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode
    node2: Running ansible-playbook...

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [node2]

TASK [include vars] ************************************************************
ok: [node2]

TASK [debug] *******************************************************************
ok: [node2] => {
    "k8s_minor_ver": "14"
}

TASK [Add Docker GPG key] ******************************************************
changed: [node2]

TASK [Add Docker APT repository] ***********************************************
changed: [node2]

TASK [Install a list of packages] **********************************************
[WARNING]: Could not find aptitude. Using apt-get instead
changed: [node2]

TASK [usermod -aG docker vagrant] **********************************************
changed: [node2]

TASK [Set sysctl] **************************************************************
changed: [node2]

TASK [Add GlusterFS Repository] ************************************************
changed: [node2]

TASK [Install GlusterFS client] ************************************************
changed: [node2]

TASK [add Kubernetes apt-key] **************************************************
changed: [node2]

TASK [add Kubernetes' APT repository] ******************************************
changed: [node2]

TASK [Install kubectl] *********************************************************
changed: [node2]

TASK [Install kubelet] *********************************************************
changed: [node2]

TASK [Install kubeadm] *********************************************************
changed: [node2]

TASK [include vars] ************************************************************
ok: [node2]

TASK [getting hostonly ip address] *********************************************
changed: [node2]

TASK [debug] *******************************************************************
ok: [node2] => {
    "ip.stdout_lines[1].split(':')[1].split(' ')[0]": "172.16.20.13"
}

TASK [change 10-kubeadm.conf for v1.11 or later] *******************************
changed: [node2]

TASK [change 10-kubeadm.conf for v1.9 or v1.10] ********************************
skipping: [node2]

TASK [daemon-reload and restart kubelet] ***************************************
changed: [node2]

PLAY RECAP *********************************************************************
node2                      : ok=20   changed=15   unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

==> master: Importing base box 'ubuntu/xenial64'...
==> master: Matching MAC address for NAT networking...
==> master: Checking if box 'ubuntu/xenial64' version '20190613.1.0' is up to date...
==> master: Setting the name of the VM: vagrant-kubernetes_master_1573712682152_54641
==> master: Fixed port collision for 22 => 2222. Now on port 2201.
==> master: Clearing any previously set network interfaces...
==> master: Preparing network interfaces based on configuration...
    master: Adapter 1: nat
    master: Adapter 2: hostonly
==> master: Forwarding ports...
    master: 22 (guest) => 2201 (host) (adapter 1)
==> master: Running 'pre-boot' VM customizations...
==> master: Booting VM...
==> master: Waiting for machine to boot. This may take a few minutes...
    master: SSH address: 127.0.0.1:2201
    master: SSH username: vagrant
    master: SSH auth method: private key
    master:
    master: Vagrant insecure key detected. Vagrant will automatically replace
    master: this with a newly generated keypair for better security.
    master:
    master: Inserting generated public key within guest...
    master: Removing insecure key from the guest if it's present...
    master: Key inserted! Disconnecting and reconnecting using new SSH key...
==> master: Machine booted and ready!
==> master: Checking for guest additions in VM...
    master: The guest additions on this VM do not match the installed version of
    master: VirtualBox! In most cases this is fine, but in rare cases it can
    master: prevent things such as shared folders from working properly. If you see
    master: shared folder errors, please make sure the guest additions within the
    master: virtual machine match the version of VirtualBox you have installed on
    master: your host and reload your VM.
    master:
    master: Guest Additions Version: 5.1.38
    master: VirtualBox Version: 6.0
==> master: Setting hostname...
==> master: Configuring and enabling network interfaces...
==> master: Mounting shared folders...
    master: /vagrant => C:/y/Vagrant/vagrant-kubernetes
==> master: Running provisioner: ansible_local...
    master: Installing Ansible...
Vagrant has automatically selected the compatibility mode '2.0'
according to the Ansible version installed (2.8.6).

Alternatively, the compatibility mode can be specified in your Vagrantfile:
https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode
    master: Running ansible-playbook...

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [master]

TASK [include vars] ************************************************************
ok: [master]

TASK [debug] *******************************************************************
ok: [master] => {
    "k8s_minor_ver": "14"
}

TASK [Add Docker GPG key] ******************************************************
changed: [master]

TASK [Add Docker APT repository] ***********************************************
changed: [master]

TASK [Install a list of packages] **********************************************
[WARNING]: Could not find aptitude. Using apt-get instead
changed: [master]

TASK [usermod -aG docker vagrant] **********************************************
changed: [master]

TASK [Set sysctl] **************************************************************
changed: [master]

TASK [Add GlusterFS Repository] ************************************************
changed: [master]

TASK [Install GlusterFS client] ************************************************
changed: [master]

TASK [add Kubernetes apt-key] **************************************************
changed: [master]

TASK [add Kubernetes' APT repository] ******************************************
changed: [master]

TASK [Install kubectl] *********************************************************
changed: [master]

TASK [Install kubelet] *********************************************************
changed: [master]

TASK [Install kubeadm] *********************************************************
changed: [master]

TASK [include vars] ************************************************************
ok: [master]

TASK [getting hostonly ip address] *********************************************
changed: [master]

TASK [debug] *******************************************************************
ok: [master] => {
    "ip.stdout_lines[1].split(':')[1].split(' ')[0]": "172.16.20.11"
}

TASK [change 10-kubeadm.conf for v1.11 or later] *******************************
changed: [master]

TASK [change 10-kubeadm.conf for v1.9 or v1.10] ********************************
skipping: [master]

TASK [daemon-reload and restart kubelet] ***************************************
changed: [master]

PLAY RECAP *********************************************************************
master                     : ok=20   changed=15   unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

==> master: Running provisioner: ansible_local...
Vagrant has automatically selected the compatibility mode '2.0'
according to the Ansible version installed (2.8.6).

Alternatively, the compatibility mode can be specified in your Vagrantfile:
https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode
    master: Running ansible-playbook...

PLAY [master] ******************************************************************

TASK [Gathering Facts] *********************************************************
ok: [master]

TASK [include k8s vartsion] ****************************************************
ok: [master]

TASK [kubeadm reset v1.11 or later] ********************************************
changed: [master]

TASK [kubeadm reset v1.10 or eary] *********************************************
skipping: [master]

TASK [getting hostonly ip address] *********************************************
changed: [master]

TASK [kubeadm init] ************************************************************
changed: [master]

TASK [file] ********************************************************************
ok: [master]

TASK [kubeadm < 1.13] **********************************************************
skipping: [master]

TASK [kubeadm >= 1.14] *********************************************************
changed: [master]

TASK [change config.yaml] ******************************************************
ok: [master]

TASK [daemon-reload and restart kubelet] ***************************************
changed: [master]

TASK [mkdir kubeconfig] ********************************************************
changed: [master]

TASK [mkdir /vagrant/kubeconfig] ***********************************************
changed: [master]

TASK [chmod admin.conf] ********************************************************
changed: [master]

TASK [copy config to home dir] *************************************************
changed: [master]

TASK [copy config to host dir] *************************************************
changed: [master]

TASK [download Manifest of Flannel for k8s ver <=11] ***************************
skipping: [master]

TASK [download Manifest of Flannel for k8s ver > 11] ***************************
changed: [master]

TASK [add a new line] **********************************************************
changed: [master]

TASK [Deploy Flannel] **********************************************************
changed: [master]

PLAY RECAP *********************************************************************
master                     : ok=17   changed=13   unreachable=0    failed=0    skipped=3    rescued=0    ignored=0

==> master: Running provisioner: ansible_local...
Vagrant has automatically selected the compatibility mode '2.0'
according to the Ansible version installed (2.8.6).

Alternatively, the compatibility mode can be specified in your Vagrantfile:
https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode
    master: Running ansible-playbook...

PLAY [nodes] *******************************************************************

TASK [Gathering Facts] *********************************************************
ok: [node2]
ok: [node1]

TASK [include k8s version] *****************************************************
ok: [node1]
ok: [node2]

TASK [include kubeadm join token etc] ******************************************
ok: [node1]
ok: [node2]

TASK [kubeadm reset v1.11 or later] ********************************************
changed: [node2]
changed: [node1]

TASK [kubeadm reset v1.10 or eary] *********************************************
skipping: [node1]
skipping: [node2]

TASK [kubeadm join] ************************************************************
changed: [node2]
changed: [node1]

TASK [change config.yaml] ******************************************************
ok: [node2]
ok: [node1]

TASK [daemon-reload and restart kubelet] ***************************************
changed: [node1]
changed: [node2]

PLAY RECAP *********************************************************************
node1                      : ok=7    changed=3    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
node2                      : ok=7    changed=3    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0


c:\y\Vagrant\vagrant-kubernetes>vagrant status
Current machine states:

node1                     running (virtualbox)
node2                     running (virtualbox)
master                    running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

約20分弱くらいで完了しました。
ログを見た感じ、やはりkubeadmを使って構成しているようです。

確認

マスターノードに接続して各種状況確認。

sshでマスターノードに接続

c:\y\Vagrant\vagrant-kubernetes>vagrant ssh master
Welcome to Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-150-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

102 packages can be updated.
43 updates are security updates.

New release '18.04.3 LTS' available.
Welcome to Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-150-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

102 packages can be updated.
43 updates are security updates.

New release '18.04.3 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

バージョン

vagrant@master:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.9", GitCommit:"500f5aba80d71253cc01ac6a8622b8377f4a7ef9", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:04Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

cluster-info

vagrant@master:~$ kubectl cluster-info
Kubernetes master is running at https://172.16.20.11:6443
KubeDNS is running at https://172.16.20.11:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

node

vagrant@master:~$ kubectl get node -o wide
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
master   Ready    master   66m   v1.14.3   172.16.20.11   <none>        Ubuntu 16.04.6 LTS   4.4.0-150-generic   docker://18.6.1
node1    Ready    <none>   65m   v1.14.3   172.16.20.12   <none>        Ubuntu 16.04.6 LTS   4.4.0-150-generic   docker://18.6.1
node2    Ready    <none>   66m   v1.14.3   172.16.20.13   <none>        Ubuntu 16.04.6 LTS   4.4.0-150-generic   docker://18.6.1

namespace

vagrant@master:~$ kubectl get namespace
NAME              STATUS   AGE
default           Active   4d18h
kube-node-lease   Active   4d18h
kube-public       Active   4d18h
kube-system       Active   4d18h

操作例

レプリカセットの操作

レプリカセットはPodの複製を管理するためのリソースです。
レプリカセットとPodの関連は以下のイメージです。

Podの数を動的に制御することが可能になります(スケール機能)。

作成

以下を少しカスタマイズしてマニフェストファイルを作成してみます。
https://github.com/takara9/codes_for_lessons/blob/master/step08/deployment1.yml
上の定義のKind: DeploymentをReplicaSetに変えただけです。

replicaset.yml

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: web-deploy
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:1.16

template以下は、Podの定義と共通のものが使えます。

補足: Labelについて
参考:ラベル(Labels)とセレクター(Selectors)
レプリカセットとPodの紐づけはLabelによって行われるようです。Podの定義となるTemplateでは、labelsという項目でラベル付けを行います。Replicaset(or後続のDeployment)のSpecでは、selectorで管理対象となるPodを指定しますが、上の例では、matchLabelsという項目で、そのlabelにマッチするPodを管理対象としますよ、という指定になっています。
labelでは、上のリンクのように、keyとvalueのペアを使うことになりますが、keyもvalueも任意の文字列が使えますが、慣例的に、keyとしてはアプリケーションの種類を表す"app"や、層の種類を表す"tier"などが使われるようです。
上の例では、app: webというラベルが1つしか付いていませんが、複数ラベルを指定することもできます。

作成

vagrant@master:~/mytest01_replicaset$ kubectl apply -f replicaset.yml
replicaset.apps/web-deploy created

確認

vagrant@master:~/mytest01_replicaset$ kubectl get all -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
pod/web-deploy-bpqnq   1/1     Running   0          65s   10.244.2.2   node1   <none>           <none>
pod/web-deploy-s2z2q   1/1     Running   0          65s   10.244.2.3   node1   <none>           <none>
pod/web-deploy-vdqcp   1/1     Running   0          65s   10.244.1.2   node2   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE     SELECTOR
service/kubernetes   ClusterIP   10.32.0.1    <none>        443/TCP   4d19h   <none>

NAME                         DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES       SELECTOR
replicaset.apps/web-deploy   3         3         3       65s   nginx        nginx:1.16   app=web

レプリカセットというリソースと、そこから派生してPodが3つ作成されているのが分かります。レプリカセットとPodは階層構造になっており、ネーミングからもその階層が分かります。

レプリカセット
replicaset.apps/web-deploy

Pod
pod/web-deploy-bpqnq
pod/web-deploy-s2z2q
pod/web-deploy-vdqcp

3つのPodは、node1上で2つ、node2上で1つ、というように、きちんとクラスター上に分散されて稼働していますね。

※スケール機能については後続のデプロイメントでも共通なので、そちらで動作検証するので、ここで作成した定義は削除しておきます。

vagrant@master:~/mytest01_replicaset$ kubectl delete -f replicaset.yml
replicaset.apps "web-deploy" deleted

デプロイメントの操作

レプリカセットのさらに上位の、デプロイメントというリソースがあります。

デプロイメントでは、新しいバージョンのコンテナのリリースを管理するための仕組みが提供されています(ロールアウト機能)。

実運用では、レプリカセットを直接使うことは少なく、デプロイメントの単位で運用することがほとんどのようです。

作成

こちらをそのまま使ってデプロイメントを作成してみます。
https://github.com/takara9/codes_for_lessons/blob/master/step08/deployment1.yml

上で試したレプリカセットの定義と、kind: Deploymentの部分のみが異なります。

作成

vagrant@master:~/codes_for_lessons/step08$ kubectl apply -f deployment1.yml
deployment.apps/web-deploy created

確認

vagrant@master:~/codes_for_lessons/step08$ kubectl get all -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
pod/web-deploy-7c74bc9bb8-49lv9   1/1     Running   0          23s   10.244.1.3   node2   <none>           <none>
pod/web-deploy-7c74bc9bb8-5czxf   1/1     Running   0          23s   10.244.2.4   node1   <none>           <none>
pod/web-deploy-7c74bc9bb8-sh22g   1/1     Running   0          23s   10.244.1.4   node2   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE     SELECTOR
service/kubernetes   ClusterIP   10.32.0.1    <none>        443/TCP   4d19h   <none>

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES       SELECTOR
deployment.apps/web-deploy   3/3     3            3           23s   nginx        nginx:1.16   app=web

NAME                                    DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES       SELECTOR
replicaset.apps/web-deploy-7c74bc9bb8   3         3         3       23s   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8

デプロイメントは、レプリカセットというリソースとPodと合わせて使用されます。デプロイメントを作成したことで、レプリカの数をコントロースするためにレプリカセットと、管理対象のPodがそれぞれ作成されていることが分かります。
それぞれに付けられている名前で、階層構造が分かります。

デプロイメント
deployment.apps/web-deploy

レプリカセット
replicaset.apps/web-deploy-7c74bc9bb8

Pod
pod/web-deploy-7c74bc9bb8-49lv9
pod/web-deploy-7c74bc9bb8-5czxf
pod/web-deploy-7c74bc9bb8-sh22g

スケール機能

マニフェストファイルのreplicasの数を 3=>10に変更してみます。
変更後のymlは以下。
https://github.com/takara9/codes_for_lessons/blob/master/step08/deployment2.yml

vagrant@master:~/codes_for_lessons/step08$ kubectl apply -f deployment2.yml
deployment.apps/web-deploy configured

vagrant@master:~/codes_for_lessons/step08$ kubectl get all -o wide
NAME                              READY   STATUS    RESTARTS   AGE     IP           NODE    NOMINATED NODE   READINESS GATES
pod/web-deploy-7c74bc9bb8-49lv9   1/1     Running   0          7m10s   10.244.1.3   node2   <none>           <none>
pod/web-deploy-7c74bc9bb8-5czxf   1/1     Running   0          7m10s   10.244.2.4   node1   <none>           <none>
pod/web-deploy-7c74bc9bb8-b2kmg   1/1     Running   0          22s     10.244.2.6   node1   <none>           <none>
pod/web-deploy-7c74bc9bb8-c7r4f   1/1     Running   0          22s     10.244.2.8   node1   <none>           <none>
pod/web-deploy-7c74bc9bb8-l2hfq   1/1     Running   0          22s     10.244.2.7   node1   <none>           <none>
pod/web-deploy-7c74bc9bb8-nfbh8   1/1     Running   0          22s     10.244.1.6   node2   <none>           <none>
pod/web-deploy-7c74bc9bb8-sh22g   1/1     Running   0          7m10s   10.244.1.4   node2   <none>           <none>
pod/web-deploy-7c74bc9bb8-sjflb   1/1     Running   0          22s     10.244.1.7   node2   <none>           <none>
pod/web-deploy-7c74bc9bb8-v74q2   1/1     Running   0          22s     10.244.1.5   node2   <none>           <none>
pod/web-deploy-7c74bc9bb8-w64wq   1/1     Running   0          22s     10.244.2.5   node1   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE     SELECTOR
service/kubernetes   ClusterIP   10.32.0.1    <none>        443/TCP   4d20h   <none>

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES       SELECTOR
deployment.apps/web-deploy   10/10   10           10          7m10s   nginx        nginx:1.16   app=web

NAME                                    DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES       SELECTOR
replicaset.apps/web-deploy-7c74bc9bb8   10        10        10      7m10s   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8

Podの数が3=>10になった！

次はコマンドでレプリカ数を5に変更してみます。

vagrant@master:~/codes_for_lessons/step08$ kubectl scale --replicas=5 deployment.apps/web-deploy
deployment.apps/web-deploy scaled

vagrant@master:~/codes_for_lessons/step08$ kubectl get all -o wide
NAME                              READY   STATUS    RESTARTS   AGE     IP           NODE    NOMINATED NODE   READINESS GATES
pod/web-deploy-7c74bc9bb8-49lv9   1/1     Running   0          9m19s   10.244.1.3   node2   <none>           <none>
pod/web-deploy-7c74bc9bb8-5czxf   1/1     Running   0          9m19s   10.244.2.4   node1   <none>           <none>
pod/web-deploy-7c74bc9bb8-sh22g   1/1     Running   0          9m19s   10.244.1.4   node2   <none>           <none>
pod/web-deploy-7c74bc9bb8-v74q2   1/1     Running   0          2m31s   10.244.1.5   node2   <none>           <none>
pod/web-deploy-7c74bc9bb8-w64wq   1/1     Running   0          2m31s   10.244.2.5   node1   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE     SELECTOR
service/kubernetes   ClusterIP   10.32.0.1    <none>        443/TCP   4d20h   <none>

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES       SELECTOR
deployment.apps/web-deploy   5/5     5            5           9m19s   nginx        nginx:1.16   app=web

NAME                                    DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES       SELECTOR
replicaset.apps/web-deploy-7c74bc9bb8   5         5         5       9m19s   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8

Pod数が10=>5になりました。

ロールアウト機能

Pod内で稼働するイメージのバージョンを変更する場合など、サービスを稼働させながら新しいPodに順次切り替える機能です。

Pod数10にもどして、デプロイメントの詳細を見てみます。

vagrant@master:~/codes_for_lessons/step08$ kubectl describe deployment web-deploy
Name:                   web-deploy
Namespace:              default
CreationTimestamp:      Tue, 19 Nov 2019 02:28:26 +0000
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"web-deploy","namespace":"default"},"spec":{"replicas":10,...
Selector:               app=web
Replicas:               10 desired | 10 updated | 10 total | 10 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=web
  Containers:
   nginx:
    Image:        nginx:1.16
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   web-deploy-7c74bc9bb8 (10/10 replicas created)
Events:
  Type    Reason             Age                  From                   Message
  ----    ------             ----                 ----                   -------
  Normal  ScalingReplicaSet  10m                  deployment-controller  Scaled up replica set web-deploy-7c74bc9bb8 to 3
  Normal  ScalingReplicaSet  83s                  deployment-controller  Scaled down replica set web-deploy-7c74bc9bb8 to 5
  Normal  ScalingReplicaSet  16s (x2 over 3m36s)  deployment-controller  Scaled up replica set web-deploy-7c74bc9bb8 to 10

```RollingUpdateStrategy: 25% max unavailable, 25% max surge``: この設定は、Podを順次切り替えていく際に、最大25%のPod停止を許容し、最大25%の稼働数超過を許容する、という意味の設定です。
Replica数10の場合...
最大停止数: 10x0.25=2.5 =>切り捨てて2個まで停止を許容する
最大超過数: 10x0.25=2.5 =>切り上げて3個まで超過して稼働を許容する
つまり、10-2=8個のPodは最低限維持してサービスを継続しつつ、10+3=13個までPodを並行稼働させながら、順次新しいPodに入れ替えていく、ということを勝手にやってくれるということになります。
参考: Deployment V1 apps

nginxのイメージを1.16=>1.17に変更したマニフェストファイルを用意します。(以下をそのまま使います)
https://github.com/takara9/codes_for_lessons/blob/master/step08/deployment3.yml

このマニフェストファイルを適用して構成変更を行いますが、状況確認用に、kubectl get replicaset -o wide -wコマンドを別シェルで実行しておきます。-wオプションは、変更が生じたらそれをwatchして表示してくれるオプションです。

ロールアウト実行

vagrant@master:~/codes_for_lessons/step08$ kubectl apply -f deployment3.yml
deployment.apps/web-deploy configured

入れ替わりの経緯が以下のように確認できます。

vagrant@master:~$ kubectl get replicaset -o wide -w
NAME                    DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES       SELECTOR
web-deploy-7c74bc9bb8   10        10        10      12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   3         0         0       0s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   8         10        10      12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   5         0         0       0s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   8         10        10      12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   5         0         0       0s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   8         8         8       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   5         3         0       0s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   5         5         0       0s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   5         5         1       9s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   7         8         8       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   7         8         8       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   6         5         1       9s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   7         7         7       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   6         5         1       9s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   6         6         1       9s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   6         6         2       9s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   6         7         7       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   7         6         2       9s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   6         7         7       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   7         6         2       9s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   7         7         2       9s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   6         6         6       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   7         7         3       10s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   5         6         6       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   8         7         3       10s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   5         6         6       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   8         7         3       10s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   8         8         3       10s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   5         5         5       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   8         8         4       10s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   4         5         5       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   4         5         5       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   9         8         4       10s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   4         4         4       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   9         8         4       10s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   9         9         4       10s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   9         9         5       11s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   3         4         4       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   3         4         4       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   10        9         5       11s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   3         3         3       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   10        9         5       11s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   10        10        5       11s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   10        10        6       12s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   2         3         3       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   2         3         3       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   2         2         2       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   10        10        7       12s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   1         2         2       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   1         2         2       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   1         1         1       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   10        10        8       13s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   0         1         1       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   0         1         1       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   0         0         0       12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   10        10        9       14s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   10        10        10      15s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf

例えば、コマンド実行直後に、新しいレプリカセット(web-deploy-6d6465dbbf)がDESIRED:3 で作成され、既存のレプリカセット(web-deploy-7c74bc9bb8)がDESIRED:8となっています。

web-deploy-6d6465dbbf   3         0         0       0s    nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   8         10        10      12m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8

このように、最低限8個のPodを維持しつつ、かつ最大13個までの並行度を保ちながら、新しいバージョンのものに順次置き換わっていきます。
最終的に旧レプリカセットのPod数は0となり、新レプリカセットのPod数が10になりロールアウトが完了しました。

ロールアウトの履歴を確認することもできます。

vagrant@master:~$ kubectl rollout history deployment.apps/web-deploy
deployment.apps/web-deploy
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

マニフェストファイルやkubectlコマンドの--recordフラグなどで、CHANGE-CAUSEの記述を指定することができるようですが、今回特に指定していないので<none>になってます。

REVISIONの番号指定で詳細を確認できます。

vagrant@master:~$ kubectl rollout history deployment.apps/web-deploy --revision=1
deployment.apps/web-deploy with revision #1
Pod Template:
  Labels:       app=web
        pod-template-hash=7c74bc9bb8
  Containers:
   nginx:
    Image:      nginx:1.16
    Port:       <none>
    Host Port:  <none>
    Environment:        <none>
    Mounts:     <none>
  Volumes:      <none>

vagrant@master:~$ kubectl rollout history deployment.apps/web-deploy --revision=2
deployment.apps/web-deploy with revision #2
Pod Template:
  Labels:       app=web
        pod-template-hash=6d6465dbbf
  Containers:
   nginx:
    Image:      nginx:1.17
    Port:       <none>
    Host Port:  <none>
    Environment:        <none>
    Mounts:     <none>
  Volumes:      <none>

ロールバック機能

ロールアウトした変更を元に戻す機能です。

ロールアウト前(1つ前)の状態に戻します。(--to-revision= 指定で特定のrevisionに戻すこともできるようです。)

vagrant@master:~/codes_for_lessons/step08$ kubectl rollout undo deployment.apps/web-deploy
deployment.apps/web-deploy rolled back

先の例と同様、変更状況をWatchしてみると

vagrant@master:~$ kubectl get replicaset -o wide -w
NAME                    DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES       SELECTOR
web-deploy-6d6465dbbf   10        10        10      4m29s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   0         0         0       16m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   0         0         0       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   3         0         0       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   8         10        10      5m11s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   3         0         0       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   3         3         0       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   8         10        10      5m11s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   5         3         0       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   8         8         8       5m11s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   5         3         0       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   5         5         0       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   5         5         1       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   7         8         8       5m12s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   6         5         1       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   7         8         8       5m12s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   7         7         7       5m12s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   6         5         1       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   6         6         1       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   6         6         2       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   6         7         7       5m13s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   7         6         2       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   6         7         7       5m13s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   7         6         3       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   6         6         6       5m13s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   7         6         3       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   7         7         3       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   5         6         6       5m13s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   8         7         3       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   5         6         6       5m13s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   8         7         3       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   5         5         5       5m13s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   8         8         3       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   8         8         4       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   4         5         5       5m13s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   9         8         4       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   4         5         5       5m13s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   4         4         4       5m13s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   9         8         4       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   9         9         4       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   9         9         5       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   3         4         4       5m13s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   10        9         5       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   3         4         4       5m13s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   10        9         5       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-7c74bc9bb8   10        10        5       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   3         3         3       5m13s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   10        10        6       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   2         3         3       5m14s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   2         3         3       5m14s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   2         2         2       5m14s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   10        10        7       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   1         2         2       5m16s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   1         2         2       5m16s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   1         1         1       5m16s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   10        10        8       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   0         1         1       5m17s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   10        10        9       17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8
web-deploy-6d6465dbbf   0         1         1       5m17s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-6d6465dbbf   0         0         0       5m17s   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
web-deploy-7c74bc9bb8   10        10        10      17m     nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8

nginx:1.16を使ったPodが10個稼働した状態になりました。

rolloutの履歴みると、revision:1がなくなってrevision3が増えてますね。

vagrant@master:~/codes_for_lessons/step08$ kubectl rollout history deployment.apps/web-deploy
deployment.apps/web-deploy
REVISION  CHANGE-CAUSE
2         <none>
3         <none>

vagrant@master:~/codes_for_lessons/step08$ kubectl rollout history deployment.apps/web-deploy --revision=3
deployment.apps/web-deploy with revision #3
Pod Template:
  Labels:       app=web
        pod-template-hash=7c74bc9bb8
  Containers:
   nginx:
    Image:      nginx:1.16
    Port:       <none>
    Host Port:  <none>
    Environment:        <none>
    Mounts:     <none>
  Volumes:      <none>

自己回復機能

コンテナレベルの障害は、ポッドが持っている機能で、RestartPolicyに従って自動再起動などの処理が行われることになりますが、ポッドレベルの障害(ノード障害など)の場合は、レプリカセットが持つ機能により、設定したレプリカ数を維持するようPodが別ノードで稼働することになります。

動作を比較するために、上のデプロイメントに加え、Pod単体の定義を1つ追加します(以下のマニフェストファイルを使用)。
https://github.com/takara9/codes_for_lessons/blob/master/step08/pod.yml

また、先に作成したデプロイメントのレプリカ数を5に変更しておきます。

vagrant@master:~/codes_for_lessons/step08$ kubectl apply -f pod.yml
pod/test1 created
vagrant@master:~/codes_for_lessons/step08$ kubectl scale --replicas=5 deployment.apps/web-deploy
deployment.apps/web-deploy scaled

状況としてはこんな感じ。

vagrant@master:~/codes_for_lessons/step08$ kubectl get all -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
pod/test1                         1/1     Running   0          66s   10.244.2.22   node1   <none>           <none>
pod/web-deploy-7c74bc9bb8-ff75w   1/1     Running   0          62m   10.244.2.19   node1   <none>           <none>
pod/web-deploy-7c74bc9bb8-r2lpz   1/1     Running   0          62m   10.244.2.18   node1   <none>           <none>
pod/web-deploy-7c74bc9bb8-s864t   1/1     Running   0          62m   10.244.1.16   node2   <none>           <none>
pod/web-deploy-7c74bc9bb8-vgbr6   1/1     Running   0          62m   10.244.2.17   node1   <none>           <none>
pod/web-deploy-7c74bc9bb8-xlh2v   1/1     Running   0          62m   10.244.1.15   node2   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE     SELECTOR
service/kubernetes   ClusterIP   10.32.0.1    <none>        443/TCP   4d21h   <none>

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES       SELECTOR
deployment.apps/web-deploy   5/5     5            5           79m   nginx        nginx:1.16   app=web

NAME                                    DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES       SELECTOR
replicaset.apps/web-deploy-6d6465dbbf   0         0         0       67m   nginx        nginx:1.17   app=web,pod-template-hash=6d6465dbbf
replicaset.apps/web-deploy-7c74bc9bb8   5         5         5       79m   nginx        nginx:1.16   app=web,pod-template-hash=7c74bc9bb8

Pod単体のpod/test1がnode1で稼働し、デプロイメントから派生して稼働したpod/web-deploy-xxxは5個のレプリカが稼働していますが、node1上で3つ、node2上で2つ稼働している状態です。

先の例と同じようにPodの状況をウォッチしておきながら、test1が稼働しているnode1を停止させてみます。

c:\y\Vagrant\vagrant-kubernetes>vagrant halt node1
==> node1: Attempting graceful shutdown of VM...

以下、node1停止時点の状態です。

vagrant@master:~$ kubectl get pod -o wide -w
NAME                          READY   STATUS    RESTARTS   AGE     IP            NODE    NOMINATED NODE   READINESS GATES
test1                         1/1     Running   0          6m48s   10.244.2.22   node1   <none>           <none>
web-deploy-7c74bc9bb8-ff75w   1/1     Running   0          67m     10.244.2.19   node1   <none>           <none>
web-deploy-7c74bc9bb8-r2lpz   1/1     Running   0          67m     10.244.2.18   node1   <none>           <none>
web-deploy-7c74bc9bb8-s864t   1/1     Running   0          67m     10.244.1.16   node2   <none>           <none>
web-deploy-7c74bc9bb8-vgbr6   1/1     Running   0          67m     10.244.2.17   node1   <none>           <none>
web-deploy-7c74bc9bb8-xlh2v   1/1     Running   0          67m     10.244.1.15   node2   <none>           <none>

5分後にノード障害を検知して...

web-deploy-7c74bc9bb8-vgbr6   1/1     Terminating   0          71m     10.244.2.17   node1   <none>           <none>
web-deploy-7c74bc9bb8-ff75w   1/1     Terminating   0          71m     10.244.2.19   node1   <none>           <none>
test1                         1/1     Terminating   0          10m     10.244.2.22   node1   <none>           <none>
web-deploy-7c74bc9bb8-r2lpz   1/1     Terminating   0          71m     10.244.2.18   node1   <none>           <none>
web-deploy-7c74bc9bb8-fsjg4   0/1     Pending       0          0s      <none>        <none>   <none>           <none>
web-deploy-7c74bc9bb8-fsjg4   0/1     Pending       0          0s      <none>        node2    <none>           <none>
web-deploy-7c74bc9bb8-chkft   0/1     Pending       0          0s      <none>        <none>   <none>           <none>
web-deploy-7c74bc9bb8-chkft   0/1     Pending       0          0s      <none>        node2    <none>           <none>
web-deploy-7c74bc9bb8-c87d9   0/1     Pending       0          0s      <none>        <none>   <none>           <none>
web-deploy-7c74bc9bb8-fsjg4   0/1     ContainerCreating   0          0s      <none>        node2    <none>           <none>
web-deploy-7c74bc9bb8-c87d9   0/1     Pending             0          0s      <none>        node2    <none>           <none>
web-deploy-7c74bc9bb8-chkft   0/1     ContainerCreating   0          1s      <none>        node2    <none>           <none>
web-deploy-7c74bc9bb8-c87d9   0/1     ContainerCreating   0          1s      <none>        node2    <none>           <none>
web-deploy-7c74bc9bb8-fsjg4   1/1     Running             0          2s      10.244.1.20   node2    <none>           <none>
web-deploy-7c74bc9bb8-c87d9   1/1     Running             0          3s      10.244.1.21   node2    <none>           <none>
web-deploy-7c74bc9bb8-chkft   1/1     Running             0          3s      10.244.1.22   node2    <none>           <none>

結果、こういう状態になっています。

vagrant@master:~$ kubectl get pod -o wide
NAME                          READY   STATUS        RESTARTS   AGE     IP            NODE    NOMINATED NODE   READINESS GATES
test1                         1/1     Terminating   0          14m     10.244.2.22   node1   <none>           <none>
web-deploy-7c74bc9bb8-c87d9   1/1     Running       0          3m46s   10.244.1.21   node2   <none>           <none>
web-deploy-7c74bc9bb8-chkft   1/1     Running       0          3m46s   10.244.1.22   node2   <none>           <none>
web-deploy-7c74bc9bb8-ff75w   1/1     Terminating   0          75m     10.244.2.19   node1   <none>           <none>
web-deploy-7c74bc9bb8-fsjg4   1/1     Running       0          3m46s   10.244.1.20   node2   <none>           <none>
web-deploy-7c74bc9bb8-r2lpz   1/1     Terminating   0          75m     10.244.2.18   node1   <none>           <none>
web-deploy-7c74bc9bb8-s864t   1/1     Running       0          75m     10.244.1.16   node2   <none>           <none>
web-deploy-7c74bc9bb8-vgbr6   1/1     Terminating   0          75m     10.244.2.17   node1   <none>           <none>
web-deploy-7c74bc9bb8-xlh2v   1/1     Running       0          75m     10.244.1.15   node2   <none>           <none>

node1で稼働していたtest1と、3つのweb-deploy-xxxがTerminatingの状態になりました。
一方で、3つのweb-deploy-xxxがnode2で新たにRunningとなっています。
つまり、web-deploy-xxxがnode2で5つ稼働している状態となりました。(test1は復旧しない)

node1を再起動してみます。

c:\y\Vagrant\vagrant-kubernetes>vagrant up node1
Bringing machine 'node1' up with 'virtualbox' provider...
==> node1: Checking if box 'ubuntu/xenial64' version '20190613.1.0' is up to date...
==> node1: A newer version of the box 'ubuntu/xenial64' for provider 'virtualbox' is
==> node1: available! You currently have version '20190613.1.0'. The latest is version
==> node1: '20191114.0.0'. Run `vagrant box update` to update.
==> node1: Clearing any previously set forwarded ports...
==> node1: Clearing any previously set network interfaces...
==> node1: Preparing network interfaces based on configuration...
    node1: Adapter 1: nat
    node1: Adapter 2: hostonly
==> node1: Forwarding ports...
    node1: 22 (guest) => 2222 (host) (adapter 1)
==> node1: Running 'pre-boot' VM customizations...
==> node1: Booting VM...
==> node1: Waiting for machine to boot. This may take a few minutes...
    node1: SSH address: 127.0.0.1:2222
    node1: SSH username: vagrant
    node1: SSH auth method: private key
==> node1: Machine booted and ready!
==> node1: Checking for guest additions in VM...
    node1: The guest additions on this VM do not match the installed version of
    node1: VirtualBox! In most cases this is fine, but in rare cases it can
    node1: prevent things such as shared folders from working properly. If you see
    node1: shared folder errors, please make sure the guest additions within the
    node1: virtual machine match the version of VirtualBox you have installed on
    node1: your host and reload your VM.
    node1:
    node1: Guest Additions Version: 5.1.38
    node1: VirtualBox Version: 6.0
==> node1: Setting hostname...
==> node1: Configuring and enabling network interfaces...
==> node1: Mounting shared folders...
    node1: /vagrant => C:/y/Vagrant/vagrant-kubernetes
==> node1: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> node1: flag to force provisioning. Provisioners marked to run always will still run.

node1起動後のPodの状態を見てみると...

vagrant@master:~/codes_for_lessons/step08$ kubectl get pod -o wide
NAME                          READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
web-deploy-7c74bc9bb8-c87d9   1/1     Running   0          10m   10.244.1.21   node2   <none>           <none>
web-deploy-7c74bc9bb8-chkft   1/1     Running   0          10m   10.244.1.22   node2   <none>           <none>
web-deploy-7c74bc9bb8-fsjg4   1/1     Running   0          10m   10.244.1.20   node2   <none>           <none>
web-deploy-7c74bc9bb8-s864t   1/1     Running   0          82m   10.244.1.16   node2   <none>           <none>
web-deploy-7c74bc9bb8-xlh2v   1/1     Running   0          82m   10.244.1.15   node2   <none>           <none>

TerminatingのステータスとなっていたPodがクリアされました。
test1のPodも削除された状態です(node1障害によりPod自体が削除され、それを管理してくれる人がいないので、復旧はされない)。

補足: 障害検知/Podの別ノードへの退避時間について
ノード障害を検知するのに5分って結構かかるなと思って、これが何に依存するのか調べてみると、以下に解説してくれている人がいました。Taint Based Evictionという機能の挙動に依存するようです。
参考: Kubernetesのノード障害時の挙動とノード復帰後の再スケジューリング

公式ドキュメントとしてはこの辺ですかね。
Taints and Tolerations

Note that Kubernetes automatically adds a toleration for node.kubernetes.io/not-ready with tolerationSeconds=300 unless the pod configuration provided by the user already has a toleration for node.kubernetes.io/not-ready. Likewise it adds a toleration for node.kubernetes.io/unreachable with tolerationSeconds=300 unless the pod configuration provided by the user already has a toleration for node.kubernetes.io/unreachable.
These automatically-added tolerations ensure that the default pod behavior of remaining bound for 5 minutes after one of these problems is detected is maintained. The two default tolerations are added by the DefaultTolerationSeconds admission controller.

また、node1復旧後、node2に片寄されたPodは片寄されたままなので(node2で動き続ける)、次にnode2に障害があったら、完全に5分間サービスが停止してしまうことになります。
node1復旧後はnode1/node2にうまく分散配置しなおして欲しいのですが、どうも勝手にそうはなってくれないようです。

以下のような記事がありましたが、どうやら結構泥臭い方法で再配置のオペレーションをしないといけなさそうです。
参考 K8s Node障害時の振る舞いについての検証記録 - 復旧後にポッドの配置をリバランスするには？
うーん、もう少しスマートな方法はないのだろうか？？？

サービスの操作

サービスは、Podの集合に対しての経路設定や外部のエンドポイントの名前解決などを行ってくれるリソースです。
複数のPodに対する代表アドレスを設定したり、それをk8sクラスターの外に公開したり、ロードバランサーとの連携を行ったりする時に使用します。

以下の4つのタイプのサービスが定義できるようです。

ClusterIP: デフォルトのタイプ。クラスタ内でPodの集合(レプリカセットで管理されているものなど)に対して代表IPアドレスを割り当てて負荷分散させることができます。(外部からのアクセスは不可)
NodePort: ClusterIPに加え、外部に対してポートを公開します(各ノードのIPアドレス+ポートでアクセス可能となる)。
LoadBalancer: LoadBalancerと連携して外部からアクセス可能にするもの。(ローカルのKubernetes環境では利用不可。IKSやGKEなどクラウドサービス環境で利用。)
ExternalName: Pod上のサービスから外部のサービスを利用する際の名前解決を行うもの。

type: ClusterIP (マルチノード環境)

一旦上で作成したデプロイメントは削除して、サービスのテスト用に、以下のマニフェストファイルを適用し、デプロイメントとサービスを作成してみます。
https://github.com/takara9/codes_for_lessons/blob/master/step09/deploy.yml
https://github.com/takara9/codes_for_lessons/blob/master/step09/svc.yml

作成

vagrant@master:~/codes_for_lessons/step09$ kubectl apply -f deploy.yml
deployment.apps/web-deploy created

vagrant@master:~/codes_for_lessons/step09$ kubectl apply -f svc.yml
service/web-service created

確認

vagrant@master:~$ kubectl get all -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
pod/web-deploy-6b5f67b747-5qr49   1/1     Running   0          40m   10.244.2.3   node2   <none>           <none>
pod/web-deploy-6b5f67b747-8cjw4   1/1     Running   0          40m   10.244.1.4   node1   <none>           <none>
pod/web-deploy-6b5f67b747-w8qbm   1/1     Running   0          40m   10.244.2.2   node2   <none>           <none>

NAME                  TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes    ClusterIP   10.32.0.1    <none>        443/TCP   61m   <none>
service/web-service   ClusterIP   10.32.0.79   <none>        80/TCP    13s   app=web

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES         SELECTOR
deployment.apps/web-deploy   3/3     3            3           40m   nginx        nginx:latest   app=web

NAME                                    DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES         SELECTOR
replicaset.apps/web-deploy-6b5f67b747   3         3         3       40m   nginx        nginx:latest   app=web,pod-template-hash=6b5f67b747

この代表となるポートはクラスター内からしかアクセスできないため、テスト用に踏み台となるPodを作成して、そこからアクセスしてみます。

busyboxのPodを立ち上げてshを実行します。

vagrant@master:~$ kubectl run -it busybox --restart=Never --rm --image=busybox:latest sh
If you don't see a command prompt, try pressing enter.
/ #

この時、別シェルでPodの状況を見ると、このbusyboxのPodはnode1で上がってきているのが分かります。

vagrant@master:~$ kubectl get pod  -o wide
NAME                          READY   STATUS    RESTARTS   AGE    IP           NODE    NOMINATED NODE   READINESS GATES
busybox                       1/1     Running   0          2m7s   10.244.1.6   node1   <none>           <none>
web-deploy-6b5f67b747-5qr49   1/1     Running   0          43m    10.244.2.3   node2   <none>           <none>
web-deploy-6b5f67b747-8cjw4   1/1     Running   0          43m    10.244.1.4   node1   <none>           <none>
web-deploy-6b5f67b747-w8qbm   1/1     Running   0          43m    10.244.2.2   node2   <none>           <none>

busyboxのシェルで、サービスにアクセスしてみると...
んんん？？？うまくいく場合とエラーになる場合がある。

NGケース

/ # wget -q -O - http://web-service
wget: can't connect to remote host (10.32.0.79): No route to host

OKケース

/ # wget -q -O - http://web-service
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

どうも、busyboxのPodと同じノード(node1)で稼働しているPodのアドレスに対してはPingは通るが、node2上で稼働しているPodにはPingが通らない様子...

node2上のPodへのping=>NG

/ # ping -c 3 10.244.2.2
PING 10.244.2.2 (10.244.2.2): 56 data bytes

--- 10.244.2.2 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

node2上のPodへのping=>NG

/ # ping -c 3 10.244.2.3
PING 10.244.2.3 (10.244.2.3): 56 data bytes

--- 10.244.2.3 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

node1上のPodへのping=>OK

/ # ping -c 3 10.244.1.4
PING 10.244.1.4 (10.244.1.4): 56 data bytes
64 bytes from 10.244.1.4: seq=0 ttl=64 time=0.057 ms
64 bytes from 10.244.1.4: seq=1 ttl=64 time=0.048 ms
64 bytes from 10.244.1.4: seq=2 ttl=64 time=0.052 ms

--- 10.244.1.4 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.048/0.052/0.057 ms

うむむ...
ノードをまたがるPod間のネットワークについては、Kubernetesでその仕様は決まっているようですが、実装はKubernetes自体には含まれていないようです。様々な実装が提供されており、必要に応じてそのネットワーク実装を選択し、導入/構成しないといけないようです。
　参考:
　　Cluster Networking
　　DockerとKubernetesのPodのネットワーキングについてまとめました
　　オンプレミスKubernetesのネットワーク
で、今回使っている環境は、vagrantから呼ばれるAnsibleのplaybookを見ると、flannelと呼ばれるネットワーク実装の構成を行っているようです。
　参考: https://github.com/takara9/vagrant-kubernetes/blob/1.14/ansible-playbook/k8s_master.yml

namespace:kube-system のリソース見てみると、flannel関連のリソースが追加されているのが分かります。

vagrant@master:~/codes_for_lessons/step09$ kubectl -n kube-system get all -o wide
NAME                                 READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
pod/coredns-584795fc57-2q98h         1/1     Running   0          93m   10.244.1.2     node1    <none>           <none>
pod/coredns-584795fc57-5854n         1/1     Running   0          93m   10.244.1.3     node1    <none>           <none>
pod/etcd-master                      1/1     Running   0          93m   172.16.20.11   master   <none>           <none>
pod/kube-apiserver-master            1/1     Running   0          93m   172.16.20.11   master   <none>           <none>
pod/kube-controller-manager-master   1/1     Running   0          93m   172.16.20.11   master   <none>           <none>
pod/kube-flannel-ds-amd64-9hfp6      1/1     Running   0          93m   172.16.20.13   node2    <none>           <none>
pod/kube-flannel-ds-amd64-blbpx      1/1     Running   0          93m   172.16.20.12   node1    <none>           <none>
pod/kube-flannel-ds-amd64-s8r6n      1/1     Running   0          93m   172.16.20.11   master   <none>           <none>
pod/kube-proxy-7fgjh                 1/1     Running   0          93m   172.16.20.12   node1    <none>           <none>
pod/kube-proxy-bvj2r                 1/1     Running   0          93m   172.16.20.11   master   <none>           <none>
pod/kube-proxy-l44k9                 1/1     Running   0          93m   172.16.20.13   node2    <none>           <none>
pod/kube-scheduler-master            1/1     Running   0          93m   172.16.20.11   master   <none>           <none>

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
service/kube-dns   ClusterIP   10.32.0.10   <none>        53/UDP,53/TCP,9153/TCP   93m   k8s-app=kube-dns

NAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE   CONTAINERS     IMAGES                                   SELECTOR
daemonset.apps/kube-flannel-ds-amd64     3         3         3       3            3           beta.kubernetes.io/arch=amd64     93m   kube-flannel   quay.io/coreos/flannel:v0.10.0-amd64     app=flannel,tier=node
daemonset.apps/kube-flannel-ds-arm       0         0         0       0            0           beta.kubernetes.io/arch=arm       93m   kube-flannel   quay.io/coreos/flannel:v0.10.0-arm       app=flannel,tier=node
daemonset.apps/kube-flannel-ds-arm64     0         0         0       0            0           beta.kubernetes.io/arch=arm64     93m   kube-flannel   quay.io/coreos/flannel:v0.10.0-arm64     app=flannel,tier=node
daemonset.apps/kube-flannel-ds-ppc64le   0         0         0       0            0           beta.kubernetes.io/arch=ppc64le   93m   kube-flannel   quay.io/coreos/flannel:v0.10.0-ppc64le   app=flannel,tier=node
daemonset.apps/kube-flannel-ds-s390x     0         0         0       0            0           beta.kubernetes.io/arch=s390x     93m   kube-flannel   quay.io/coreos/flannel:v0.10.0-s390x     app=flannel,tier=node
daemonset.apps/kube-proxy                3         3         3       3            3           <none>                            93m   kube-proxy     k8s.gcr.io/kube-proxy:v1.14.9            k8s-app=kube-proxy

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                     SELECTOR
deployment.apps/coredns   2/2     2            2           93m   coredns      k8s.gcr.io/coredns:1.3.1   k8s-app=kube-dns

NAME                                 DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                     SELECTOR
replicaset.apps/coredns-584795fc57   2         2         2       93m   coredns      k8s.gcr.io/coredns:1.3.1   k8s-app=kube-dns,pod-template-hash=584795fc57

んー、この辺定義されていれば、ノードを跨ってもクラスター内のPod間では疎通できると思うんだが...
ということで、ネットワーク関連の問題判別実施。詳細は以下参照。
コンテナ型仮想化技術 Study06' / Kubernetesネットワーク問題判別

ということで、上の対応を実施した後、一旦リソース削除してやり直し。

vagrant@master:~/codes_for_lessons/step09$ kubectl apply -f deploy.yml
deployment.apps/web-deploy created

vagrant@master:~/codes_for_lessons/step09$ kubectl apply -f svc.yml
service/web-service created

確認

vagrant@master:~$ kubectl get all -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
pod/web-deploy-6b5f67b747-4bl7p   1/1     Running   0          50s   10.244.2.22   node2   <none>           <none>
pod/web-deploy-6b5f67b747-wfslh   1/1     Running   0          50s   10.244.2.23   node2   <none>           <none>
pod/web-deploy-6b5f67b747-x7g2k   1/1     Running   0          50s   10.244.1.34   node1   <none>           <none>

NAME                  TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes    ClusterIP   10.32.0.1     <none>        443/TCP   39d   <none>
service/web-service   ClusterIP   10.32.0.218   <none>        80/TCP    36s   app=web

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES         SELECTOR
deployment.apps/web-deploy   3/3     3            3           50s   nginx        nginx:latest   app=web

NAME                                    DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES         SELECTOR
replicaset.apps/web-deploy-6b5f67b747   3         3         3       50s   nginx        nginx:latest   app=web,pod-template-hash=6b5f67b747

テスト用に踏み台となるPodを作成して、そこからアクセスしてみます。
busyboxのPodを立ち上げてshを実行します。

vagrant@master:~$ kubectl run -it busybox --restart=Never --rm --image=busybox:latest sh
If you don't see a command prompt, try pressing enter.
/ #

この時、別シェルでPodの状況を見ると、このbusyboxのPodはnode1で上がってきているのが分かります。

vagrant@master:~$ kubectl get pod -o wide
NAME                          READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
busybox                       1/1     Running   0          13s   10.244.1.35   node1   <none>           <none>
web-deploy-6b5f67b747-4bl7p   1/1     Running   0          98s   10.244.2.22   node2   <none>           <none>
web-deploy-6b5f67b747-wfslh   1/1     Running   0          98s   10.244.2.23   node2   <none>           <none>
web-deploy-6b5f67b747-x7g2k   1/1     Running   0          98s   10.244.1.34   node1   <none>           <none>

busyboxのシェルで、サービスにアクセスしてみると...

/ # wget -q -O - http://web-service
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

サービス名の大文字をキーに環境変数を確認すると...

/ # env | grep WEB_SERVICE
WEB_SERVICE_SERVICE_PORT=80
WEB_SERVICE_PORT=tcp://10.32.0.218:80
WEB_SERVICE_PORT_80_TCP_ADDR=10.32.0.218
WEB_SERVICE_PORT_80_TCP_PORT=80
WEB_SERVICE_PORT_80_TCP_PROTO=tcp
WEB_SERVICE_PORT_80_TCP=tcp://10.32.0.218:80
WEB_SERVICE_SERVICE_HOST=10.32.0.218

pingは通らないのね...

/ # ping -c 3 web-service
PING web-service (10.32.0.218): 56 data bytes

--- web-service ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

type: ClusterIP (minikube環境)

デプロイメントとサービスを作成

vagrant@minikube:~/codes_for_lessons/step09$ kubectl apply -f deploy.yml
deployment.apps/web-deploy created
vagrant@minikube:~/codes_for_lessons/step09$ kubectl apply -f svc.yml
service/web-service created

確認

vagrant@minikube:~$ kubectl get all -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
pod/web-deploy-654cd6d4d8-gsn8r   1/1     Running   0          24s   172.17.0.4   minikube   <none>           <none>
pod/web-deploy-654cd6d4d8-qz9fm   1/1     Running   0          24s   172.17.0.3   minikube   <none>           <none>
pod/web-deploy-654cd6d4d8-xt78f   1/1     Running   0          24s   172.17.0.2   minikube   <none>           <none>

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP   20d   <none>
service/web-service   ClusterIP   10.96.176.255   <none>        80/TCP    17s   app=web

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES         SELECTOR
deployment.apps/web-deploy   3/3     3            3           24s   nginx        nginx:latest   app=web

NAME                                    DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES         SELECTOR
replicaset.apps/web-deploy-654cd6d4d8   3         3         3       24s   nginx        nginx:latest   app=web,pod-template-hash=654cd6d4d8

busyboxから確認

vagrant@minikube:~$ kubectl run -it busybox --restart=Never --rm --image=busybox sh
If you don't see a command prompt, try pressing enter.
/ # wget -q -O - http://web-service
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

/ # nslookup web-service
Server:         10.96.0.10
Address:        10.96.0.10:53

Name:   web-service.default.svc.cluster.local
Address: 10.96.176.255

*** Can't find web-service.svc.cluster.local: No answer
*** Can't find web-service.cluster.local: No answer
*** Can't find web-service.dhcp.makuhari.japan.ibm.com: No answer
*** Can't find web-service.default.svc.cluster.local: No answer
*** Can't find web-service.svc.cluster.local: No answer
*** Can't find web-service.cluster.local: No answer
*** Can't find web-service.dhcp.makuhari.japan.ibm.com: No answer


/ # ping -c 3 web-service
PING web-service (10.96.176.255): 56 data bytes

--- web-service ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

minikubeの環境だと、ノードが1つしかないからか、何度やってもhttp://web-serviceへのアクセスは成功しますね(意図した通りに動いている！)。
nslookupで名前解決もできている。
"web-service"に対してpingは通らないのだが、この挙動は正しいのかな？

type: NodePort (マルチノード環境)

以下のサービス追加
https://github.com/takara9/codes_for_lessons/blob/master/step09/svc-np.yml

vagrant@master:~/codes_for_lessons/step09$ kubectl apply -f svc-np.yml
service/web-service-np created

確認

vagrant@master:~$ kubectl get service -o wide
NAME             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE    SELECTOR
kubernetes       ClusterIP   10.32.0.1     <none>        443/TCP        39d    <none>
web-service      ClusterIP   10.32.0.192   <none>        80/TCP         107s   app=web
web-service-np   NodePort    10.32.0.127   <none>        80:32399/TCP   65s    app=web

32300ポートで公開された。
node1でListenされているポートを確認

vagrant@node1:~$ netstat -an |grep 32399
tcp6       0      0 :::32399                :::*                    LISTEN

Listenされてますね。
このノードはWindows上のVirtualBoxのゲストOSとして動いているので、このポートを同じ番号でホストOSからもアクセスできるようにしてあげて、ホストOS(Windows)のブラウザからアクセスしてみると...

ちゃんとnginxのサーバーにアクセスできました！

type: NodePort (minikube環境)

以下のサービス追加
https://github.com/takara9/codes_for_lessons/blob/master/step09/svc-np.yml

vagrant@minikube:~/codes_for_lessons/step09$ kubectl apply -f svc-np.yml
service/web-service-np created

確認

vagrant@minikube:~$ kubectl get service -o wide
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE   SELECTOR
kubernetes       ClusterIP   10.96.0.1       <none>        443/TCP        20d   <none>
web-service      ClusterIP   10.96.176.255   <none>        80/TCP         10m   app=web
web-service-np   NodePort    10.111.60.245   <none>        80:31014/TCP   19s   app=web

31014ポートで公開された。ノード上でポートの様子を見てみると...

vagrant@minikube:~$ netstat -an | grep 31014
tcp6       0      0 :::31014                :::*                    LISTEN

Listenされてますね。
このノードはWindows上のVirtualBoxのゲストOSとして動いているので、このポートを同じ番号でホストOSからもアクセスできるようにしてあげて、ホストOS(Windows)のブラウザからアクセスしてみると...

ちゃんとnginxのサーバーにアクセスできました！

type: ExternalName

以下のサービスを追加して、Pod内から、www.yaho.co.jpに対してyahooという名前でアクセスできるようにしてみます。
https://github.com/takara9/codes_for_lessons/blob/master/step09/svc-ext-dns.yml

サービス追加前のPod(busybox)からのアクセス状況

/ # ping -c 3 yahoo
ping: bad address 'yahoo'

/ # nslookup yahoo
Server:         10.32.0.10
Address:        10.32.0.10:53

** server can't find yahoo.default.svc.cluster.local: NXDOMAIN

*** Can't find yahoo.svc.cluster.local: No answer
*** Can't find yahoo.cluster.local: No answer
*** Can't find yahoo.dhcp.makuhari.japan.ibm.com: No answer
*** Can't find yahoo.default.svc.cluster.local: No answer
*** Can't find yahoo.svc.cluster.local: No answer
*** Can't find yahoo.cluster.local: No answer
*** Can't find yahoo.dhcp.makuhari.japan.ibm.com: No answer

サービス追加

vagrant@master:~/codes_for_lessons/step09$ kubectl apply -f svc-ext-dns.yml
service/yahoo created

vagrant@master:~/codes_for_lessons/step09$ kubectl get service -o wide
NAME          TYPE           CLUSTER-IP   EXTERNAL-IP       PORT(S)   AGE    SELECTOR
kubernetes    ClusterIP      10.32.0.1    <none>            443/TCP   165m   <none>
web-service   ClusterIP      10.32.0.79   <none>            80/TCP    100m   app=web
yahoo         ExternalName   <none>       www.yahoo.co.jp   <none>    17s    <none>

サービス追加後のPod(busybox)からのアクセス状況

/ # ping -c 3 yahoo
PING yahoo (183.79.217.124): 56 data bytes
64 bytes from 183.79.217.124: seq=0 ttl=44 time=25.644 ms
64 bytes from 183.79.217.124: seq=1 ttl=44 time=24.332 ms
64 bytes from 183.79.217.124: seq=2 ttl=44 time=24.400 ms

--- yahoo ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 24.332/24.792/25.644 ms

/ # nslookup yahoo
Server:         10.32.0.10
Address:        10.32.0.10:53

** server can't find yahoo.default.svc.cluster.local: NXDOMAIN

*** Can't find yahoo.svc.cluster.local: No answer
*** Can't find yahoo.cluster.local: No answer
*** Can't find yahoo.dhcp.makuhari.japan.ibm.com: No answer
*** Can't find yahoo.default.svc.cluster.local: No answer
*** Can't find yahoo.svc.cluster.local: No answer
*** Can't find yahoo.cluster.local: No answer
*** Can't find yahoo.dhcp.makuhari.japan.ibm.com: No answer

Pod内のコンテナからyahooに対して名前解決/Pingが通ることが確認できました。

※というか、Pod内から外のサーバーへのアクセスはできるのね。ネットワークむつかし。