KubernetesのヘルスチェックでLiveness Probeを使う

参考にさせてもらったもの

Kubernetes完全ガイドをそのままやらせていただきました！

Kubernetesのヘルスチェック

• Liveness Probe　⇨　Podが動作しているか確認し、NGであればPodの再起動をする
• Readiness Probe　⇨　Podが動作しているか確認し、NGであっても再起動しない
• LoadBarancerサービスのヘルスチェックはICMPのみ
• チェック方法は３種類で、exce（コマンドでのリターンコード）、httpGet（ステータスコード）、tcpSocket（TCPセッション）

マニュフェストファイル作成

sample-healthcheck.yaml

apiVersion: v1
kind: Pod
metadata:
  name: sample-healthcheck
  labels:
    app: sample-app
spec:
  containers:
    - name: nginx-container
      image: nginx:1.12
      ports:
      - containerPort: 80
      livenessProbe:
        httpGet:
          path: /index.html
          port: 80
          scheme: HTTP
        timeoutSeconds: 1
        successThreshold: 1
        failureThreshold: 2
        initialDelaySeconds: 5
        periodSeconds: 3
      readinessProbe:
        exec:
          command: ["ls", "/usr/share/nginx/html/50x.html"]
        timeoutSeconds: 1
        successThreshold: 2
        failureThreshold: 1
        initialDelaySeconds: 5
        periodSeconds: 3

デプロイ（Readiness Probeも一緒にされる見たい）

k apply -f sample-healthcheck.yaml

Podに設定されたProbeの確認

~/Desktop ❯❯❯ k describe pod sample-healthcheck | egrep "Liveness|Readness"
    Liveness:       http-get http://:80/index.html delay=5s timeout=1s period=3s #success=1 #failure=2
~/Desktop ❯❯❯

インデックスファイルを削除してわざとヘルスチェックを失敗させる

k exec -it sample-liveness rm /usr/share/nginx/html/index.html

別窓で見てみるとRESTARTが繰り返されている

~/Desktop ❯❯❯ k get pods sample-liveness --watch
NAME              READY   STATUS    RESTARTS   AGE
sample-liveness   1/1     Running   1          4m14s
NAME              READY   STATUS    RESTARTS   AGE
sample-liveness   1/1     Running   2          4m26s
sample-liveness   1/1     Running   3          5m15s

ヘルスチェックの履歴はdescribeで見れる

k describe pod sample-liveness

Name:         sample-liveness
Namespace:    default
Priority:     0
Node:         ip-172-16-139-45.us-east-1.compute.internal/172.16.139.45
Start Time:   Mon, 18 Nov 2019 17:09:46 +0900
Labels:       app=sample-app
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"app":"sample-app"},"name":"sample-liveness","namespace":"default"}...
              kubernetes.io/psp: eks.privileged
Status:       Running
IP:           172.16.143.102
Containers:
  nginx-container:
    Container ID:   docker://3ef5c4f01ce5913e474af0c67d8d38added4f436f98a3a9cb60c1b2341796994
    Image:          nginx:1.12
    Image ID:       docker-pullable://nginx@sha256:72daaf46f11cc753c4eab981cbf869919bd1fee3d2170a2adeac12400f494728
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 18 Nov 2019 17:16:39 +0900
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 18 Nov 2019 17:15:33 +0900
      Finished:     Mon, 18 Nov 2019 17:15:48 +0900
    Ready:          True
    Restart Count:  5
    Liveness:       http-get http://:80/index.html delay=5s timeout=1s period=3s #success=1 #failure=2
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-hb7gs (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-hb7gs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-hb7gs
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                                                  Message
  ----     ------     ----                   ----                                                  -------
  Normal   Scheduled  7m36s                  default-scheduler                                     Successfully assigned default/sample-liveness to ip-172-16-139-45.us-east-1.compute.internal
  Normal   Started    2m22s (x4 over 7m35s)  kubelet, ip-172-16-139-45.us-east-1.compute.internal  Started container nginx-container
  Normal   Pulled     109s (x5 over 7m35s)   kubelet, ip-172-16-139-45.us-east-1.compute.internal  Container image "nginx:1.12" already present on machine
  Normal   Created    109s (x5 over 7m35s)   kubelet, ip-172-16-139-45.us-east-1.compute.internal  Created container nginx-container
  Warning  Unhealthy  109s (x8 over 4m22s)   kubelet, ip-172-16-139-45.us-east-1.compute.internal  Liveness probe failed: HTTP probe failed with statuscode: 404
  Normal   Killing    109s (x4 over 4m19s)   kubelet, ip-172-16-139-45.us-east-1.compute.internal  Container nginx-container failed liveness probe, will be restarted