初めにPodとContainerのStatusについて

Overview

PodのStatusは、PodStatusというObjectで、その中に以下のFieldがある (公式: Pod Lifecycle)

conditions
containerStatuses
hostIP
phase
podIP
qosClass
startTime

おおまかに以下のようになっている。 Documentには phase, conditions, container statusについての記載がある。

status:
  conditions:
  - <conditionがArrayで入っている>
  containerStatuses:
  - <containerのStatusesが入っている>
  hostIP: <hostIp>
  phase: <phase>
  podIP: <podIp>
  qosClass: <qosClass>
  startTime: "2019-12-17T11:51:17Z"

重要な部分は、

podのphaseとstatus (conditionの中のfield) は異なる
containerにはstateとreadyという状態を表すfieldがある

1. Phase

Phaseは以下の5種類のみ

Pending
Running
Succeeded
Failed
Unknown

2. Conditions

Arrayで以下の項目で構成される

lastProbeTime
lastTransitionTime
status
message
reason
type

例:


status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-12-17T11:51:17Z"
    status: "True"
    type: Initialized
...
  - lastProbeTime: null
    lastTransitionTime: "2020-01-24T09:58:21Z"
    message: 'containers with unready status: [web-app]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady

3. ContainerStatuses

ContainerのstatusもArrayである

Stateは以下の3つのみ

Waiting
Running
Terminated

Stateの中に messageとreasonがある

  containerStatuses:
  - containerID: docker://<hash>
    image: /path/to/image/web-app:tag
    imageID: docker-pullable:///path/to/image/web-app:tag@sha256:<hash>
    lastState:
      terminated:
        containerID: docker://<hash>
        exitCode: 0
        finishedAt: "2020-01-24T09:58:20Z"
        reason: Completed
        startedAt: "2020-01-24T09:58:19Z"
    name: web-app
    ready: false
    restartCount: 1148
    state:
      waiting:
        message: Back-off 5m0s restarting failed container=web-app pod=web-app-<hash>_<namespace>(<hash>)
        reason: CrashLoopBackOff

4. get podした時に出てくる STATUSは？

普段良く使う kubectl get pod で見るSTATUSは、単純に上記のどれかのFieldと対応していない！ (詳細は、Kubernetes: kubectl 上の Pod のステータス表記について
のページがとても詳しいので是非)

kubectl get pod <pod_name> -n <namespace>
NAME         READY   STATUS             RESTARTS   AGE
<pod_name>   0/1     CrashLoopBackOff   1152       37d

phaseはRunningで, ContainerStatusは waitingで reasonが CrashLoopBackOffという状態
READYが 0/1なのは、 containerが一つで、readyがfalseのため

readinessProbeとlivenessProbe

Configure Liveness, Readiness and Startup Probes

containerがスタートした時に、readyかどうかをチェックして、readyの状態になってからtrafficを流すようにしてくれる (readinessProbe)
containerが動いているとき、定期的にチェック (health check) して、もしもfalseになった場合には、containerをrestartする (livenessProbe)

チェック: Multi-Containerで一つのcontainerのReadinessProbe とLivenessProbeがずっとReadyにならないケース

kubectl get pod で状態を観察した

Status:Running Ready:1/2 ↔ Status:CrashLoopBackOff Ready:1/2 このふたつの状態を繰り返していた。Readyとならないcontainerはずっとrestartされていたが、もう一つのコンテナ(今回nginxを使用)は問題なく動いていた

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          limits:
            memory: "20Mi"
          requests:
            memory: "20Mi"
        ports:
        - containerPort: 80
      - name: memory-demo-ctr
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20
        image: polinux/stress
        resources:
          limits:
            memory: "15Mi"
          requests:
            memory: "10Mi"
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "10M", "--vm-hang", "1"]

Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-dsdk2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dsdk2
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From                                                       Message
  ----     ------     ----                  ----                                                       -------
  Normal   Scheduled  39m                   default-scheduler                                          Successfully
assigned naka/nginx-deployment-669b897679-msndc to ip-192-168-4-159.ap-northeast-1.compute.internal
  Normal   Pulled     39m                   kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Container ima
ge "nginx:1.7.9" already present on machine
  Normal   Created    39m                   kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Created conta
iner nginx
  Normal   Started    39m                   kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Started conta
iner nginx
  Normal   Killing    38m                   kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Container mem
ory-demo-ctr failed liveness probe, will be restarted
  Normal   Pulling    38m (x2 over 39m)     kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Pulling image
 "polinux/stress"
  Normal   Pulled     38m (x2 over 39m)     kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Successfully
pulled image "polinux/stress"
  Normal   Created    38m (x2 over 39m)     kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Created conta
iner memory-demo-ctr
  Normal   Started    38m (x2 over 39m)     kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Started conta
iner memory-demo-ctr
  Warning  Unhealthy  37m (x8 over 39m)     kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Readiness pro
be failed: dial tcp 192.168.4.201:8080: connect: connection refused
  Warning  Unhealthy  9m56s (x32 over 39m)  kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Liveness prob
e failed: dial tcp 192.168.4.201:8080: connect: connection refused
  Warning  BackOff    4m52s (x79 over 28m)  kubelet, ip-192-168-4-159.ap-northeast-1.compute.internal  Back-off rest
arting failed container

NAME                                READY   STATUS             RESTARTS   AG
E
nginx-deployment-669b897679-kq84s   1/2     CrashLoopBackOff   13         42
m
nginx-deployment-669b897679-msndc   1/2     CrashLoopBackOff   13         42
m

OOMKilled

次にOOMKilledとは、Containerが ResourceLimit で設定されたMemoryの値以上にメモリを使おうとした時に、killされたということ

Configure Out of Resource Handling

if a Pod container is OOM killed, it may be restarted by the kubelet based on its RestartPolicy.

Pod is running and has one Container. Container runs out of memory.

Container terminates in failure.
Log OOM event.
If restartPolicy is:
Always: Restart Container; Pod phase stays Running.
OnFailure: Restart Container; Pod phase stays Running.
Never: Log failure event; Pod phase becomes Failed.

チェック: Multi-Containerで一つのcontainerが常にOOMKilledされるケース

ContainerCreating -> OOMKilled　-> CrashLoopBackOff -> OOMKilled ... STAUSは繰り返し (exponential back-off) が、もう一つのcontainerは問題なく動いている

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          limits:
            memory: "20Mi"
          requests:
            memory: "20Mi"
        ports:
        - containerPort: 80
      - name: memory-demo-ctr
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20
        image: polinux/stress
        resources:
          limits:
            memory: "15Mi"
          requests:
            memory: "10Mi"
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "15M", "--vm-hang", "1"]

get

kubectl get pod -n <namespace>
NAME                                READY   STATUS             RESTARTS   AG
E
nginx-deployment-6bd9c88968-dssp4   1/2     CrashLoopBackOff   11         35
m
nginx-deployment-6bd9c88968-xfmx6   1/2     CrashLoopBackOff   11         35
m

一つのPodをdescribeしてみる

kubectl describe pod <pod-name> -n <namespace>

...

Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-dsdk2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dsdk2
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                                                       Message
  ----     ------     ----                   ----                                                       -------
  Normal   Scheduled  33m                    default-scheduler                                          Successfully assigned naka/nginx-deployment-6bd9c88968-xfmx6 to ip-192-168-5-113.ap-northeast-1.compute.internal
  Normal   Pulled     33m                    kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Container image "nginx:1.7.9" already present on machine
  Normal   Created    33m                    kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Created container nginx
  Normal   Started    33m                    kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Started container nginx
  Normal   Pulling    32m (x4 over 33m)      kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Pulling image "polinux/stress"
  Normal   Pulled     32m (x4 over 33m)      kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Successfully pulled image "polinux/stress"
  Normal   Created    32m (x4 over 33m)      kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Created container memory-demo-ctr
  Normal   Started    32m (x4 over 33m)      kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Started container memory-demo-ctr
  Warning  BackOff    3m52s (x145 over 33m)  kubelet, ip-192-168-5-113.ap-northeast-1.compute.internal  Back-off restarting failed container

Service

概要

コンテナごとにReadinessProbeが設定できるので、Multi-containerのPodで一つContainerがサービスが続くかと思っていた。が、違う。

With the introduction of new Pod conditions, a Pod is evaluated to be ready only when both the following statements are true:

All containers in the Pod are ready.

All conditions specified in ReadinessGates are “True”.

Serviceを利用する場合は、すべてのコンテナがreadyでないといけない。上記の例では、nginxコンテナは問題なく動作していたとしても、serviceを介してアクセスしている場合には、同居しているcontainerが死ぬと、そのpodがreadyではなくなり、serviceのendpointから外されてしまう。

検証

以下のように二組 Deploymentと Serviceを用意する.


tree practice/resource/02
practice/resource/02
├── README.md
├── mulit-container-oom.yaml
├── mulit-container.yaml
├── service-oom.yaml
└── service.yaml

nginxとOOMされるコンテナを同居させたdeploymentとnginxのservice
nginxとReadyになるコンテナを同居させたdeploymentとnginxのservice

practice/resource/02/mulit-container-oom.yaml

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-oom
  namespace: naka
spec:
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  selector:
    matchLabels:
      app: nginx-oom
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx-oom
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          limits:
            memory: "20Mi"
          requests:
            memory: "20Mi"
        ports:
        - containerPort: 80
      - name: memory-demo-ctr
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20
        image: polinux/stress
        resources:
          limits:
            memory: "15Mi"
          requests:
            memory: "10Mi"
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "15M", "--vm-hang", "1"]

practice/resource/02/service-oom.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-oom
  labels:
    app: nginx-oom
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: nginx-oom

practice/resource/02/mulit-container.yaml

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx
  namespace: naka
spec:
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          limits:
            memory: "20Mi"
          requests:
            memory: "20Mi"
        ports:
        - containerPort: 80
      - name: memory-demo-ctr
        image: polinux/stress
        resources:
          limits:
            memory: "15Mi"
          requests:
            memory: "10Mi"
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "1M", "--vm-hang", "1"]

practice/resource/02/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: nginx

DeploymentとServiceの作成

kubectl create namespace naka
kubectl apply -n naka -f practice/resource/02/
deployment.apps/nginx-oom created
deployment.apps/nginx created
service/nginx-oom created
service/nginx created

Podを確認

kubectl get pod -n naka
NAME                         READY   STATUS             RESTARTS   AGE
nginx-7fb468f99f-b252k       2/2     Running            0          13m
nginx-7fb468f99f-rf7x2       2/2     Running            0          13m
nginx-oom-6655d46664-khbr6   1/2     CrashLoopBackOff   7          14m
nginx-oom-6655d46664-v5d6r   1/2     CrashLoopBackOff   7          14m

Serviceの確認

kubectl get svc -n naka
NAME        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
nginx       ClusterIP   10.100.104.244   <none>        80/TCP    15m
nginx-oom   ClusterIP   10.100.133.104   <none>        80/TCP    15m

port-forwardで確認

正常に動いている方をport-forward

kubectl port-forward svc/nginx -n naka 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

別windowでcurlを叩いてみる

curl localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

正常に動いている

OOMKilledされるcontainerと同居させたほう

kubectl port-forward svc/nginx-oom -n naka 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

curl localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

こちらもnginxは動いている

Endpointの確認

kubectl get endpoints -n naka
NAME        ENDPOINTS                         AGE
nginx       192.168.5.22:80,192.168.6.60:80   18m
nginx-oom                                     18m

ドキュメント通り2つのコンテナがReadyとなってない nginx-oom では EndpointsにPodのIPがふられていないことがわかる!

別のPodからcurlでサービス通してアクセス

別のPodをたてる

kubectl run curl --image=radial/busyboxplus:curl -i --tty
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
[ root@curl-66bdcf564-g7cl8:/ ]$

正常に動いている方を確認

サービス名を使って名前解決してアクセスできる

[ root@curl-66bdcf564-g7cl8:/ ]$ nslookup nginx
Server:    10.100.0.10
Address 1: 10.100.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nginx
Address 1: 10.100.104.244 nginx.naka.svc.cluster.local
[ root@curl-66bdcf564-g7cl8:/ ]$ curl nginx.naka.svc.cluster.local:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

OOMのほうを確認

EndpointがServiceから削除されてるので、サービスからアクセスが出来ずTimeoutするのが確認できた

nslookup nginx-oom
Server:    10.100.0.10
Address 1: 10.100.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nginx-oom
Address 1: 10.100.133.104 nginx-oom.naka.svc.cluster.local
curl --connect-timeout 10 nginx-oom.naka.svc.cluster.local:80
curl: (28) Connection timed out after 10001 milliseconds