追記
SlackのKubernetesのjp-usersチャンネルに投稿したところ、nekopさんより下記の助言をいただきました。ありがとうございます！
https://github.com/kubernetes/kubernetes/pull/46371/files
https://github.com/sjenning/kubernetes/blob/2c866a7aaa45dd552df9ea53cc1ca20b0d502a52/pkg/kubelet/prober/worker.go#L227
https://github.com/sjenning/kubernetes/blob/master/pkg/kubelet/prober/worker.go#L243
https://github.com/sjenning/kubernetes/commit/b49e07846f6779fdb35efb33eab5456bb85ec9f8
https://github.com/kubernetes/kubernetes/commit/b49e07846f6779fdb35efb33eab5456bb85ec9f8
「1.11以降では直っているのではないか？」ということなので、実際に1.11で動かしてみました。
環境
minikube v0.28.2
ただし、今回は--kubernetes-versio v1.11.3をつけてバージョンを変更します。
（前回は1.10系）
kubectl version
# Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
# Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
1.11.3で起動しているようです。
Deployment.ymlは前回と同じものをそのまま使います。
疎通確認
0s 0s 1 nginx-deployment-54d7b449d9-xwmsf.15588f8e14305452 Pod spec.containers{nginx} Normal Pulled kubelet, minikube Container image "nginx:1.7.9" already present on machine
0s 0s 1 nginx-deployment-54d7b449d9-xwmsf.15588f8e24353f4d Pod spec.containers{nginx} Normal Created kubelet, minikube Created container
0s 0s 1 nginx-deployment-54d7b449d9-xwmsf.15588f8e2f30bd16 Pod spec.containers{nginx} Normal Started kubelet, minikube Started container
0s 0s 1 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 5s 2 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 10s 3 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 15s 4 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 20s 5 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 0s 1 nginx-deployment-54d7b449d9-xwmsf.15588f9607af13d7 Pod spec.containers{nginx} Normal Killing kubelet, minikube Killing container with id docker://nginx:Container failed liveness probe.. Container will be killed and recreated.
0s 34s 2 nginx-deployment-54d7b449d9-xwmsf.15588f8e14305452 Pod spec.containers{nginx} Normal Pulled kubelet, minikube Container image "nginx:1.7.9" already present on machine
0s 34s 2 nginx-deployment-54d7b449d9-xwmsf.15588f8e24353f4d Pod spec.containers{nginx} Normal Created kubelet, minikube Created container
0s 34s 2 nginx-deployment-54d7b449d9-xwmsf.15588f8e2f30bd16 Pod spec.containers{nginx} Normal Started kubelet, minikube Started container
0s 35s 6 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 40s 7 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 45s 8 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 50s 9 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 55s 10 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 35s 2 nginx-deployment-54d7b449d9-xwmsf.15588f9607af13d7 Pod spec.containers{nginx} Normal Killing kubelet, minikube Killing container with id docker://nginx:Container failed liveness probe.. Container will be killed and recreated.
0s 1m 3 nginx-deployment-54d7b449d9-xwmsf.15588f8e14305452 Pod spec.containers{nginx} Normal Pulled kubelet, minikube Container image "nginx:1.7.9" already present on machine
2回目のLiveness Probeのチェックも想定通り5回出ていました。
nekopさんご指摘の通りの結果となりました！
まとめ
- 1.10系での2回目移行のfailureThresholdが1回分少ないのは不具合
- v1.11系で修正済み
「多分不具合じゃなくてそういうものなんだろうな」と思ってしまっていたのですが、自分でもよく調べないとダメですね……。
どのコンポーネントが影響していそうかは、なんとなくわかるようにならないといけないと感じました。
（少しずつでもソースに触れないと…。）
旧記事[調査中] KubernetesのLiveness ProbeのfailureThreshold
はじめに
KubernetesのヘルスチェックのfailureThresholdの挙動確認時、自分の想定と異なっていたのでメモ
初回のfailureThresholdは指定値でコンテナ再起動を行うが、再起動後はfailureThreshold-1回で再起動を行うような挙動に見える。
疎通環境
minikube v0.28.2
Deployment
存在しないパスに5秒間隔でヘルスチェックを行い、わざと失敗させる。
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /pathnotfound # 存在しないパスのため404
port: 80
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 5
想定していた挙動と実際の挙動
想定していた動き
[Created container]
[Started container]
[Unhealthy(1回目)]
[Unhealthy(2回目)]
[Unhealthy(3回目)]
[Unhealthy(4回目)]
[Unhealthy(5回目)]（failureThreshold）
[Kill Container]
[Created container]
[Started container]
[Unhealthy(6回目)]
[Unhealthy(7回目)]
[Unhealthy(8回目)]
[Unhealthy(9回目)]
[Unhealthy(10回目)]（failureThreshold）
[kill Container]
...
実際の動き
[Created container]
[Started container]
[Unhealthy(1回目)]
[Unhealthy(2回目)]
[Unhealthy(3回目)]
[Unhealthy(4回目)]
[Unhealthy(5回目)]（failureThreshold）OK
[Kill Container]
[Created container]
[Started container]
[Unhealthy(6回目)]
[Unhealthy(7回目)]
[Unhealthy(8回目)]
[Unhealthy(9回目)]（failureThreshold?）想定と異なる部分
[kill Container]
...
kubectl get eventの結果
0s 0s 1 nginx-deployment-5cdfc956db-n56dj.155845c2e184ea50 Pod spec.containers{nginx} Normal Pulled kubelet, minikube Container image "nginx:1.7.9" already present on machine
0s 0s 1 nginx-deployment-5cdfc956db-n56dj.155845c2edc3eb4f Pod spec.containers{nginx} Normal Created kubelet, minikube Created container
0s 0s 1 nginx-deployment-5cdfc956db-n56dj.155845c2fa234bdc Pod spec.containers{nginx} Normal Started kubelet, minikube Started container
0s 0s 1 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 5s 2 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 10s 3 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 15s 4 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 20s 5 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 0s 1 nginx-deployment-5cdfc956db-n56dj.155845cadd2a06c2 Pod spec.containers{nginx} Normal Killing kubelet, minikube Killing container with id docker://nginx:Container failed liveness probe.. Container will be killed and recreated.
0s 34s 2 nginx-deployment-5cdfc956db-n56dj.155845c2e184ea50 Pod spec.containers{nginx} Normal Pulled kubelet, minikube Container image "nginx:1.7.9" already present on machine
0s 34s 2 nginx-deployment-5cdfc956db-n56dj.155845c2edc3eb4f Pod spec.containers{nginx} Normal Created kubelet, minikube Created container
0s 34s 2 nginx-deployment-5cdfc956db-n56dj.155845c2fa234bdc Pod spec.containers{nginx} Normal Started kubelet, minikube Started container
0s 35s 6 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 40s 7 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 45s 8 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 50s 9 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 30s 2 nginx-deployment-5cdfc956db-n56dj.155845cadd2a06c2 Pod spec.containers{nginx} Normal Killing kubelet, minikube Killing container with id docker://nginx:Container failed liveness probe.. Container will be killed and recreated.
0s 1m 3 nginx-deployment-5cdfc956db-n56dj.155845c2e184ea50 Pod spec.containers{nginx} Normal Pulled kubelet, minikube Container image "nginx:1.7.9" already present on machine