概要
Redis Sentinel構成を取った際に、検討すべきパラメータとしてdown-after-millisecondsというのがあります。
down-after-milliseconds is the time in milliseconds an instance should not be reachable (either does not reply to our PINGs or it is replying with an error) for a Sentinel starting to think it is down.
Redis Sentinel Documentation – Redis
SentinelプロセスがRedisプロセスがダウンしたと判断する時間[ミリ秒]とあります。つまり、これが短いほどフェイルオーバーが速く完了して、クライアントのエラーになる可能性が減るということになります。
デフォルトは30秒ですが、どこまで減らせるかを実験してみました。
前提
サーバ：Amazon Linux2
インスタンスタイプ：t2.micro
Redis：5.0.8
1台のサーバにRedis(master)x1、Redis(replica)x2、Sentinelx2を起動する構成。
フェイルオーバーはmasterをshutdownして発生させる。
結論
10秒くらいが妥当ではないかと判断します。（down-after-milliseconds=10000）
それ以上下げると、ネットワーク瞬断等でフェイルオーバーしてしまう可能性が高まります。フェイルオーバーするとマニュアルオペレーションで戻す必要があり、Redis Sentinel構成の運用負荷が上がるので、自動でフェイルオーバーする仕組みをスポイルするのではと思います。
実験
以下の秒数を実験した。
30秒、20秒、10秒、1秒、1ミリ秒、0ミリ秒
まとめ
| 秒数 | フェイルオーバーにかかった秒数（※1） | OK/NG |
|---|---|---|
| 30秒 | 32.3秒 | OK |
| 20秒 | 22.5秒 | OK |
| 10秒 | 12.1秒 | OK |
| 1秒 | 3.3秒 | OK（※2） |
| 1ミリ秒 | 4.8秒 | NG |
| 0ミリ秒 | - | NG |
※１・・User requested shutdown... と +switch-master mymasterの間。
※２・・今回は同一サーバ内でネットワークレイテンシがない状態だったが、分散配置した時はNGになると思われる。
実験詳細
30秒
[ec2-user@ip-172-30-2-184 redis-5.0.8]$ src/redis-cli -p 6379 shutdown
3457:M 12 Jul 2020 12:41:44.639 # User requested shutdown...
3457:M 12 Jul 2020 12:41:44.639 * Saving the final RDB snapshot before exiting.
3457:M 12 Jul 2020 12:41:44.641 * DB saved on disk
3457:M 12 Jul 2020 12:41:44.641 * Removing the pid file.
3457:M 12 Jul 2020 12:41:44.641 # Redis is now ready to exit, bye bye...
3466:S 12 Jul 2020 12:41:44.641 # Connection with master lost.
3466:S 12 Jul 2020 12:41:44.641 * Caching the disconnected master state.
3461:S 12 Jul 2020 12:41:44.641 # Connection with master lost.
3461:S 12 Jul 2020 12:41:44.641 * Caching the disconnected master state.
[1] Done ./src/redis-server ./redis1.conf
[ec2-user@ip-172-30-2-184 redis-5.0.8]$ 3461:S 12 Jul 2020 12:41:44.643 * Connecting to MASTER 172.30.2.184:6379
3461:S 12 Jul 2020 12:41:44.643 * MASTER <-> REPLICA sync started
3461:S 12 Jul 2020 12:41:44.643 # Error condition on socket for SYNC: Connection refused
3466:S 12 Jul 2020 12:41:45.318 * Connecting to MASTER 172.30.2.184:6379
3466:S 12 Jul 2020 12:41:45.318 * MASTER <-> REPLICA sync started
3466:S 12 Jul 2020 12:41:45.318 # Error condition on socket for SYNC: Connection refused
3461:S 12 Jul 2020 12:41:45.646 * Connecting to MASTER 172.30.2.184:6379
3461:S 12 Jul 2020 12:41:45.646 * MASTER <-> REPLICA sync started
3461:S 12 Jul 2020 12:41:45.646 # Error condition on socket for SYNC: Connection refused
3466:S 12 Jul 2020 12:41:46.320 * Connecting to MASTER 172.30.2.184:6379
3466:S 12 Jul 2020 12:41:46.320 * MASTER <-> REPLICA sync started
3466:S 12 Jul 2020 12:41:46.320 # Error condition on socket for SYNC: Connection refused
3461:S 12 Jul 2020 12:41:46.647 * Connecting to MASTER 172.30.2.184:6379
3461:S 12 Jul 2020 12:41:46.648 * MASTER <-> REPLICA sync started
3461:S 12 Jul 2020 12:41:46.648 # Error condition on socket for SYNC: Connection refused
3466:S 12 Jul 2020 12:41:47.323 * Connecting to MASTER 172.30.2.184:6379
3466:S 12 Jul 2020 12:41:47.323 * MASTER <-> REPLICA sync started
3466:S 12 Jul 2020 12:41:47.323 # Error condition on socket for SYNC: Connection refused
3461:S 12 Jul 2020 12:41:47.651 * Connecting to MASTER 172.30.2.184:6379
3461:S 12 Jul 2020 12:41:47.651 * MASTER <-> REPLICA sync started
3461:S 12 Jul 2020 12:41:47.651 # Error condition on socket for SYNC: Connection refused
3466:S 12 Jul 2020 12:41:48.326 * Connecting to MASTER 172.30.2.184:6379
3466:S 12 Jul 2020 12:41:48.326 * MASTER <-> REPLICA sync started
3466:S 12 Jul 2020 12:41:48.326 # Error condition on socket for SYNC: Connection refused
3461:S 12 Jul 2020 12:41:48.655 * Connecting to MASTER 172.30.2.184:6379
3461:S 12 Jul 2020 12:41:48.655 * MASTER <-> REPLICA sync started
3461:S 12 Jul 2020 12:41:48.655 # Error condition on socket for SYNC: Connection refused
3466:S 12 Jul 2020 12:41:49.329 * Connecting to MASTER 172.30.2.184:6379
3466:S 12 Jul 2020 12:41:49.330 * MASTER <-> REPLICA sync started
3466:S 12 Jul 2020 12:41:49.330 # Error condition on socket for SYNC: Connection refused
3461:S 12 Jul 2020 12:41:49.659 * Connecting to MASTER 172.30.2.184:6379
3461:S 12 Jul 2020 12:41:49.659 * MASTER <-> REPLICA sync started
3461:S 12 Jul 2020 12:41:49.659 # Error condition on socket for SYNC: Connection refused
（中略）
3466:S 12 Jul 2020 12:42:11.389 * Connecting to MASTER 172.30.2.184:6379
3466:S 12 Jul 2020 12:42:11.389 * MASTER <-> REPLICA sync started
3466:S 12 Jul 2020 12:42:11.389 # Error condition on socket for SYNC: Connection refused
3461:S 12 Jul 2020 12:42:11.718 * Connecting to MASTER 172.30.2.184:6379
3461:S 12 Jul 2020 12:42:11.718 * MASTER <-> REPLICA sync started
3461:S 12 Jul 2020 12:42:11.718 # Error condition on socket for SYNC: Connection refused
3466:S 12 Jul 2020 12:42:12.391 * Connecting to MASTER 172.30.2.184:6379
3466:S 12 Jul 2020 12:42:12.391 * MASTER <-> REPLICA sync started
3466:S 12 Jul 2020 12:42:12.391 # Error condition on socket for SYNC: Connection refused
3461:S 12 Jul 2020 12:42:12.720 * Connecting to MASTER 172.30.2.184:6379
3461:S 12 Jul 2020 12:42:12.720 * MASTER <-> REPLICA sync started
3461:S 12 Jul 2020 12:42:12.720 # Error condition on socket for SYNC: Connection refused
3466:S 12 Jul 2020 12:42:13.394 * Connecting to MASTER 172.30.2.184:6379
3466:S 12 Jul 2020 12:42:13.394 * MASTER <-> REPLICA sync started
3466:S 12 Jul 2020 12:42:13.395 # Error condition on socket for SYNC: Connection refused
3461:S 12 Jul 2020 12:42:13.723 * Connecting to MASTER 172.30.2.184:6379
3461:S 12 Jul 2020 12:42:13.723 * MASTER <-> REPLICA sync started
3461:S 12 Jul 2020 12:42:13.723 # Error condition on socket for SYNC: Connection refused
3466:S 12 Jul 2020 12:42:14.396 * Connecting to MASTER 172.30.2.184:6379
3466:S 12 Jul 2020 12:42:14.396 * MASTER <-> REPLICA sync started
3466:S 12 Jul 2020 12:42:14.397 # Error condition on socket for SYNC: Connection refused
3478:X 12 Jul 2020 12:42:14.710 # +sdown master mymaster 172.30.2.184 6379
3474:X 12 Jul 2020 12:42:14.719 # +sdown master mymaster 172.30.2.184 6379
3461:S 12 Jul 2020 12:42:14.725 * Connecting to MASTER 172.30.2.184:6379
3461:S 12 Jul 2020 12:42:14.725 * MASTER <-> REPLICA sync started
3461:S 12 Jul 2020 12:42:14.725 # Error condition on socket for SYNC: Connection refused
3474:X 12 Jul 2020 12:42:14.785 # +odown master mymaster 172.30.2.184 6379 #quorum 2/2
3474:X 12 Jul 2020 12:42:14.785 # +new-epoch 13
3474:X 12 Jul 2020 12:42:14.785 # +try-failover master mymaster 172.30.2.184 6379
3474:X 12 Jul 2020 12:42:14.787 # +vote-for-leader 9a72799337b637763343e01e971a580d51c7d4be 13
3478:X 12 Jul 2020 12:42:14.789 # +new-epoch 13
3478:X 12 Jul 2020 12:42:14.790 # +vote-for-leader 9a72799337b637763343e01e971a580d51c7d4be 13
3474:X 12 Jul 2020 12:42:14.790 # 9a72799337b637763343e01e971a580d51c7d4bf voted for 9a72799337b637763343e01e971a580d51c7d4be 13
3474:X 12 Jul 2020 12:42:14.849 # +elected-leader master mymaster 172.30.2.184 6379
3474:X 12 Jul 2020 12:42:14.849 # +failover-state-select-slave master mymaster 172.30.2.184 6379
3474:X 12 Jul 2020 12:42:14.950 # +selected-slave slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
3474:X 12 Jul 2020 12:42:14.950 * +failover-state-send-slaveof-noone slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
3474:X 12 Jul 2020 12:42:15.051 * +failover-state-wait-promotion slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
3466:M 12 Jul 2020 12:42:15.051 # Setting secondary replication ID to a99b1062bf55c59dc92efaa28d15629759725025, valid up to offset: 37809. New replication ID is 2a8a8c8bca5ece9b0df8dce813e00eb5dcb18803
3466:M 12 Jul 2020 12:42:15.051 * Discarding previously cached master state.
3466:M 12 Jul 2020 12:42:15.051 * MASTER MODE enabled (user request from 'id=5 addr=172.30.2.184:42552 fd=9 name=sentinel-9a727993-cmd age=304 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=140 qbuf-free=32628 obl=36 oll=0 omem=0 events=r cmd=exec')
3466:M 12 Jul 2020 12:42:15.052 # CONFIG REWRITE executed with success.
3461:S 12 Jul 2020 12:42:15.727 * Connecting to MASTER 172.30.2.184:6379
3461:S 12 Jul 2020 12:42:15.727 * MASTER <-> REPLICA sync started
3461:S 12 Jul 2020 12:42:15.727 # Error condition on socket for SYNC: Connection refused
3478:X 12 Jul 2020 12:42:15.782 # +odown master mymaster 172.30.2.184 6379 #quorum 2/2
3478:X 12 Jul 2020 12:42:15.782 # Next failover delay: I will not start a failover before Sun Jul 12 12:48:14 2020
3474:X 12 Jul 2020 12:42:15.806 # +promoted-slave slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
3474:X 12 Jul 2020 12:42:15.806 # +failover-state-reconf-slaves master mymaster 172.30.2.184 6379
3474:X 12 Jul 2020 12:42:15.870 * +slave-reconf-sent slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
3461:S 12 Jul 2020 12:42:15.870 * REPLICAOF 172.30.2.184:6381 enabled (user request from 'id=5 addr=172.30.2.184:40246 fd=9 name=sentinel-9a727993-cmd age=304 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=293 qbuf-free=32475 obl=36 oll=0 omem=0 events=r cmd=exec')
3461:S 12 Jul 2020 12:42:15.871 # CONFIG REWRITE executed with success.
3478:X 12 Jul 2020 12:42:15.872 # +config-update-from sentinel 9a72799337b637763343e01e971a580d51c7d4be 172.30.2.184 26379 @ mymaster 172.30.2.184 6379
3478:X 12 Jul 2020 12:42:15.872 # +switch-master mymaster 172.30.2.184 6379 172.30.2.184 6381
3478:X 12 Jul 2020 12:42:15.872 * +slave slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
3478:X 12 Jul 2020 12:42:15.872 * +slave slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6381
3461:S 12 Jul 2020 12:42:16.730 * Connecting to MASTER 172.30.2.184:6381
3461:S 12 Jul 2020 12:42:16.730 * MASTER <-> REPLICA sync started
3461:S 12 Jul 2020 12:42:16.730 * Non blocking connect for SYNC fired the event.
3461:S 12 Jul 2020 12:42:16.730 * Master replied to PING, replication can continue...
3461:S 12 Jul 2020 12:42:16.730 * Trying a partial resynchronization (request a99b1062bf55c59dc92efaa28d15629759725025:37809).
3466:M 12 Jul 2020 12:42:16.730 * Replica 172.30.2.184:6380 asks for synchronization
3466:M 12 Jul 2020 12:42:16.730 * Partial resynchronization request from 172.30.2.184:6380 accepted. Sending 446 bytes of backlog starting from offset 37809.
3461:S 12 Jul 2020 12:42:16.730 * Successful partial resynchronization with master.
3461:S 12 Jul 2020 12:42:16.731 # Master replication ID changed to 2a8a8c8bca5ece9b0df8dce813e00eb5dcb18803
3461:S 12 Jul 2020 12:42:16.731 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.
3474:X 12 Jul 2020 12:42:16.868 * +slave-reconf-inprog slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
3474:X 12 Jul 2020 12:42:16.868 * +slave-reconf-done slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
3474:X 12 Jul 2020 12:42:16.929 # -odown master mymaster 172.30.2.184 6379
3474:X 12 Jul 2020 12:42:16.929 # +failover-end master mymaster 172.30.2.184 6379
3474:X 12 Jul 2020 12:42:16.929 # +switch-master mymaster 172.30.2.184 6379 172.30.2.184 6381
3474:X 12 Jul 2020 12:42:16.930 * +slave slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
3474:X 12 Jul 2020 12:42:16.930 * +slave slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6381
3478:X 12 Jul 2020 12:42:45.889 # +sdown slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6381
3474:X 12 Jul 2020 12:42:46.972 # +sdown slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6381
[ec2-user@ip-172-30-2-184 redis-5.0.8]$
20秒
[ec2-user@ip-172-30-2-184 redis-5.0.8]$ src/redis-cli shutdown
4284:M 12 Jul 2020 13:47:29.290 # User requested shutdown...
4284:M 12 Jul 2020 13:47:29.290 * Saving the final RDB snapshot before exiting.
4284:M 12 Jul 2020 13:47:29.292 * DB saved on disk
4284:M 12 Jul 2020 13:47:29.292 * Removing the pid file.
4284:M 12 Jul 2020 13:47:29.292 # Redis is now ready to exit, bye bye...
4285:S 12 Jul 2020 13:47:29.292 # Connection with master lost.
4285:S 12 Jul 2020 13:47:29.292 * Caching the disconnected master state.
4286:S 12 Jul 2020 13:47:29.292 # Connection with master lost.
4286:S 12 Jul 2020 13:47:29.292 * Caching the disconnected master state.
[ec2-user@ip-172-30-2-184 redis-5.0.8]$ 4286:S 12 Jul 2020 13:47:30.258 * Connecting to MASTER 172.30.2.184:6379
4286:S 12 Jul 2020 13:47:30.258 * MASTER <-> REPLICA sync started
4286:S 12 Jul 2020 13:47:30.258 # Error condition on socket for SYNC: Connection refused
4285:S 12 Jul 2020 13:47:30.258 * Connecting to MASTER 172.30.2.184:6379
4285:S 12 Jul 2020 13:47:30.258 * MASTER <-> REPLICA sync started
4285:S 12 Jul 2020 13:47:30.258 # Error condition on socket for SYNC: Connection refused
4286:S 12 Jul 2020 13:47:31.261 * Connecting to MASTER 172.30.2.184:6379
4285:S 12 Jul 2020 13:47:31.261 * Connecting to MASTER 172.30.2.184:6379
4285:S 12 Jul 2020 13:47:31.261 * MASTER <-> REPLICA sync started
4286:S 12 Jul 2020 13:47:31.261 * MASTER <-> REPLICA sync started
4286:S 12 Jul 2020 13:47:31.261 # Error condition on socket for SYNC: Connection refused
4285:S 12 Jul 2020 13:47:31.261 # Error condition on socket for SYNC: Connection refused
4285:S 12 Jul 2020 13:47:32.263 * Connecting to MASTER 172.30.2.184:6379
4285:S 12 Jul 2020 13:47:32.263 * MASTER <-> REPLICA sync started
4285:S 12 Jul 2020 13:47:32.263 # Error condition on socket for SYNC: Connection refused
4286:S 12 Jul 2020 13:47:32.263 * Connecting to MASTER 172.30.2.184:6379
4286:S 12 Jul 2020 13:47:32.263 * MASTER <-> REPLICA sync started
4286:S 12 Jul 2020 13:47:32.263 # Error condition on socket for SYNC: Connection refused
4285:S 12 Jul 2020 13:47:33.265 * Connecting to MASTER 172.30.2.184:6379
4285:S 12 Jul 2020 13:47:33.265 * MASTER <-> REPLICA sync started
4285:S 12 Jul 2020 13:47:33.265 # Error condition on socket for SYNC: Connection refused
（中略）
4286:S 12 Jul 2020 13:47:46.301 * Connecting to MASTER 172.30.2.184:6379
4286:S 12 Jul 2020 13:47:46.301 * MASTER <-> REPLICA sync started
4286:S 12 Jul 2020 13:47:46.301 # Error condition on socket for SYNC: Connection refused
4285:S 12 Jul 2020 13:47:47.303 * Connecting to MASTER 172.30.2.184:6379
4285:S 12 Jul 2020 13:47:47.303 * MASTER <-> REPLICA sync started
4285:S 12 Jul 2020 13:47:47.303 # Error condition on socket for SYNC: Connection refused
4286:S 12 Jul 2020 13:47:47.303 * Connecting to MASTER 172.30.2.184:6379
4286:S 12 Jul 2020 13:47:47.304 * MASTER <-> REPLICA sync started
4286:S 12 Jul 2020 13:47:47.304 # Error condition on socket for SYNC: Connection refused
4286:S 12 Jul 2020 13:47:48.307 * Connecting to MASTER 172.30.2.184:6379
4286:S 12 Jul 2020 13:47:48.307 * MASTER <-> REPLICA sync started
4286:S 12 Jul 2020 13:47:48.307 # Error condition on socket for SYNC: Connection refused
4285:S 12 Jul 2020 13:47:48.307 * Connecting to MASTER 172.30.2.184:6379
4285:S 12 Jul 2020 13:47:48.307 * MASTER <-> REPLICA sync started
4285:S 12 Jul 2020 13:47:48.307 # Error condition on socket for SYNC: Connection refused
4285:S 12 Jul 2020 13:47:49.309 * Connecting to MASTER 172.30.2.184:6379
4285:S 12 Jul 2020 13:47:49.309 * MASTER <-> REPLICA sync started
4285:S 12 Jul 2020 13:47:49.309 # Error condition on socket for SYNC: Connection refused
4286:S 12 Jul 2020 13:47:49.310 * Connecting to MASTER 172.30.2.184:6379
4286:S 12 Jul 2020 13:47:49.310 * MASTER <-> REPLICA sync started
4286:S 12 Jul 2020 13:47:49.310 # Error condition on socket for SYNC: Connection refused
4287:X 12 Jul 2020 13:47:49.324 # +sdown master mymaster 172.30.2.184 6379
4288:X 12 Jul 2020 13:47:49.391 # +sdown master mymaster 172.30.2.184 6379
4288:X 12 Jul 2020 13:47:49.467 # +odown master mymaster 172.30.2.184 6379 #quorum 2/2
4288:X 12 Jul 2020 13:47:49.467 # +new-epoch 1
4288:X 12 Jul 2020 13:47:49.467 # +try-failover master mymaster 172.30.2.184 6379
4288:X 12 Jul 2020 13:47:49.469 # +vote-for-leader 9a72799337b637763343e01e971a580d51c7d4bf 1
4287:X 12 Jul 2020 13:47:49.471 # +new-epoch 1
4287:X 12 Jul 2020 13:47:49.472 # +vote-for-leader 9a72799337b637763343e01e971a580d51c7d4bf 1
4288:X 12 Jul 2020 13:47:49.472 # 9a72799337b637763343e01e971a580d51c7d4be voted for 9a72799337b637763343e01e971a580d51c7d4bf 1
4288:X 12 Jul 2020 13:47:49.552 # +elected-leader master mymaster 172.30.2.184 6379
4288:X 12 Jul 2020 13:47:49.552 # +failover-state-select-slave master mymaster 172.30.2.184 6379
4288:X 12 Jul 2020 13:47:49.619 # +selected-slave slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4288:X 12 Jul 2020 13:47:49.619 * +failover-state-send-slaveof-noone slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4288:X 12 Jul 2020 13:47:49.674 * +failover-state-wait-promotion slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4285:M 12 Jul 2020 13:47:49.674 # Setting secondary replication ID to dea4e6d9164f1ca225b8f4d5277fa3a98400283f, valid up to offset: 14064. New replication ID is 177f6a24d9489e62371172980a6e3c99b7a38a7e
4285:M 12 Jul 2020 13:47:49.674 * Discarding previously cached master state.
4285:M 12 Jul 2020 13:47:49.674 * MASTER MODE enabled (user request from 'id=6 addr=172.30.2.184:48484 fd=12 name=sentinel-9a727993-cmd age=122 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=140 qbuf-free=32628 obl=36 oll=0 omem=0 events=r cmd=exec')
4285:M 12 Jul 2020 13:47:49.676 # CONFIG REWRITE executed with success.
4286:S 12 Jul 2020 13:47:50.312 * Connecting to MASTER 172.30.2.184:6379
4286:S 12 Jul 2020 13:47:50.312 * MASTER <-> REPLICA sync started
4286:S 12 Jul 2020 13:47:50.312 # Error condition on socket for SYNC: Connection refused
4287:X 12 Jul 2020 13:47:50.482 # +odown master mymaster 172.30.2.184 6379 #quorum 2/2
4287:X 12 Jul 2020 13:47:50.482 # Next failover delay: I will not start a failover before Sun Jul 12 13:53:49 2020
4288:X 12 Jul 2020 13:47:50.668 # +promoted-slave slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4288:X 12 Jul 2020 13:47:50.668 # +failover-state-reconf-slaves master mymaster 172.30.2.184 6379
4288:X 12 Jul 2020 13:47:50.725 * +slave-reconf-sent slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
4286:S 12 Jul 2020 13:47:50.725 * REPLICAOF 172.30.2.184:6380 enabled (user request from 'id=7 addr=172.30.2.184:50798 fd=12 name=sentinel-9a727993-cmd age=123 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=291 qbuf-free=32477 obl=36 oll=0 omem=0 events=r cmd=exec')
4286:S 12 Jul 2020 13:47:50.726 # CONFIG REWRITE executed with success.
4287:X 12 Jul 2020 13:47:50.726 # +config-update-from sentinel 9a72799337b637763343e01e971a580d51c7d4bf 172.30.2.184 26380 @ mymaster 172.30.2.184 6379
4287:X 12 Jul 2020 13:47:50.726 # +switch-master mymaster 172.30.2.184 6379 172.30.2.184 6380
4287:X 12 Jul 2020 13:47:50.726 * +slave slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4287:X 12 Jul 2020 13:47:50.726 * +slave slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
4286:S 12 Jul 2020 13:47:51.314 * Connecting to MASTER 172.30.2.184:6380
4286:S 12 Jul 2020 13:47:51.314 * MASTER <-> REPLICA sync started
4286:S 12 Jul 2020 13:47:51.314 * Non blocking connect for SYNC fired the event.
4286:S 12 Jul 2020 13:47:51.314 * Master replied to PING, replication can continue...
4286:S 12 Jul 2020 13:47:51.315 * Trying a partial resynchronization (request dea4e6d9164f1ca225b8f4d5277fa3a98400283f:14064).
4285:M 12 Jul 2020 13:47:51.315 * Replica 172.30.2.184:6381 asks for synchronization
4285:M 12 Jul 2020 13:47:51.315 * Partial resynchronization request from 172.30.2.184:6381 accepted. Sending 579 bytes of backlog starting from offset 14064.
4286:S 12 Jul 2020 13:47:51.315 * Successful partial resynchronization with master.
4286:S 12 Jul 2020 13:47:51.315 # Master replication ID changed to 177f6a24d9489e62371172980a6e3c99b7a38a7e
4286:S 12 Jul 2020 13:47:51.315 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.
4288:X 12 Jul 2020 13:47:51.600 # -odown master mymaster 172.30.2.184 6379
4288:X 12 Jul 2020 13:47:51.676 * +slave-reconf-inprog slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
4288:X 12 Jul 2020 13:47:51.677 * +slave-reconf-done slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
4288:X 12 Jul 2020 13:47:51.747 # +failover-end master mymaster 172.30.2.184 6379
4288:X 12 Jul 2020 13:47:51.747 # +switch-master mymaster 172.30.2.184 6379 172.30.2.184 6380
4288:X 12 Jul 2020 13:47:51.747 * +slave slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4288:X 12 Jul 2020 13:47:51.747 * +slave slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
[ec2-user@ip-172-30-2-184 redis-5.0.8]$ 4287:X 12 Jul 2020 13:48:10.778 # +sdown slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
4288:X 12 Jul 2020 13:48:11.788 # +sdown slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
[ec2-user@ip-172-30-2-184 redis-5.0.8]$
10秒
[ec2-user@ip-172-30-2-184 redis-5.0.8]$ src/redis-cli shutdown
4385:M 12 Jul 2020 13:52:48.652 # User requested shutdown...
4385:M 12 Jul 2020 13:52:48.652 * Saving the final RDB snapshot before exiting.
4385:M 12 Jul 2020 13:52:48.654 * DB saved on disk
4385:M 12 Jul 2020 13:52:48.654 * Removing the pid file.
4385:M 12 Jul 2020 13:52:48.654 # Redis is now ready to exit, bye bye...
4386:S 12 Jul 2020 13:52:48.654 # Connection with master lost.
4386:S 12 Jul 2020 13:52:48.654 * Caching the disconnected master state.
4387:S 12 Jul 2020 13:52:48.654 # Connection with master lost.
4387:S 12 Jul 2020 13:52:48.654 * Caching the disconnected master state.
[ec2-user@ip-172-30-2-184 redis-5.0.8]$ 4387:S 12 Jul 2020 13:52:48.965 * Connecting to MASTER 172.30.2.184:6379
4387:S 12 Jul 2020 13:52:48.965 * MASTER <-> REPLICA sync started
4387:S 12 Jul 2020 13:52:48.965 # Error condition on socket for SYNC: Connection refused
4386:S 12 Jul 2020 13:52:48.967 * Connecting to MASTER 172.30.2.184:6379
4386:S 12 Jul 2020 13:52:48.967 * MASTER <-> REPLICA sync started
4386:S 12 Jul 2020 13:52:48.967 # Error condition on socket for SYNC: Connection refused
4387:S 12 Jul 2020 13:52:49.967 * Connecting to MASTER 172.30.2.184:6379
4387:S 12 Jul 2020 13:52:49.967 * MASTER <-> REPLICA sync started
4387:S 12 Jul 2020 13:52:49.967 # Error condition on socket for SYNC: Connection refused
4386:S 12 Jul 2020 13:52:49.969 * Connecting to MASTER 172.30.2.184:6379
4386:S 12 Jul 2020 13:52:49.969 * MASTER <-> REPLICA sync started
4386:S 12 Jul 2020 13:52:49.969 # Error condition on socket for SYNC: Connection refused
4387:S 12 Jul 2020 13:52:50.968 * Connecting to MASTER 172.30.2.184:6379
4387:S 12 Jul 2020 13:52:50.968 * MASTER <-> REPLICA sync started
4387:S 12 Jul 2020 13:52:50.969 # Error condition on socket for SYNC: Connection refused
4386:S 12 Jul 2020 13:52:50.970 * Connecting to MASTER 172.30.2.184:6379
4386:S 12 Jul 2020 13:52:50.970 * MASTER <-> REPLICA sync started
4386:S 12 Jul 2020 13:52:50.970 # Error condition on socket for SYNC: Connection refused
4386:S 12 Jul 2020 13:52:51.972 * Connecting to MASTER 172.30.2.184:6379
4386:S 12 Jul 2020 13:52:51.972 * MASTER <-> REPLICA sync started
4386:S 12 Jul 2020 13:52:51.972 # Error condition on socket for SYNC: Connection refused
4387:S 12 Jul 2020 13:52:51.972 * Connecting to MASTER 172.30.2.184:6379
4387:S 12 Jul 2020 13:52:51.972 * MASTER <-> REPLICA sync started
4387:S 12 Jul 2020 13:52:51.972 # Error condition on socket for SYNC: Connection refused
4386:S 12 Jul 2020 13:52:52.975 * Connecting to MASTER 172.30.2.184:6379
4386:S 12 Jul 2020 13:52:52.975 * MASTER <-> REPLICA sync started
4386:S 12 Jul 2020 13:52:52.975 # Error condition on socket for SYNC: Connection refused
4387:S 12 Jul 2020 13:52:52.975 * Connecting to MASTER 172.30.2.184:6379
4387:S 12 Jul 2020 13:52:52.975 * MASTER <-> REPLICA sync started
4387:S 12 Jul 2020 13:52:52.975 # Error condition on socket for SYNC: Connection refused
4386:S 12 Jul 2020 13:52:53.977 * Connecting to MASTER 172.30.2.184:6379
4386:S 12 Jul 2020 13:52:53.977 * MASTER <-> REPLICA sync started
4386:S 12 Jul 2020 13:52:53.977 # Error condition on socket for SYNC: Connection refused
4387:S 12 Jul 2020 13:52:53.977 * Connecting to MASTER 172.30.2.184:6379
4387:S 12 Jul 2020 13:52:53.977 * MASTER <-> REPLICA sync started
4387:S 12 Jul 2020 13:52:53.977 # Error condition on socket for SYNC: Connection refused
4386:S 12 Jul 2020 13:52:54.979 * Connecting to MASTER 172.30.2.184:6379
4386:S 12 Jul 2020 13:52:54.979 * MASTER <-> REPLICA sync started
4386:S 12 Jul 2020 13:52:54.979 # Error condition on socket for SYNC: Connection refused
4387:S 12 Jul 2020 13:52:54.980 * Connecting to MASTER 172.30.2.184:6379
4387:S 12 Jul 2020 13:52:54.980 * MASTER <-> REPLICA sync started
4387:S 12 Jul 2020 13:52:54.980 # Error condition on socket for SYNC: Connection refused
4386:S 12 Jul 2020 13:52:55.981 * Connecting to MASTER 172.30.2.184:6379
4386:S 12 Jul 2020 13:52:55.981 * MASTER <-> REPLICA sync started
4386:S 12 Jul 2020 13:52:55.981 # Error condition on socket for SYNC: Connection refused
4387:S 12 Jul 2020 13:52:55.982 * Connecting to MASTER 172.30.2.184:6379
4387:S 12 Jul 2020 13:52:55.982 * MASTER <-> REPLICA sync started
4387:S 12 Jul 2020 13:52:55.982 # Error condition on socket for SYNC: Connection refused
4386:S 12 Jul 2020 13:52:56.983 * Connecting to MASTER 172.30.2.184:6379
4386:S 12 Jul 2020 13:52:56.983 * MASTER <-> REPLICA sync started
4386:S 12 Jul 2020 13:52:56.983 # Error condition on socket for SYNC: Connection refused
4387:S 12 Jul 2020 13:52:56.984 * Connecting to MASTER 172.30.2.184:6379
4387:S 12 Jul 2020 13:52:56.984 * MASTER <-> REPLICA sync started
4387:S 12 Jul 2020 13:52:56.984 # Error condition on socket for SYNC: Connection refused
4386:S 12 Jul 2020 13:52:57.985 * Connecting to MASTER 172.30.2.184:6379
4386:S 12 Jul 2020 13:52:57.985 * MASTER <-> REPLICA sync started
4386:S 12 Jul 2020 13:52:57.985 # Error condition on socket for SYNC: Connection refused
4387:S 12 Jul 2020 13:52:57.986 * Connecting to MASTER 172.30.2.184:6379
4387:S 12 Jul 2020 13:52:57.986 * MASTER <-> REPLICA sync started
4387:S 12 Jul 2020 13:52:57.986 # Error condition on socket for SYNC: Connection refused
4388:X 12 Jul 2020 13:52:58.721 # +sdown master mymaster 172.30.2.184 6379
4389:X 12 Jul 2020 13:52:58.722 # +sdown master mymaster 172.30.2.184 6379
4389:X 12 Jul 2020 13:52:58.785 # +odown master mymaster 172.30.2.184 6379 #quorum 2/2
4389:X 12 Jul 2020 13:52:58.785 # +new-epoch 1
4389:X 12 Jul 2020 13:52:58.785 # +try-failover master mymaster 172.30.2.184 6379
4389:X 12 Jul 2020 13:52:58.787 # +vote-for-leader 9a72799337b637763343e01e971a580d51c7d4bf 1
4388:X 12 Jul 2020 13:52:58.789 # +new-epoch 1
4388:X 12 Jul 2020 13:52:58.790 # +vote-for-leader 9a72799337b637763343e01e971a580d51c7d4bf 1
4389:X 12 Jul 2020 13:52:58.790 # 9a72799337b637763343e01e971a580d51c7d4be voted for 9a72799337b637763343e01e971a580d51c7d4bf 1
4389:X 12 Jul 2020 13:52:58.877 # +elected-leader master mymaster 172.30.2.184 6379
4389:X 12 Jul 2020 13:52:58.877 # +failover-state-select-slave master mymaster 172.30.2.184 6379
4389:X 12 Jul 2020 13:52:58.949 # +selected-slave slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4389:X 12 Jul 2020 13:52:58.949 * +failover-state-send-slaveof-noone slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4386:S 12 Jul 2020 13:52:58.987 * Connecting to MASTER 172.30.2.184:6379
4386:S 12 Jul 2020 13:52:58.987 * MASTER <-> REPLICA sync started
4386:S 12 Jul 2020 13:52:58.987 # Error condition on socket for SYNC: Connection refused
4387:S 12 Jul 2020 13:52:58.987 * Connecting to MASTER 172.30.2.184:6379
4387:S 12 Jul 2020 13:52:58.987 * MASTER <-> REPLICA sync started
4387:S 12 Jul 2020 13:52:58.987 # Error condition on socket for SYNC: Connection refused
4389:X 12 Jul 2020 13:52:59.039 * +failover-state-wait-promotion slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4386:M 12 Jul 2020 13:52:59.039 # Setting secondary replication ID to 09763946986da00e53793f0e69a0650277b1840c, valid up to offset: 19402. New replication ID is a1b501dea123c55bc1e1bb992e3a5cb6d52ddb57
4386:M 12 Jul 2020 13:52:59.039 * Discarding previously cached master state.
4386:M 12 Jul 2020 13:52:59.039 * MASTER MODE enabled (user request from 'id=4 addr=172.30.2.184:49574 fd=10 name=sentinel-9a727993-cmd age=152 idle=1 flags=x db=0 sub=0 psub=0 multi=3 qbuf=140 qbuf-free=32628 obl=36 oll=0 omem=0 events=r cmd=exec')
4386:M 12 Jul 2020 13:52:59.040 # CONFIG REWRITE executed with success.
4389:X 12 Jul 2020 13:52:59.669 # +promoted-slave slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4389:X 12 Jul 2020 13:52:59.669 # +failover-state-reconf-slaves master mymaster 172.30.2.184 6379
4389:X 12 Jul 2020 13:52:59.738 * +slave-reconf-sent slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
4387:S 12 Jul 2020 13:52:59.738 * REPLICAOF 172.30.2.184:6380 enabled (user request from 'id=6 addr=172.30.2.184:51888 fd=12 name=sentinel-9a727993-cmd age=152 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=291 qbuf-free=32477 obl=36 oll=0 omem=0 events=r cmd=exec')
4387:S 12 Jul 2020 13:52:59.739 # CONFIG REWRITE executed with success.
4388:X 12 Jul 2020 13:52:59.739 # +config-update-from sentinel 9a72799337b637763343e01e971a580d51c7d4bf 172.30.2.184 26380 @ mymaster 172.30.2.184 6379
4388:X 12 Jul 2020 13:52:59.739 # +switch-master mymaster 172.30.2.184 6379 172.30.2.184 6380
4388:X 12 Jul 2020 13:52:59.740 * +slave slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4388:X 12 Jul 2020 13:52:59.740 * +slave slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
4389:X 12 Jul 2020 13:52:59.881 # -odown master mymaster 172.30.2.184 6379
4387:S 12 Jul 2020 13:52:59.990 * Connecting to MASTER 172.30.2.184:6380
4387:S 12 Jul 2020 13:52:59.990 * MASTER <-> REPLICA sync started
4387:S 12 Jul 2020 13:52:59.990 * Non blocking connect for SYNC fired the event.
4387:S 12 Jul 2020 13:52:59.990 * Master replied to PING, replication can continue...
4387:S 12 Jul 2020 13:52:59.990 * Trying a partial resynchronization (request 09763946986da00e53793f0e69a0650277b1840c:19402).
4386:M 12 Jul 2020 13:52:59.990 * Replica 172.30.2.184:6381 asks for synchronization
4386:M 12 Jul 2020 13:52:59.990 * Partial resynchronization request from 172.30.2.184:6381 accepted. Sending 440 bytes of backlog starting from offset 19402.
4387:S 12 Jul 2020 13:52:59.990 * Successful partial resynchronization with master.
4387:S 12 Jul 2020 13:52:59.990 # Master replication ID changed to a1b501dea123c55bc1e1bb992e3a5cb6d52ddb57
4387:S 12 Jul 2020 13:52:59.990 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.
4389:X 12 Jul 2020 13:53:00.679 * +slave-reconf-inprog slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
4389:X 12 Jul 2020 13:53:00.679 * +slave-reconf-done slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
4389:X 12 Jul 2020 13:53:00.754 # +failover-end master mymaster 172.30.2.184 6379
4389:X 12 Jul 2020 13:53:00.754 # +switch-master mymaster 172.30.2.184 6379 172.30.2.184 6380
4389:X 12 Jul 2020 13:53:00.755 * +slave slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4389:X 12 Jul 2020 13:53:00.755 * +slave slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
4388:X 12 Jul 2020 13:53:09.804 # +sdown slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
4389:X 12 Jul 2020 13:53:10.770 # +sdown slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
[ec2-user@ip-172-30-2-184 redis-5.0.8]$
1秒
src/redis-cli shutdown
4586:M 12 Jul 2020 14:03:42.757 # User requested shutdown...
4586:M 12 Jul 2020 14:03:42.757 * Saving the final RDB snapshot before exiting.
4586:M 12 Jul 2020 14:03:42.758 * DB saved on disk
4586:M 12 Jul 2020 14:03:42.758 * Removing the pid file.
4586:M 12 Jul 2020 14:03:42.758 # Redis is now ready to exit, bye bye...
4588:S 12 Jul 2020 14:03:42.759 # Connection with master lost.
4588:S 12 Jul 2020 14:03:42.759 * Caching the disconnected master state.
4587:S 12 Jul 2020 14:03:42.759 # Connection with master lost.
4587:S 12 Jul 2020 14:03:42.759 * Caching the disconnected master state.
[ec2-user@ip-172-30-2-184 redis-5.0.8]$ 4590:X 12 Jul 2020 14:03:42.843 # +sdown master mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:42.863 # +sdown master mymaster 172.30.2.184 6379
4587:S 12 Jul 2020 14:03:42.874 * Connecting to MASTER 172.30.2.184:6379
4587:S 12 Jul 2020 14:03:42.874 * MASTER <-> REPLICA sync started
4587:S 12 Jul 2020 14:03:42.874 # Error condition on socket for SYNC: Connection refused
4588:S 12 Jul 2020 14:03:42.875 * Connecting to MASTER 172.30.2.184:6379
4588:S 12 Jul 2020 14:03:42.875 * MASTER <-> REPLICA sync started
4588:S 12 Jul 2020 14:03:42.875 # Error condition on socket for SYNC: Connection refused
4589:X 12 Jul 2020 14:03:42.963 # +odown master mymaster 172.30.2.184 6379 #quorum 2/2
4589:X 12 Jul 2020 14:03:42.963 # +new-epoch 1
4589:X 12 Jul 2020 14:03:42.963 # +try-failover master mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:42.965 # +vote-for-leader 9a72799337b637763343e01e971a580d51c7d4be 1
4590:X 12 Jul 2020 14:03:42.967 # +new-epoch 1
4590:X 12 Jul 2020 14:03:42.968 # +vote-for-leader 9a72799337b637763343e01e971a580d51c7d4be 1
4589:X 12 Jul 2020 14:03:42.968 # 9a72799337b637763343e01e971a580d51c7d4bf voted for 9a72799337b637763343e01e971a580d51c7d4be 1
4589:X 12 Jul 2020 14:03:43.027 # +elected-leader master mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:43.027 # +failover-state-select-slave master mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:43.127 # +selected-slave slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:43.127 * +failover-state-send-slaveof-noone slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:43.227 * +failover-state-wait-promotion slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4587:M 12 Jul 2020 14:03:43.227 # Setting secondary replication ID to fbc14afa8e4edbac6ef51b8c05d04ad7ba822ba1, valid up to offset: 13772. New replication ID is 59f3c39228e700cc9ef8d7d8e8af625f54704301
4587:M 12 Jul 2020 14:03:43.227 * Discarding previously cached master state.
4587:M 12 Jul 2020 14:03:43.227 * MASTER MODE enabled (user request from 'id=12 addr=172.30.2.184:53188 fd=9 name=sentinel-9a727993-cmd age=25 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=154 qbuf-free=32614 obl=36 oll=0 omem=0 events=r cmd=exec')
4587:M 12 Jul 2020 14:03:43.228 # CONFIG REWRITE executed with success.
4588:S 12 Jul 2020 14:03:43.881 * Connecting to MASTER 172.30.2.184:6379
4588:S 12 Jul 2020 14:03:43.881 * MASTER <-> REPLICA sync started
4588:S 12 Jul 2020 14:03:43.881 # Error condition on socket for SYNC: Connection refused
4590:X 12 Jul 2020 14:03:43.971 # +odown master mymaster 172.30.2.184 6379 #quorum 2/2
4590:X 12 Jul 2020 14:03:43.971 # Next failover delay: I will not start a failover before Sun Jul 12 14:09:43 2020
4589:X 12 Jul 2020 14:03:44.007 # +promoted-slave slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:44.007 # +failover-state-reconf-slaves master mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:44.088 * +slave-reconf-sent slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
4588:S 12 Jul 2020 14:03:44.088 * REPLICAOF 172.30.2.184:6380 enabled (user request from 'id=10 addr=172.30.2.184:55490 fd=9 name=sentinel-9a727993-cmd age=68 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=305 qbuf-free=32463 obl=36 oll=0 omem=0 events=r cmd=exec')
4588:S 12 Jul 2020 14:03:44.089 # CONFIG REWRITE executed with success.
4590:X 12 Jul 2020 14:03:44.089 # +config-update-from sentinel 9a72799337b637763343e01e971a580d51c7d4be 172.30.2.184 26379 @ mymaster 172.30.2.184 6379
4590:X 12 Jul 2020 14:03:44.089 # +switch-master mymaster 172.30.2.184 6379 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:44.089 * +slave slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:44.090 * +slave slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:44.118 # +sdown master mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:44.119 # +odown master mymaster 172.30.2.184 6380 #quorum 2/2
4590:X 12 Jul 2020 14:03:44.119 # +new-epoch 2
4590:X 12 Jul 2020 14:03:44.119 # +try-failover master mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:44.120 # +vote-for-leader 9a72799337b637763343e01e971a580d51c7d4bf 2
4590:X 12 Jul 2020 14:03:44.121 # +sdown slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:44.121 # +sdown slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:44.180 # -odown master mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:44.180 # -sdown slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:44.453 # +new-epoch 2
4588:S 12 Jul 2020 14:03:44.883 * Connecting to MASTER 172.30.2.184:6380
4588:S 12 Jul 2020 14:03:44.883 * MASTER <-> REPLICA sync started
4588:S 12 Jul 2020 14:03:44.883 * Non blocking connect for SYNC fired the event.
4588:S 12 Jul 2020 14:03:44.883 * Master replied to PING, replication can continue...
4588:S 12 Jul 2020 14:03:44.884 * Trying a partial resynchronization (request fbc14afa8e4edbac6ef51b8c05d04ad7ba822ba1:13772).
4587:M 12 Jul 2020 14:03:44.884 * Replica 172.30.2.184:6381 asks for synchronization
4587:M 12 Jul 2020 14:03:44.884 * Partial resynchronization request from 172.30.2.184:6381 accepted. Sending 301 bytes of backlog starting from offset 13772.
4588:S 12 Jul 2020 14:03:44.884 * Successful partial resynchronization with master.
4588:S 12 Jul 2020 14:03:44.884 # Master replication ID changed to 59f3c39228e700cc9ef8d7d8e8af625f54704301
4588:S 12 Jul 2020 14:03:44.884 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.
4590:X 12 Jul 2020 14:03:44.915 # -sdown master mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.091 * +slave-reconf-inprog slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:45.091 * +slave-reconf-done slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:45.157 # +failover-end master mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:45.157 # +sdown sentinel 9a72799337b637763343e01e971a580d51c7d4bf 172.30.2.184 26380 @ mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:45.157 # +switch-master mymaster 172.30.2.184 6379 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.157 * +slave slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.158 * +slave slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.242 # +sdown master mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.242 # +odown master mymaster 172.30.2.184 6380 #quorum 2/2
4589:X 12 Jul 2020 14:03:45.242 # +new-epoch 3
4589:X 12 Jul 2020 14:03:45.242 # +try-failover master mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.244 # +vote-for-leader 9a72799337b637763343e01e971a580d51c7d4be 3
4589:X 12 Jul 2020 14:03:45.244 # +sdown slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.244 # +sdown slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.244 # -sdown sentinel 9a72799337b637763343e01e971a580d51c7d4bf 172.30.2.184 26380 @ mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:45.246 # +new-epoch 3
4590:X 12 Jul 2020 14:03:45.247 # +vote-for-leader 9a72799337b637763343e01e971a580d51c7d4be 3
4589:X 12 Jul 2020 14:03:45.247 # 9a72799337b637763343e01e971a580d51c7d4bf voted for 9a72799337b637763343e01e971a580d51c7d4be 3
4589:X 12 Jul 2020 14:03:45.310 # -odown master mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.310 # +elected-leader master mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.310 # +failover-state-select-slave master mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.310 # -sdown slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.393 # +selected-slave slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.393 * +failover-state-send-slaveof-noone slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:45.477 * +failover-state-wait-promotion slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4588:M 12 Jul 2020 14:03:45.477 # Setting secondary replication ID to 59f3c39228e700cc9ef8d7d8e8af625f54704301, valid up to offset: 14073. New replication ID is c8bdae796932d3bd3426ccebb62cae11a76abe89
4588:M 12 Jul 2020 14:03:45.477 # Connection with master lost.
4588:M 12 Jul 2020 14:03:45.477 * Caching the disconnected master state.
4588:M 12 Jul 2020 14:03:45.477 * Discarding previously cached master state.
4588:M 12 Jul 2020 14:03:45.477 * MASTER MODE enabled (user request from 'id=15 addr=172.30.2.184:55570 fd=9 name=sentinel-9a727993-cmd age=0 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=154 qbuf-free=32614 obl=36 oll=0 omem=0 events=r cmd=exec')
4587:M 12 Jul 2020 14:03:45.478 # Connection with replica 172.30.2.184:6381 lost.
4588:M 12 Jul 2020 14:03:45.478 # CONFIG REWRITE executed with success.
4589:X 12 Jul 2020 14:03:46.041 # -sdown master mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:46.250 # +promoted-slave slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:46.250 # +failover-state-reconf-slaves master mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:46.303 # +failover-end master mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:46.303 # +switch-master mymaster 172.30.2.184 6380 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:46.303 * +slave slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:46.304 * +slave slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
4590:X 12 Jul 2020 14:03:46.305 # +config-update-from sentinel 9a72799337b637763343e01e971a580d51c7d4be 172.30.2.184 26379 @ mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:46.305 # +switch-master mymaster 172.30.2.184 6380 172.30.2.184 6381
4590:X 12 Jul 2020 14:03:46.305 * +slave slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6381
4590:X 12 Jul 2020 14:03:46.306 * +slave slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
4590:X 12 Jul 2020 14:03:46.366 # +sdown master mymaster 172.30.2.184 6381
4590:X 12 Jul 2020 14:03:46.366 # +sdown slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
4590:X 12 Jul 2020 14:03:46.366 # +sdown slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:46.389 # +sdown master mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:46.389 # +sdown slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:46.389 # +sdown slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:46.466 # +odown master mymaster 172.30.2.184 6381 #quorum 2/2
4589:X 12 Jul 2020 14:03:46.466 # +new-epoch 4
4589:X 12 Jul 2020 14:03:46.466 # +try-failover master mymaster 172.30.2.184 6381
4590:X 12 Jul 2020 14:03:46.467 # -sdown master mymaster 172.30.2.184 6381
4590:X 12 Jul 2020 14:03:46.467 # -sdown slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:46.467 # +vote-for-leader 9a72799337b637763343e01e971a580d51c7d4be 4
4589:X 12 Jul 2020 14:03:46.468 # -sdown slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
4590:X 12 Jul 2020 14:03:46.469 # +new-epoch 4
4590:X 12 Jul 2020 14:03:46.470 # +vote-for-leader 9a72799337b637763343e01e971a580d51c7d4be 4
4589:X 12 Jul 2020 14:03:46.470 # 9a72799337b637763343e01e971a580d51c7d4bf voted for 9a72799337b637763343e01e971a580d51c7d4be 4
4589:X 12 Jul 2020 14:03:46.521 # -odown master mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:46.521 # +elected-leader master mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:46.521 # +failover-state-select-slave master mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:46.573 # +selected-slave slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:46.573 * +failover-state-send-slaveof-noone slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:46.625 * +failover-state-wait-promotion slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
4587:M 12 Jul 2020 14:03:46.626 # CONFIG REWRITE executed with success.
4590:X 12 Jul 2020 14:03:46.649 # +sdown slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:47.070 # -sdown master mymaster 172.30.2.184 6381
4590:X 12 Jul 2020 14:03:47.402 # -sdown slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:47.423 # +promoted-slave slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:47.423 # +failover-state-reconf-slaves master mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:47.512 # +failover-end master mymaster 172.30.2.184 6381
4589:X 12 Jul 2020 14:03:47.512 # +switch-master mymaster 172.30.2.184 6381 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:47.512 * +slave slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:47.512 * +slave slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:47.514 # +config-update-from sentinel 9a72799337b637763343e01e971a580d51c7d4be 172.30.2.184 26379 @ mymaster 172.30.2.184 6381
4590:X 12 Jul 2020 14:03:47.514 # +switch-master mymaster 172.30.2.184 6381 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:47.514 * +slave slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:47.514 * +slave slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:47.580 # +sdown master mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:47.580 # +sdown slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:47.580 # +sdown slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:47.594 # +sdown master mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:47.594 # +sdown slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:47.594 # +sdown slave 172.30.2.184:6379 172.30.2.184 6379 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:47.633 # -sdown slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:47.656 # -sdown master mymaster 172.30.2.184 6380
4590:X 12 Jul 2020 14:03:47.656 # -sdown slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4589:X 12 Jul 2020 14:03:48.117 # -sdown master mymaster 172.30.2.184 6380
[ec2-user@ip-172-30-2-184 redis-5.0.8]$
[ec2-user@ip-172-30-2-184 redis-5.0.8]$
[ec2-user@ip-172-30-2-184 redis-5.0.8]$ 4589:X 12 Jul 2020 14:03:57.594 * +convert-to-slave slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6380
4588:S 12 Jul 2020 14:03:57.594 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
4588:S 12 Jul 2020 14:03:57.594 * REPLICAOF 172.30.2.184:6380 enabled (user request from 'id=22 addr=172.30.2.184:55626 fd=8 name=sentinel-9a727993-cmd age=10 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=152 qbuf-free=32616 obl=36 oll=0 omem=0 events=r cmd=exec')
4588:S 12 Jul 2020 14:03:57.595 # CONFIG REWRITE executed with success.
4588:S 12 Jul 2020 14:03:57.911 * Connecting to MASTER 172.30.2.184:6380
4588:S 12 Jul 2020 14:03:57.911 * MASTER <-> REPLICA sync started
4588:S 12 Jul 2020 14:03:57.911 * Non blocking connect for SYNC fired the event.
4588:S 12 Jul 2020 14:03:57.911 * Master replied to PING, replication can continue...
4588:S 12 Jul 2020 14:03:57.911 * Trying a partial resynchronization (request c8bdae796932d3bd3426ccebb62cae11a76abe89:15625).
4587:M 12 Jul 2020 14:03:57.911 * Replica 172.30.2.184:6381 asks for synchronization
4587:M 12 Jul 2020 14:03:57.911 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'c8bdae796932d3bd3426ccebb62cae11a76abe89', my replication IDs are '59f3c39228e700cc9ef8d7d8e8af625f54704301' and 'fbc14afa8e4edbac6ef51b8c05d04ad7ba822ba1')
4587:M 12 Jul 2020 14:03:57.911 * Starting BGSAVE for SYNC with target: disk
4587:M 12 Jul 2020 14:03:57.912 * Background saving started by pid 4627
4588:S 12 Jul 2020 14:03:57.912 * Full resync from master: 59f3c39228e700cc9ef8d7d8e8af625f54704301:15601
4588:S 12 Jul 2020 14:03:57.912 * Discarding previously cached master state.
4627:C 12 Jul 2020 14:03:57.914 * DB saved on disk
4627:C 12 Jul 2020 14:03:57.914 * RDB: 0 MB of memory used by copy-on-write
4587:M 12 Jul 2020 14:03:58.012 * Background saving terminated with success
4587:M 12 Jul 2020 14:03:58.012 * Synchronization with replica 172.30.2.184:6381 succeeded
4588:S 12 Jul 2020 14:03:58.012 * MASTER <-> REPLICA sync: receiving 197 bytes from master
4588:S 12 Jul 2020 14:03:58.012 * MASTER <-> REPLICA sync: Flushing old data
4588:S 12 Jul 2020 14:03:58.012 * MASTER <-> REPLICA sync: Loading DB in memory
4588:S 12 Jul 2020 14:03:58.012 * MASTER <-> REPLICA sync: Finished with success
[ec2-user@ip-172-30-2-184 redis-5.0.8]$
1ミリ秒
failover前から下記のようなログが出続けていて不安定。
[ec2-user@ip-172-30-2-184 redis-5.0.8]$ 4589:X 12 Jul 2020 14:02:28.378 # +sdown master mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:02:28.433 # -sdown master mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:02:32.453 # +sdown sentinel 9a72799337b637763343e01e971a580d51c7d4bf 172.30.2.184 26380 @ mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:02:32.544 # -sdown sentinel 9a72799337b637763343e01e971a580d51c7d4bf 172.30.2.184 26380 @ mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:02:36.272 # +sdown slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:02:36.356 # -sdown slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:02:36.843 # +sdown slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:02:36.896 # -sdown slave 172.30.2.184:6381 172.30.2.184 6381 @ mymaster 172.30.2.184 6379
4590:X 12 Jul 2020 14:02:59.727 # +sdown master mymaster 172.30.2.184 6379
4590:X 12 Jul 2020 14:02:59.793 # -sdown master mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:04.176 # +sdown master mymaster 172.30.2.184 6379
4589:X 12 Jul 2020 14:03:04.267 # -sdown master mymaster 172.30.2.184 6379
4590:X 12 Jul 2020 14:03:05.428 # +sdown slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
4590:X 12 Jul 2020 14:03:05.481 # -sdown slave 172.30.2.184:6380 172.30.2.184 6380 @ mymaster 172.30.2.184 6379
[ec2-user@ip-172-30-2-184 redis-5.0.8]$
0ミリ秒
Sentinelプロセスが起動しない。
[ec2-user@ip-172-30-2-184 redis-5.0.8]$ src/redis-sentinel ./sentinel1.conf
*** FATAL CONFIG FILE ERROR ***
Reading the configuration file, at line 108
>>> 'sentinel down-after-milliseconds mymaster 0'
negative or zero time parameter.
[ec2-user@ip-172-30-2-184 redis-5.0.8]$