1. 参考
-
Redis Sentinelで自動フェイルオーバー
- https://qiita.com/wellflat/items/8935016fdee25d4866d9
- ※この記事をかなり参考にさせていただいております。
-
Redis Sentinel Documentation - Redis
2. 今回作る構成(Redis Sentinel Documentationより引用)
+----+
| M1 |
| S1 |
+----+
|
+----+ | +----+
| R2 |----+----| R3 |
| S2 | | S3 |
+----+ +----+
Configuration: quorum = 2
Masters are called M1, M2, M3, ..., Mn.
replicas are called R1, R2, R3, ..., Rn (R stands for replica).
Sentinels are called S1, S2, S3, ..., Sn.
1号機:M1/S1 172.30.1.99
2号機:R2/S2 172.30.1.220
3号機:R3/S3 172.30.1.18
とする。
3. 構築
3.1. サーバ3台起動して、Redisをインストール
(参考)
RHELに最新バージョンのRedisをソースからコンパイルしてインストールした
https://qiita.com/taka_22/items/31b364491b4f36dc986e
セキュリティグループで開けるポートは、下記の2つ。
- 6379
- 26379
3.2. Replicationの設定
redis.confを下記の通り修正する。
3.2.1. bindの設定
(修正前)
bind 127.0.0.1
(修正後)
#bind 127.0.0.1
コメントアウトする
3.2.2. protected-modeの設定
(修正前)
protected-mode yes
(修正後)
protected-mode no
3.2.3. replicaの設定 ※2号機/3号機のみ※
(修正前)
replicaof <masterip> <masterport>
(修正後)
replicaof <masterip> <masterport>
replicaof [1号機のPrivateIP] 6379
※5.0からslaveof -> replicaofに変更されている
(参考)
RedisのReplicationをEC2で構築する
https://qiita.com/taka_22/items/a99796fc61dca55b1e11
3.3. Sentinelの設定
sentinel.confを下記の通り修正する。
3.3.1. protected-modeの設定
(修正前)
protected-mode no
(修正後)
protected-mode no
※コメントインする
3.3.2. sentinel-monitorの設定
(修正前)
sentinel monitor mymaster 127.0.0.1 6379 2
(修正後)
sentinel monitor mymaster [1号機のPrivateIP] 6379 2
4. プロセス起動
4.1. Redis Serverプロセスの起動
src/redis-server ./redis.conf &
[ec2-user@ip-172-30-1-99 redis-5.0.8]$ src/redis-server ./redis.conf &
[1] 3251
[ec2-user@ip-172-30-1-99 redis-5.0.8]$ 3251:C 27 Apr 2020 09:27:38.240 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3251:C 27 Apr 2020 09:27:38.240 # Redis version=5.0.8, bits=64, commit=00000000, modified=0, pid=3251, just started
3251:C 27 Apr 2020 09:27:38.240 # Configuration loaded
3251:M 27 Apr 2020 09:27:38.240 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
3251:M 27 Apr 2020 09:27:38.240 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
3251:M 27 Apr 2020 09:27:38.240 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 5.0.8 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 3251
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
3251:M 27 Apr 2020 09:27:38.246 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
3251:M 27 Apr 2020 09:27:38.246 # Server initialized
3251:M 27 Apr 2020 09:27:38.246 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
3251:M 27 Apr 2020 09:27:38.246 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
3251:M 27 Apr 2020 09:27:38.247 * DB loaded from disk: 0.001 seconds
3251:M 27 Apr 2020 09:27:38.247 * Ready to accept connections
[ec2-user@ip-172-30-1-99 redis-5.0.8]$
2号機、3号機も同様に起動。
4.2. Sentinelプロセスの起動
src/redis-sentinel ./sentinel.conf &
[ec2-user@ip-172-30-1-99 redis-5.0.8]$ src/redis-sentinel ./sentinel.conf &
[2] 3260
[ec2-user@ip-172-30-1-99 redis-5.0.8]$ 3260:X 27 Apr 2020 09:32:45.863 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3260:X 27 Apr 2020 09:32:45.863 # Redis version=5.0.8, bits=64, commit=00000000, modified=0, pid=3260, just started
3260:X 27 Apr 2020 09:32:45.863 # Configuration loaded
3260:X 27 Apr 2020 09:32:45.864 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
3260:X 27 Apr 2020 09:32:45.864 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
3260:X 27 Apr 2020 09:32:45.864 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 5.0.8 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in sentinel mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 26379
| `-._ `._ / _.-' | PID: 3260
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
3260:X 27 Apr 2020 09:32:45.867 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
3260:X 27 Apr 2020 09:32:45.868 # Sentinel ID is 662563f953cc81703472715d33226b3b8d66de01
3260:X 27 Apr 2020 09:32:45.868 # +monitor master mymaster 172.30.1.99 6379 quorum 2
[ec2-user@ip-172-30-1-99 redis-5.0.8]$
2号機、3号機も同様に起動。
5. 動作確認
5.1. 1号機のRedis Serverプロセスダウン
5.1.1. 実行前確認
1号機の操作
[ec2-user@ip-172-30-1-99 redis-5.0.8]$ src/redis-cli info replication
# Replication
role:master
connected_slaves:2
...
1号機がmasterであることを確認した。
5.1.2. 実行
1号機のRedis Serverプロセスをダウンさせる
1号機の操作
[ec2-user@ip-172-30-1-99 redis-5.0.8]$ src/redis-cli shutdown
3251:M 27 Apr 2020 09:44:10.632 # User requested shutdown...
3251:M 27 Apr 2020 09:44:10.632 * Saving the final RDB snapshot before exiting.
3251:M 27 Apr 2020 09:44:10.634 * DB saved on disk
3251:M 27 Apr 2020 09:44:10.634 * Removing the pid file.
3251:M 27 Apr 2020 09:44:10.634 # Redis is now ready to exit, bye bye...
[1]- Done src/redis-server ./redis.conf
[ec2-user@ip-172-30-1-99 redis-5.0.8]$ 3260:X 27 Apr 2020 09:44:40.729 # +sdown master mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:40.806 # +odown master mymaster 172.30.1.99 6379 #quorum 2/2
3260:X 27 Apr 2020 09:44:40.806 # +new-epoch 1
3260:X 27 Apr 2020 09:44:40.806 # +try-failover master mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:40.809 # +vote-for-leader 662563f953cc81703472715d33226b3b8d66de01 1
3260:X 27 Apr 2020 09:44:40.812 # 0291edcb56d1bd88f78612325d05cc305854cedd voted for 662563f953cc81703472715d33226b3b8d66de01 1
3260:X 27 Apr 2020 09:44:40.813 # 5f546bf1cae0e79981d2767ec9659aa320e488f6 voted for 662563f953cc81703472715d33226b3b8d66de01 1
3260:X 27 Apr 2020 09:44:40.875 # +elected-leader master mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:40.875 # +failover-state-select-slave master mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:40.934 # +selected-slave slave 172.30.1.18:6379 172.30.1.18 6379 @ mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:40.934 * +failover-state-send-slaveof-noone slave 172.30.1.18:6379 172.30.1.18 6379 @ mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:40.996 * +failover-state-wait-promotion slave 172.30.1.18:6379 172.30.1.18 6379 @ mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:41.386 # +promoted-slave slave 172.30.1.18:6379 172.30.1.18 6379 @ mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:41.386 # +failover-state-reconf-slaves master mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:41.467 * +slave-reconf-sent slave 172.30.1.220:6379 172.30.1.220 6379 @ mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:41.939 # -odown master mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:42.427 * +slave-reconf-inprog slave 172.30.1.220:6379 172.30.1.220 6379 @ mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:42.427 * +slave-reconf-done slave 172.30.1.220:6379 172.30.1.220 6379 @ mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:42.527 # +failover-end master mymaster 172.30.1.99 6379
3260:X 27 Apr 2020 09:44:42.527 # +switch-master mymaster 172.30.1.99 6379 172.30.1.18 6379
3260:X 27 Apr 2020 09:44:42.527 * +slave slave 172.30.1.220:6379 172.30.1.220 6379 @ mymaster 172.30.1.18 6379
3260:X 27 Apr 2020 09:44:42.527 * +slave slave 172.30.1.99:6379 172.30.1.99 6379 @ mymaster 172.30.1.18 6379
3260:X 27 Apr 2020 09:45:12.560 # +sdown slave 172.30.1.99:6379 172.30.1.99 6379 @ mymaster 172.30.1.18 6379
172.30.1.18:6379がmasterに昇格したことなどが読み取れる。
3号機のログ
[ec2-user@ip-172-30-1-18 redis-5.0.8]$ 3239:S 27 Apr 2020 09:40:19.153 # Timeout connecting to the MASTER...
3239:S 27 Apr 2020 09:40:19.153 * Connecting to MASTER 172.30.1.99:6379
3239:S 27 Apr 2020 09:40:19.153 * MASTER <-> REPLICA sync started
3247:X 27 Apr 2020 09:40:39.729 # -sdown master mymaster 172.30.1.99 6379
3247:X 27 Apr 2020 09:40:40.148 * +sentinel sentinel 662563f953cc81703472715d33226b3b8d66de01 172.30.1.99 26379 @ mymaster 172.30.1.99 6379
3247:X 27 Apr 2020 09:40:43.155 * +sentinel sentinel 0291edcb56d1bd88f78612325d05cc305854cedd 172.30.1.220 26379 @ mymaster 172.30.1.99 6379
3247:X 27 Apr 2020 09:40:49.715 * +slave slave 172.30.1.220:6379 172.30.1.220 6379 @ mymaster 172.30.1.99 6379
3239:S 27 Apr 2020 09:40:50.232 * Non blocking connect for SYNC fired the event.
3239:S 27 Apr 2020 09:40:50.232 * Master replied to PING, replication can continue...
3239:S 27 Apr 2020 09:40:50.232 * Partial resynchronization not possible (no cached master)
3239:S 27 Apr 2020 09:40:50.234 * Full resync from master: 028caa35c850faa14dc755b3c394768a6874faf1:7895
3239:S 27 Apr 2020 09:40:50.310 * MASTER <-> REPLICA sync: receiving 193 bytes from master
3239:S 27 Apr 2020 09:40:50.310 * MASTER <-> REPLICA sync: Flushing old data
3239:S 27 Apr 2020 09:40:50.310 * MASTER <-> REPLICA sync: Loading DB in memory
3239:S 27 Apr 2020 09:40:50.310 * MASTER <-> REPLICA sync: Finished with success
3247:X 27 Apr 2020 09:40:59.747 * +slave slave 172.30.1.18:6379 172.30.1.18 6379 @ mymaster 172.30.1.99 6379
3239:S 27 Apr 2020 09:44:10.636 # Connection with master lost.
3239:S 27 Apr 2020 09:44:10.636 * Caching the disconnected master state.
3239:S 27 Apr 2020 09:44:10.712 * Connecting to MASTER 172.30.1.99:6379
3239:S 27 Apr 2020 09:44:10.712 * MASTER <-> REPLICA sync started
3239:S 27 Apr 2020 09:44:10.712 # Error condition on socket for SYNC: Connection refused
3239:S 27 Apr 2020 09:44:11.714 * Connecting to MASTER 172.30.1.99:6379
...
3239:S 27 Apr 2020 09:44:38.777 * MASTER <-> REPLICA sync started
3239:S 27 Apr 2020 09:44:38.777 # Error condition on socket for SYNC: Connection refused
3239:S 27 Apr 2020 09:44:39.779 * Connecting to MASTER 172.30.1.99:6379
3239:S 27 Apr 2020 09:44:39.779 * MASTER <-> REPLICA sync started
3239:S 27 Apr 2020 09:44:39.779 # Error condition on socket for SYNC: Connection refused
3247:X 27 Apr 2020 09:44:40.764 # +sdown master mymaster 172.30.1.99 6379
3239:S 27 Apr 2020 09:44:40.782 * Connecting to MASTER 172.30.1.99:6379
3239:S 27 Apr 2020 09:44:40.782 * MASTER <-> REPLICA sync started
3239:S 27 Apr 2020 09:44:40.783 # Error condition on socket for SYNC: Connection refused
3247:X 27 Apr 2020 09:44:40.811 # +new-epoch 1
3247:X 27 Apr 2020 09:44:40.813 # +vote-for-leader 662563f953cc81703472715d33226b3b8d66de01 1
3247:X 27 Apr 2020 09:44:40.848 # +odown master mymaster 172.30.1.99 6379 #quorum 3/2
3247:X 27 Apr 2020 09:44:40.848 # Next failover delay: I will not start a failover before Mon Apr 27 09:50:41 2020
3239:M 27 Apr 2020 09:44:40.996 # Setting secondary replication ID to 028caa35c850faa14dc755b3c394768a6874faf1, valid up to offset: 48712. New replication ID is 0f36de4f0134dfe2ebb76519c7e740788999117e
3239:M 27 Apr 2020 09:44:40.997 * Discarding previously cached master state.
3239:M 27 Apr 2020 09:44:40.997 * MASTER MODE enabled (user request from 'id=5 addr=172.30.1.99:50726 fd=9 name=sentinel-662563f9-cmd age=223 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=140 qbuf-free=32628 obl=36 oll=0 omem=0 events=r cmd=exec')
3239:M 27 Apr 2020 09:44:40.998 # CONFIG REWRITE executed with success.
3247:X 27 Apr 2020 09:44:41.468 # +config-update-from sentinel 662563f953cc81703472715d33226b3b8d66de01 172.30.1.99 26379 @ mymaster 172.30.1.99 6379
3247:X 27 Apr 2020 09:44:41.468 # +switch-master mymaster 172.30.1.99 6379 172.30.1.18 6379
3247:X 27 Apr 2020 09:44:41.468 * +slave slave 172.30.1.220:6379 172.30.1.220 6379 @ mymaster 172.30.1.18 6379
3247:X 27 Apr 2020 09:44:41.468 * +slave slave 172.30.1.99:6379 172.30.1.99 6379 @ mymaster 172.30.1.18 6379
3239:M 27 Apr 2020 09:44:41.579 * Replica 172.30.1.220:6379 asks for synchronization
3239:M 27 Apr 2020 09:44:41.579 * Partial resynchronization request from 172.30.1.220:6379 accepted. Sending 160 bytes of backlog starting from offset 48712.
このサーバ(3号機)がmasterになったこと、172.30.1.99:6379から引き継がれたことなどが読み取れる。
5.1.3. 結果
3号機がmasterを自動で引き継いだ
3号機の操作
[ec2-user@ip-172-30-1-18 redis-5.0.8]$ src/redis-cli info replication
# Replication
role:master
connected_slaves:1
slave0:ip=172.30.1.220,port=6379,state=online,offset=95612,lag=0
master_replid:0f36de4f0134dfe2ebb76519c7e740788999117e
master_replid2:028caa35c850faa14dc755b3c394768a6874faf1
master_repl_offset:95612
second_repl_offset:48712
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:7896
repl_backlog_histlen:87717
[ec2-user@ip-172-30-1-18 redis-5.0.8]$
3号機がmasterであることを確認した。
5.1.4. フェイルオーバー後の構成
+----+
| M1 |
| S1 | <- C1 (writes will be lost)
+----+
|
/
/
+----+ | +------+
| R2 |------+----| [M3] |
| S2 | | S3 |
+----+ +------+
5.2. 1号機のRedis Serverプロセスの復旧
5.2.1. 実行
1号機のRedis Serverプロセスの起動
1号機の操作
[ec2-user@ip-172-30-1-99 redis-5.0.8]$ src/redis-server ./redis.conf &
[3] 3300
[ec2-user@ip-172-30-1-99 redis-5.0.8]$ 3300:C 27 Apr 2020 09:51:05.653 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3300:C 27 Apr 2020 09:51:05.653 # Redis version=5.0.8, bits=64, commit=00000000, modified=0, pid=3300, just started
3300:C 27 Apr 2020 09:51:05.653 # Configuration loaded
3300:M 27 Apr 2020 09:51:05.653 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
3300:M 27 Apr 2020 09:51:05.654 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
3300:M 27 Apr 2020 09:51:05.654 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 5.0.8 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 3300
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
3300:M 27 Apr 2020 09:51:05.654 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
3300:M 27 Apr 2020 09:51:05.654 # Server initialized
3300:M 27 Apr 2020 09:51:05.654 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
3300:M 27 Apr 2020 09:51:05.654 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
3300:M 27 Apr 2020 09:51:05.654 * DB loaded from disk: 0.000 seconds
3300:M 27 Apr 2020 09:51:05.654 * Ready to accept connections
3260:X 27 Apr 2020 09:51:06.315 # -sdown slave 172.30.1.99:6379 172.30.1.99 6379 @ mymaster 172.30.1.18 6379
3300:S 27 Apr 2020 09:51:15.898 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
3300:S 27 Apr 2020 09:51:15.898 * REPLICAOF 172.30.1.18:6379 enabled (user request from 'id=3 addr=172.30.1.220:32998 fd=8 name=sentinel-0291edcb-cmd age=10 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=151 qbuf-free=32617 obl=36 oll=0 omem=0 events=r cmd=exec')
3300:S 27 Apr 2020 09:51:15.899 # CONFIG REWRITE executed with success.
3300:S 27 Apr 2020 09:51:16.683 * Connecting to MASTER 172.30.1.18:6379
3300:S 27 Apr 2020 09:51:16.683 * MASTER <-> REPLICA sync started
3300:S 27 Apr 2020 09:51:16.683 * Non blocking connect for SYNC fired the event.
3300:S 27 Apr 2020 09:51:16.684 * Master replied to PING, replication can continue...
3300:S 27 Apr 2020 09:51:16.684 * Trying a partial resynchronization (request 2daae5e2e1885778659b505b1983a442eac04f83:1).
3300:S 27 Apr 2020 09:51:16.685 * Full resync from master: 0f36de4f0134dfe2ebb76519c7e740788999117e:129345
3300:S 27 Apr 2020 09:51:16.685 * Discarding previously cached master state.
3300:S 27 Apr 2020 09:51:16.695 * MASTER <-> REPLICA sync: receiving 195 bytes from master
3300:S 27 Apr 2020 09:51:16.695 * MASTER <-> REPLICA sync: Flushing old data
3300:S 27 Apr 2020 09:51:16.695 * MASTER <-> REPLICA sync: Loading DB in memory
3300:S 27 Apr 2020 09:51:16.695 * MASTER <-> REPLICA sync: Finished with success
[ec2-user@ip-172-30-1-99 redis-5.0.8]$
5.2.2. 結果
1号機の操作
[ec2-user@ip-172-30-1-99 redis-5.0.8]$ src/redis-cli info replication
# Replication
role:slave
master_host:172.30.1.18
master_port:6379
...
1号機がreplicaであることを確認した。
5.2.3. 1号機復旧後の構成
+----+
| R1 |
| S1 |
+----+
|
+----+ | +----+
| R2 |----+----| M3 |
| S2 | | S3 |
+----+ +----+