課題

ReplicaSetでread_secondary => tureにしている時、読みに行くmongoはランダムに決定される。
これが失敗するとMongo::ConnectionFailureが発生するのだが、デフォルトだとそのままリトライせずにエラーとなる。

failover復旧時やindex作成時などsecondaryがblock状態の時にも、何度かリトライするようにしたい。

※注: background => trueで実行したときprimaryは確かにbackground実行になっているが、secondaryはblockされる

解決

mongoid.ymlにmax_retries_on_connection_failureオプションを追加する。

mongoid.yml

production:
  database: myapp
  hosts: [[100.100.100.100, 27017],
          [200.200.200.200, 27017],
          [300.300.300.300, 27017]]
  pool_size: 99
  read_secondary: true
  max_retries_on_connection_failure: 5

参考

Connection Failures

Imagine that either the master node or one of the read nodes goes offline. How will the driver respond?

If any read operation fails, the driver will raise a ConnectionFailure exception. It then becomes the client's responsibility to decide how to handle this.

If the client decides to retry, it's not guaranteed that another member of the replica set will have been promoted to master right away, so it's still possible that the driver will raise another ConnectionFailure. However, once a member has been promoted to master, typically within a few seconds, subsequent operations will succeed. Note that this does not prevent exception in the event of a primary failover.

The driver will essentially cycle through all known seed addresses until a node identifies itself as master.

Ruby Mongo drive

If you would also like Mongoid to retry operations if a Mongo::ConnectionFailure occurs you may specify this option in your config. Mongoid will retry the operation every half second up to the limit that is set. This defaults to 0.
max_retries_on_connection_failure: 3
Note that Mongoid does not handle reconnection at the time of this documentation, so you'll have to catch the connection errors and retry yourself in the meantime.

Mongoid公式ドキュメント

その他ソース

http://groups.google.com/group/mongoid/browse_thread/thread/700b5e0c5249baed?pli=1
http://stackoverflow.com/questions/10221198/mongooperationfailure-repeatedly-happening-in-ec2-what-to-do