MongoDB 6.0 on Rocky Linux で Replication

ほんとは （日本語の解説記事がいくらでもあるし） MongoDB 5.0 を使おうと思ってたが、いざ構築段階になって見てみたら 6.0 がリリースされてたのでとりあえずやってみようというだけの記事。

OS は Rocky Linux の 2022/11/17 時点で最新版を利用

$ cat /etc/redhat-release
Rocky Linux release 8.7 (Green Obsidian)

（minimal install + yum update 済みのやつだけど、普段共通で入れてる package を事前にインストール済みなので、場合によっては dependency が出てくるかも）
なお、 面倒なので SELinux は Disabled にしている。

install

Rocky デフォルトのリポジトリだと MongoDB が入ってないので、適当に RHEL のリポジトリを追加 （お金無いので Community 版）

repository

$ cat /etc/yum.repos.d/mongodb-org-6.0.repo
[mongodb-org-6.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/6.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-6.0.asc

Ansible でインストールする時にバージョンを固定しておきたいのでバージョン名付きの名前を調べる

$ sudo dnf search --showduplicates mongodb-org
メタデータの期限切れの最終確認: 0:11:13 時間前の 2022年11月17日 03時52分33秒 に実施しました。
=================== 名前 完全一致: mongodb-org ====================
mongodb-org-6.0.3-1.el8.x86_64 : MongoDB open source
     ...: document-oriented database system (metapackage)
mongodb-org-6.0.0-1.el8.x86_64 : MongoDB open source
     ...: document-oriented database system (metapackage)
mongodb-org-6.0.1-1.el8.x86_64 : MongoDB open source
     ...: document-oriented database system (metapackage)
mongodb-org-6.0.2-1.el8.x86_64 : MongoDB open source
     ...: document-oriented database system (metapackage)
mongodb-org-6.0.3-1.el8.x86_64 : MongoDB open source
     ...: document-oriented database system (metapackage)

mongodb-org-6.0.3-1.el8.x86_64 にする。

install して起動してみる。

$ sudo systemctl start mongod
$ sudo systemctl status mongod
● mongod.service - MongoDB Database Server
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled>
   Active: active (running) since Thu 2022-11-17 04:26:28 EST; 3s >
     Docs: https://docs.mongodb.org/manual
  Process: 106174 ExecStart=/usr/bin/mongod $OPTIONS (code=exited,>
  Process: 106171 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongod>
  Process: 106170 ExecStartPre=/usr/bin/chown mongod:mongod /var/r>
  Process: 106168 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb >
 Main PID: 106177 (mongod)
   Memory: 144.3M
   CGroup: /system.slice/mongod.service
           └─106177 /usr/bin/mongod -f /etc/mongod.conf

11月 17 04:26:27 STMNG001 systemd[1]: Starting MongoDB Database Se>
11月 17 04:26:27 STMNG001 mongod[106174]: about to fork child proc>
11月 17 04:26:27 STMNG001 mongod[106177]: forked process: 106177
11月 17 04:26:28 STMNG001 mongod[106174]: child process started su>
11月 17 04:26:28 STMNG001 systemd[1]: Started MongoDB Database Ser>
lines 1-18/18 (END)

とりあえず動いた。
関係ないが、MongoDB は 5.0 から SandyBridge（AMD なら Bulldozer）以降の CPU じゃないと動かないらしく、古い検証機でやろうとしてたら全く起動しなくて無駄な時間を過ごしたことがある。

Replication の設定

ネットワーク構成

今回は primary 1 台、secondary 2 台の 3 replica 構成。

node1

    inet 192.168.163.41/24 brd 192.168.163.255 scope global noprefixroute ens224
       valid_lft forever preferred_lft forever

node2

    inet 192.168.163.42/24 brd 192.168.163.255 scope global noprefixroute ens224
       valid_lft forever preferred_lft forever

node3

    inet 192.168.163.43/24 brd 192.168.163.255 scope global noprefixroute ens224
       valid_lft forever preferred_lft forever

面倒なので 該当ネットワークは外部から完全に切り離されていて DB の通信だけが流れるという前提のもと、 firewall は該当セグメントを全通し。

$ sudo firewall-cmd --list-all --zone=drop
drop (active)
  target: DROP
  icmp-block-inversion: no
  interfaces: ens224
  sources:
  services:
  ports:
  protocols:
  forward: no
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
	rule family="ipv4" source address="192.168.163.0/24" accep

configure

公式ドキュメントとか読みつつ設定（AccessControl はとりあえず考えない）

mongod.conf

$ cat /etc/mongod.conf
# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Where and how to store data.
storage:
  dbPath: /var/lib/mongo
  journal:
    enabled: true
#  engine:
#  wiredTiger:

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0 # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.

#security:

#operationProfiling:

replication:
  replSetName: "staging_rs"

#sharding:

## Enterprise-Only Options

#auditLog:

#snmp:

と言っても変更したのは二箇所だけ

$ diff /etc/mongod.conf /tmp/mongod.conf
29c29,30
<   bindIp: 0.0.0.0 # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.
---
>   bindIp: 127.0.0.1  # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.
>
35,36c36
< replication:
<   replSetName: "staging_rs"
---
> #replication:

bindIp については、localhost,hostname,... のように記載しろと公式ドキュメントは言っているのだが、それで設定しても何故か exit 48 してしまう し、わざわざ原因を調べるのも面倒だしそもそもこのセグメントは物理的に（以下略）な ので、0.0.0.0 でフルオープンにしている。
何か知ってたら教えて下さい。

疎通確認

一応 hosts で hostname アクセスできるように

$ cat /etc/hosts
127.0.0.1   node1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         node1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.163.41 node1
192.168.163.42 node2
192.168.163.43 node3

接続してみる

$ mongosh --host node2
Current Mongosh Log ID:	63760d87b8ca90e34cfd2773
Connecting to:		mongodb://node2:27017/?directConnection=true&appName=mongosh+1.6.0
Using MongoDB:		6.0.3
Using Mongosh:		1.6.0

For mongosh info see: https://docs.mongodb.com/mongodb-shell/


To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.

------
   The server generated these startup warnings when booting
   2022-11-17T05:31:28.076-05:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
   2022-11-17T05:31:28.076-05:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
   2022-11-17T05:31:28.076-05:00: vm.max_map_count is too low
------

------
   Enable MongoDB's free cloud-based monitoring service, which will then receive and display
   metrics about your deployment (disk utilization, CPU, operation statistics, etc).

   The monitoring data will be available on a MongoDB website with a unique URL accessible to you
   and anyone you share the URL with. MongoDB may use this information to make product
   improvements and to suggest MongoDB products and deployment options to you.

   To enable free monitoring, run the following command: db.enableFreeMonitoring()
   To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
------

test> exit

$ mongosh --host node3
Current Mongosh Log ID:	63760e92516565ee42298ecc
Connecting to:		mongodb://node3:27017/?directConnection=true&appName=mongosh+1.6.0
Using MongoDB:		6.0.3
Using Mongosh:		1.6.0

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

------
   The server generated these startup warnings when booting
   2022-11-17T05:31:28.106-05:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
   2022-11-17T05:31:28.107-05:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
   2022-11-17T05:31:28.107-05:00: vm.max_map_count is too low
------

------
   Enable MongoDB's free cloud-based monitoring service, which will then receive and display
   metrics about your deployment (disk utilization, CPU, operation statistics, etc).

   The monitoring data will be available on a MongoDB website with a unique URL accessible to you
   and anyone you share the URL with. MongoDB may use this information to make product
   improvements and to suggest MongoDB products and deployment options to you.

   To enable free monitoring, run the following command: db.enableFreeMonitoring()
   To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
------

test> exit

問題なく接続できてそう。

同期してみる

自分自身に接続

$ mongosh
Current Mongosh Log ID:	63760f4babfcbf3520a2f501
Connecting to:		mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.6.0
Using MongoDB:		6.0.3
Using Mongosh:		1.6.0

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

------
   The server generated these startup warnings when booting
   2022-11-17T05:31:28.082-05:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
   2022-11-17T05:31:28.082-05:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
   2022-11-17T05:31:28.082-05:00: vm.max_map_count is too low
------

------
   Enable MongoDB's free cloud-based monitoring service, which will then receive and display
   metrics about your deployment (disk utilization, CPU, operation statistics, etc).

   The monitoring data will be available on a MongoDB website with a unique URL accessible to you
   and anyone you share the URL with. MongoDB may use this information to make product
   improvements and to suggest MongoDB products and deployment options to you.

   To enable free monitoring, run the following command: db.enableFreeMonitoring()
   To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
------

test>

rs.initiate() という名のミス

test> rs.initiate()
{
  info2: 'no configuration specified. Using a default configuration for the set',
  me: 'STMNG001:27017',
  ok: 1
}
staging_rs [direct: other] test> rs.conf()
{
  _id: 'staging_rs',
  version: 1,
  term: 1,
  members: [
    {
      _id: 0,
      host: 'STMNG001:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      secondaryDelaySecs: Long("0"),
      votes: 1
    }
  ],
  protocolVersion: Long("1"),
  writeConcernMajorityJournalDefault: true,
  settings: {
    chainingAllowed: true,
    heartbeatIntervalMillis: 2000,
    heartbeatTimeoutSecs: 10,
    electionTimeoutMillis: 10000,
    catchUpTimeoutMillis: -1,
    catchUpTakeoverDelayMillis: 30000,
    getLastErrorModes: {},
    getLastErrorDefaults: { w: 1, wtimeout: 0 },
    replicaSetId: ObjectId("63760f6dcc9599202db21af9")
  }
}

※これはやらかしです

rs.initiate() に parameter を入れて一気に初期化してあげればよかったのに、一回初期化しちゃうともう initiate は動いてくれませんね。

staging_rs [direct: primary] test> rs.initiate( {
...    _id : "staging_rs",
...    members: [
...       { _id: 0, host: "node1:27017" },
...       { _id: 1, host: "node2:27017" },
...       { _id: 2, host: "node3:27017" }
...    ]
... })
MongoServerError: already initialized

やっちまったもんは仕方ないので rs.add していく（Replica set の情報消すのは結構面倒そうだったため）

staging_rs [direct: primary] test> rs.add(
...     { _id: 1, host: "node2:27017" }
... )
{
    ok: 1,
    '$clusterTime': {
    clusterTime: Timestamp({ t: 1668682278, i: 1 }),
    signature: {
        hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
        keyId: Long("0")
    }
    },
    operationTime: Timestamp({ t: 1668682278, i: 1 })
}

staging_rs [direct: primary] test> rs.add(
...     { _id: 2, host: "node3:27017" }
... )
{
    ok: 1,
    '$clusterTime': {
    clusterTime: Timestamp({ t: 1668682321, i: 1 }),
    signature: {
        hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
        keyId: Long("0")
    }
    },
    operationTime: Timestamp({ t: 1668682321, i: 1 })
}

rs.status() で状態確認

staging_rs [direct: primary] test> rs.status()
{
  set: 'staging_rs',
  date: ISODate("2022-11-17T10:53:07.205Z"),
  myState: 1,
  term: Long("2"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1668682379, i: 1 }), t: Long("2") },
    lastCommittedWallTime: ISODate("2022-11-17T10:52:59.238Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1668682379, i: 1 }), t: Long("2") },
    appliedOpTime: { ts: Timestamp({ t: 1668682379, i: 1 }), t: Long("2") },
    durableOpTime: { ts: Timestamp({ t: 1668682379, i: 1 }), t: Long("2") },
    lastAppliedWallTime: ISODate("2022-11-17T10:52:59.238Z"),
    lastDurableWallTime: ISODate("2022-11-17T10:52:59.238Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1668682339, i: 1 }),
  electionCandidateMetrics: {
    lastElectionReason: 'electionTimeout',
    lastElectionDate: ISODate("2022-11-17T10:43:19.198Z"),
    electionTerm: Long("2"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1668681761, i: 1 }), t: Long("1") },
    numVotesNeeded: 1,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    newTermStartDate: ISODate("2022-11-17T10:43:19.207Z"),
    wMajorityWriteAvailabilityDate: ISODate("2022-11-17T10:43:19.225Z")
  },
  members: [
    {
      _id: 0,
      name: 'node1:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 590,
      optime: { ts: Timestamp({ t: 1668682379, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-11-17T10:52:59.000Z"),
      lastAppliedWallTime: ISODate("2022-11-17T10:52:59.238Z"),
      lastDurableWallTime: ISODate("2022-11-17T10:52:59.238Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1668681799, i: 1 }),
      electionDate: ISODate("2022-11-17T10:43:19.000Z"),
      configVersion: 5,
      configTerm: 2,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 1,
      name: 'node2:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 108,
      optime: { ts: Timestamp({ t: 1668682379, i: 1 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1668682379, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-11-17T10:52:59.000Z"),
      optimeDurableDate: ISODate("2022-11-17T10:52:59.000Z"),
      lastAppliedWallTime: ISODate("2022-11-17T10:52:59.238Z"),
      lastDurableWallTime: ISODate("2022-11-17T10:52:59.238Z"),
      lastHeartbeat: ISODate("2022-11-17T10:53:05.733Z"),
      lastHeartbeatRecv: ISODate("2022-11-17T10:53:05.734Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'node1:27017',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 5,
      configTerm: 2
    },
    {
      _id: 2,
      name: 'node3:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 65,
      optime: { ts: Timestamp({ t: 1668682379, i: 1 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1668682379, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-11-17T10:52:59.000Z"),
      optimeDurableDate: ISODate("2022-11-17T10:52:59.000Z"),
      lastAppliedWallTime: ISODate("2022-11-17T10:52:59.238Z"),
      lastDurableWallTime: ISODate("2022-11-17T10:52:59.238Z"),
      lastHeartbeat: ISODate("2022-11-17T10:53:05.735Z"),
      lastHeartbeatRecv: ISODate("2022-11-17T10:53:06.255Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'node2:27017',
      syncSourceId: 1,
      infoMessage: '',
      configVersion: 5,
      configTerm: 2
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1668682379, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1668682379, i: 1 })
}

PRIMARY 1, SECONDARY 2 の一般的な MongoDB ReplicaSet ができてそう。

動作確認

db 確認

staging_rs [direct: primary] test> show dbs
admin    80.00 KiB
config  176.00 KiB
local   436.00 KiB

新規作成

staging_rs [direct: primary] test> use hogedb
switched to db hogedb

INSERT

staging_rs [direct: primary] hogedb> db.hoge.insert({fuga: "FUGA"})
DeprecationWarning: Collection.insert() is deprecated. Use insertOne, insertMany, or bulkWrite.
{
  acknowledged: true,
  insertedIds: { '0': ObjectId("6376142de6809436052f37ce") }
}

怒られたが気にしないで 確認

staging_rs [direct: primary] hogedb> show dbs
admin    80.00 KiB
config  208.00 KiB
hogedb   40.00 KiB
local   444.00 KiB
staging_rs [direct: primary] hogedb> db
hogedb

staging_rs [direct: primary] hogedb> db.hoge.find()
[ { _id: ObjectId("6376142de6809436052f37ce"), fuga: 'FUGA' } ]

ちゃんと入ってそうなので SECONDARY の DB 確認

staging_rs [direct: secondary] test> show dbs
admin    80.00 KiB
config  248.00 KiB
hogedb   40.00 KiB
local   436.00 KiB

きちんと同期されてそう。

長くなってしまったので、ちゃんとクライアントつないで secondary から read されるかとかは別途時間ができたら。

PR

最近 AS 番号を取得した株式会社ねこじゃらし（AS 146981）で働いてみたい人はこちらへどうぞ。
https://recruit.nekojarashi.com/