LXD(システムコンテナ)上でPacemakerをセットアップしてみました。
環境 - まずは単ノードで
KVM上のCentOS 8
CentOS-Stream-8-x86_64-2021102
LXDのインストール
以下のページを参考にしました。
手順
以下の操作は途中まではrootユーザで実行しています
KVMホスト上にCentOSのインストールを実行
インストール後にselinuxをdisabledに設定。firewallも停止。
EPELの有効化とアップデート
# yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
(snip)
# yum -y update
(snip)
snapdのインストール前の設定と再起動
[root@cent8-01 ~]# yum search snapd
メタデータの期限切れの最終確認: 0:10:26 時間前の 2021年11月29日 16時29分31秒 に実施しました。
=============================================================================== 名前 完全一致: snapd ================================================================================
snapd.x86_64 : A transactional software package manager
============================================================================== 名前 & 概要 一致: snapd ==============================================================================
snapd-devel.noarch : Development files for snapd
snapd-glib.x86_64 : Library providing a GLib interface to snapd
snapd-glib-devel.x86_64 : Development files for snapd-glib
snapd-glib-tests.x86_64 : Installed tests for snapd-glib
snapd-qt.x86_64 : Library providing a Qt5 interface to snapd
snapd-qt-devel.x86_64 : Development files for snapd-qt
snapd-qt-qml.x86_64 : Library providing a Qt5 QML interface to snapd
snapd-qt-tests.x86_64 : Installed tests for snapd-qt
snapd-selinux.noarch : SELinux module for snapd
[root@cent8-01 ~]# grubby --args="namespace.unpriv_enable=1" --update-kernel="$(grubby --default-kernel)"
[root@cent8-01 ~]# sh -c 'echo "user.max_user_namespaces=2147483647" >> /etc/sysctl.d/99-userns.conf'
[root@cent8-01 ~]# shutdown -r now
設定の確認とsnapdのインストールと再起動
[root@cent8-01 ~]# cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-348.2.1.el8_5.x86_64 root=/dev/mapper/cs-root ro resume=/dev/mapper/cs-swap rd.lvm.lv=cs/root rd.lvm.lv=cs/swap rhgb quiet namespace.unpriv_enable=1
[root@cent8-01 ~]# yum -y install snapd
[root@cent8-01 ~]# ln -s /var/lib/snapd/snap /snap
[root@cent8-01 ~]#
[root@cent8-01 ~]# systemctl enable --now snapd.socket
Created symlink /etc/systemd/system/sockets.target.wants/snapd.socket → /usr/lib/systemd/system/snapd.socket.
[root@cent8-01 ~]# systemctl status snapd.socket
● snapd.socket - Socket activation for snappy daemon
Loaded: loaded (/usr/lib/systemd/system/snapd.socket; enabled; vendor preset: disabled)
Active: active (listening) since Mon 2021-11-29 16:46:49 JST; 6s ago
Listen: /run/snapd.socket (Stream)
/run/snapd-snap.socket (Stream)
Tasks: 0 (limit: 49678)
Memory: 4.0K
CGroup: /system.slice/snapd.socket
11月 29 16:46:49 cent8-01 systemd[1]: Starting Socket activation for snappy daemon.
11月 29 16:46:49 cent8-01 systemd[1]: Listening on Socket activation for snappy daemon.
[root@cent8-01 ~]# systemctl status snapd.service
● snapd.service - Snap Daemon
Loaded: loaded (/usr/lib/systemd/system/snapd.service; disabled; vendor preset: disabled)
Active: inactive (dead)
[root@cent8-01 ~]# systemctl enable snapd.service
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.service → /usr/lib/systemd/system/snapd.service.
[root@cent8-01 ~]# systemctl start snapd.service
[root@cent8-01 ~]# shutdown -r now
lxdのインストール
[root@cent8-01 ~]# snap install lxd
2021-11-29T16:50:24+09:00 INFO Waiting for automatic snapd restart...
lxd 4.20 from Canonical✓ installed
[root@cent8-01 ~]# useradd yamauchi
[root@cent8-01 ~]# usermod -a -G lxd yamauchi
[root@cent8-01 ~]# su - yamauchi
[yamauchi@cent8-01 ~]$ newgrp lxd
[yamauchi@cent8-01 ~]$ id
uid=1001(yamauchi) gid=971(lxd) groups=971(lxd),1001(yamauchi)
[yamauchi@cent8-01 ~]$ lxc list
If this is your first time running LXD on this machine, you should also run: lxd init
To start your first container, try: lxc launch ubuntu:20.04
Or for a virtual machine: lxc launch ubuntu:20.04 --vm
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+
lxdのconfigurationの実行
途中storage backendだけ、OSインストール時のファイルシステムのlvmで指定しています。
[yamauchi@cent8-01 ~]$ lxd init
Would you like to use LXD clustering? (yes/no) [default=no]:
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]:
Name of the storage backend to use (ceph, btrfs, dir, lvm) [default=btrfs]: lvm
Create a new LVM pool? (yes/no) [default=yes]:
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]:
Size in GB of the new loop device (1GB minimum) [default=5GB]:
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]:
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
Would you like the LXD server to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:
イメージの確認
多くの公式イメージが出力される。
[yamauchi@cent8-01 ~] lxc image list images:
とりあえず、centos8のイメージを取得
[yamauchi@cent8-01 ~]$ lxc launch images:centos/8/amd64 cent8-01
Creating cent8-01
Starting cent8-01
[yamauchi@cent8-01 ~]$ lxc list
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| cent8-01 | RUNNING | 10.144.107.190 (eth0) | fd42:26e0:cb93:870a:216:3eff:fe63:367d (eth0) | CONTAINER | 0 |
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
起動しているコンテナへ接続
[yamauchi@cent8-01 ~]$ lxc exec cent8-01 bash
High Availabilityを有効にしてコンテナにpcs,pacemaker,fence-agentsをインストール
[root@cent8-01 ~]# dnf --enablerepo=ha -y install pacemaker pcs fence-agents-all
CentOS Linux 8 - AppStream 3.7 MB/s | 8.1 MB 00:02
CentOS Linux 8 - BaseOS 2.9 MB/s | 3.5 MB 00:01
CentOS Linux 8 - Extras 17 kB/s | 10 kB 00:00
CentOS Linux 8 - HighAvailability 528 kB/s | 521 kB 00:00
Dependencies resolved.
=====================================================================================================================================================================================
Package Architecture Version Repository Size
=====================================================================================================================================================================================
Installing:
fence-agents-all x86_64 4.2.1-75.el8 appstream 22 k
pacemaker x86_64 2.1.0-8.el8 ha 456 k
pcs x86_64 0.10.8-1.el8 ha 13 M
(snip)
一旦、コンテナから抜けて、コンテナを停止
[root@cent8-01 ~]# exit
exit
[yamauchi@cent8-01 ~]$ lxc stop cent8-01
[yamauchi@cent8-01 ~]$ lxc list
+----------+---------+------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+----------+---------+------+------+-----------+-----------+
| cent8-01 | STOPPED | | | CONTAINER | 0 |
+----------+---------+------+------+-----------+-----------+
cont8-01をpublishしてから、cent8-02のイメージを作成
[yamauchi@cent8-01 ~]$ lxc publish cent8-01 --alias pacemaker
Instance published with fingerprint: f6716cdfd9643b59c6f96786b8d2306726dae386259cd9cfd3e77dfce78e8900
[yamauchi@cent8-01 ~]$ lxc list
+----------+---------+------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+----------+---------+------+------+-----------+-----------+
| cent8-01 | STOPPED | | | CONTAINER | 0 |
+----------+---------+------+------+-----------+-----------+
[yamauchi@cent8-01 ~]$ lxc image list
+-----------+--------------+--------+----------------------------------+--------------+-----------+----------+------------------------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCHITECTURE | TYPE | SIZE | UPLOAD DATE |
+-----------+--------------+--------+----------------------------------+--------------+-----------+----------+------------------------------+
| pacemaker | f6716cdfd964 | no | Centos 8 x86_64 (20211127_07:08) | x86_64 | CONTAINER | 308.77MB | Nov 29, 2021 at 8:04am (UTC) |
+-----------+--------------+--------+----------------------------------+--------------+-----------+----------+------------------------------+
| | 6b5db6ae52c3 | no | Centos 8 amd64 (20211127_07:08) | x86_64 | CONTAINER | 127.49MB | Nov 29, 2021 at 7:56am (UTC) |
+-----------+--------------+--------+----------------------------------+--------------+-----------+----------+------------------------------+
[yamauchi@cent8-01 ~]$ lxc launch pacemaker cent8-02
Creating cent8-02
Starting cent8-02
[yamauchi@cent8-01 ~]$ lxc list
+----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| cent8-01 | STOPPED | | | CONTAINER | 0 |
+----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| cent8-02 | RUNNING | 10.144.107.86 (eth0) | fd42:26e0:cb93:870a:216:3eff:fe88:8ae1 (eth0) | CONTAINER | 0 |
+----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
cont8-01も再起動
[yamauchi@cent8-01 ~]$ lxc start cent8-01
[yamauchi@cent8-01 ~]$ lxc list
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| cent8-01 | RUNNING | 10.144.107.190 (eth0) | fd42:26e0:cb93:870a:216:3eff:fe63:367d (eth0) | CONTAINER | 0 |
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| cent8-02 | RUNNING | 10.144.107.86 (eth0) | fd42:26e0:cb93:870a:216:3eff:fe88:8ae1 (eth0) | CONTAINER | 0 |
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
クラスタの構築
2つのコンソールからそれぞれのコンテナへ接続
[yamauchi@cent8-01 ~]$ lxc exec cent8-01 bash
[root@cent8-01 ~]#
[yamauchi@cent8-01 ~]$ lxc exec cent8-02 bash
[root@cent8-02 ~]#
それぞれのコンテナでhaclusterユーザのパスワードを設定し、pcsdサービスを起動
[root@cent8-01 ~]# passwd hacluster
Changing password for user hacluster.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[root@cent8-01 ~]# systemctl enable pcsd --now
Created symlink /etc/systemd/system/multi-user.target.wants/pcsd.service → /usr/lib/systemd/system/pcsd.service.
[root@cent8-01 ~]# systemctl status pcsd
● pcsd.service - PCS GUI and remote configuration interface
Loaded: loaded (/usr/lib/systemd/system/pcsd.service; enabled; vendor preset: disabled)
Drop-In: /run/systemd/system/pcsd.service.d
└─zzz-lxc-service.conf
Active: active (running) since Mon 2021-11-29 08:10:11 UTC; 5s ago
Docs: man:pcsd(8)
man:pcs(8)
Main PID: 1367 (pcsd)
Tasks: 1 (limit: 49678)
Memory: 38.8M
CGroup: /system.slice/pcsd.service
└─1367 /usr/libexec/platform-python -Es /usr/sbin/pcsd
Nov 29 08:10:10 cent8-01 systemd[1]: Starting PCS GUI and remote configuration interface...
Nov 29 08:10:11 cent8-01 systemd[1]: Started PCS GUI and remote configuration interface.
片方のコンテナで、pcsコマンドで各コンテナを認証
[root@cent8-01 ~]# pcs host auth cent8-01 cent8-02
Username: hacluster
Password:
cent8-01: Authorized
cent8-02: Authorized
一旦、pcsコマンドでクラスタをセットアップ
インターフェースは、最初は既存のインターフェースを指定
[root@cent8-01 ~]# pcs cluster setup cluster_name cent8-01 addr=10.144.107.190 cent8-02 addr=10.144.107.86
Destroying cluster on hosts: 'cent8-01', 'cent8-02'...
cent8-01: Successfully destroyed cluster
cent8-02: Successfully destroyed cluster
Requesting remove 'pcsd settings' from 'cent8-01', 'cent8-02'
cent8-01: successful removal of the file 'pcsd settings'
cent8-02: successful removal of the file 'pcsd settings'
Sending 'corosync authkey', 'pacemaker authkey' to 'cent8-01', 'cent8-02'
cent8-01: successful distribution of the file 'corosync authkey'
cent8-01: successful distribution of the file 'pacemaker authkey'
cent8-02: successful distribution of the file 'corosync authkey'
cent8-02: successful distribution of the file 'pacemaker authkey'
Sending 'corosync.conf' to 'cent8-01', 'cent8-02'
cent8-01: successful distribution of the file 'corosync.conf'
cent8-02: successful distribution of the file 'corosync.conf'
Cluster has been successfully set up.
pcsでクラスタを起動しようとするが失敗
[root@cent8-01 ~]# pcs cluster start --all
cent8-01: Error connecting to cent8-01 - (HTTP error: 400)
cent8-02: Error connecting to cent8-02 - (HTTP error: 400)
Error: unable to start all nodes
cent8-01: Error connecting to cent8-01 - (HTTP error: 400)
cent8-02: Error connecting to cent8-02 - (HTTP error: 400)
どうやら、問題は、knet指定がうまく動作していないようなので、両コンテナのcorosync.confのtotemブロックを以下のように変更
transportをudpに、crypto_cipher/crypto_hashをnoneに、interfaceブロックを追加
totem {
version: 2
cluster_name: cluster_name
transport: udp
crypto_cipher: none
crypto_hash: none
interface {
ringnumber: 0
}
}
(snip)
再度、クラスタを起動
[root@cent8-01 ~]# pcs cluster start --all
cent8-01: Starting Cluster...
cent8-02: Starting Cluster...
無事に起動
[root@cent8-01 ~]# crm_mon -rfA1
Cluster Summary:
* Stack: corosync
* Current DC: cent8-02 (version 2.1.0-8.el8-7c3f660707) - partition with quorum
* Last updated: Mon Nov 29 08:18:54 2021
* Last change: Mon Nov 29 08:18:49 2021 by hacluster via crmd on cent8-02
* 2 nodes configured
* 0 resource instances configured
Node List:
* Online: [ cent8-01 cent8-02 ]
Full List of Resources:
* No resources
Migration Summary:
関連リンク
https://linuxcontainers.org/ja/lxd/introduction/
https://computingforgeeks.com/run-linux-containers-with-lxc-lxd-on-rocky-almalinux/
以上、