いま再びのシステムまぎちゃんへの道～SLURMのインストール～

SLURMとは？

SLURM は、いわゆるリソースマネジメントを行うツールで
バッチジョブを流すと、適切なCPUへジョブを割り当ててくれるものです。
その昔はPBS あるいはPBSProが使われてきましたが、最近ではSLURMも捨てたもんじゃありません。

おりょ？SLURMがパッケージになっている。

ということで、このパッケージを入れていきます。
これもいたって簡単、

# dnf install slurm*

ということで、SLURMも入りました。

さてここからがちょっとキツイ

SLURM を入れると、MUNGE という認証アプリを通すことになります。
まずは、MUGNEを立ち上げるところから。

[root@localhost system]# systemctl start munge
Job for munge.service failed because the control process exited with error code.
See "systemctl status munge.service" and "journalctl -xe" for details.

おや？立ち上げりませんね、ステータスを見てみましょう。

[root@localhost system]# systemctl status munge
● munge.service - MUNGE authentication service
   Loaded: loaded (/usr/lib/systemd/system/munge.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2019-12-02 09:46:54 EST; 7s ago
     Docs: man:munged(8)
  Process: 27359 ExecStart=/usr/sbin/munged (code=exited, status=1/FAILURE)
      CPU: 28ms

Dec 02 09:46:54 vpn.sc-magi.com systemd[1]: Starting MUNGE authentication service...
Dec 02 09:46:54 vpn.sc-magi.com munged[27359]: munged: Error: Failed to check keyfile "/etc/munge/munge.key": No such file or directory
Dec 02 09:46:54 vpn.sc-magi.com systemd[1]: munge.service: Control process exited, code=exited, status=1/FAILURE
Dec 02 09:46:54 vpn.sc-magi.com systemd[1]: munge.service: Failed with result 'exit-code'.
Dec 02 09:46:54 vpn.sc-magi.com systemd[1]: Failed to start MUNGE authentication service.

/etc/munge/munge.keyがないって言われてしまいました。

そこで、

# man munge

をすると、一番下に、WebページのURLがあったのでそれをクリック。
https://dun.github.io/munge/

で、インストール後の設定を見ていると

# dd if=/dev/random bs=1 count=1024 >/etc/munge/munge.key

をせよとのこと、というわけで上記を実行します。

[root@localhost munge]# dd if=/dev/random bs=1 count=1024 >/etc/munge/munge.key
1024+0 records in
1024+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.029465 s, 34.8 kB/s

で、このキーの所有者を変えます。

[root@localhost munge]# chown munge:munge /etc/munge/munge.key

さぁ、これで動くだろうと実行してみると…。

[root@localhost munge]# systemctl start munge
Job for munge.service failed because the control process exited with error code.
See "systemctl status munge.service" and "journalctl -xe" for details.
[root@localhost munge]# systemctl status munge
● munge.service - MUNGE authentication service
   Loaded: loaded (/usr/lib/systemd/system/munge.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2019-12-02 09:51:21 EST; 5s ago
     Docs: man:munged(8)
  Process: 27526 ExecStart=/usr/sbin/munged (code=exited, status=1/FAILURE)
      CPU: 31ms

Dec 02 09:51:19 vpn.sc-magi.com systemd[1]: Starting MUNGE authentication service...
Dec 02 09:51:21 vpn.sc-magi.com munged[27526]: munged: Error: Keyfile is insecure: "/etc/munge/munge.key" should not be readable or writable by group or world
Dec 02 09:51:21 vpn.sc-magi.com systemd[1]: munge.service: Control process exited, code=exited, status=1/FAILURE
Dec 02 09:51:21 vpn.sc-magi.com systemd[1]: munge.service: Failed with result 'exit-code'.
Dec 02 09:51:21 vpn.sc-magi.com systemd[1]: Failed to start MUNGE authentication service.

あちゃー、また怒られました。ので、変更します。

[root@localhost munge]# chmod 600 /etc/munge/munge.key

さぁ、これでもう動くだろうと思って実行、

[root@localhost munge]# systemctl status munge
● munge.service - MUNGE authentication service
   Loaded: loaded (/usr/lib/systemd/system/munge.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2019-12-02 09:51:21 EST; 51s ago
     Docs: man:munged(8)
  Process: 27526 ExecStart=/usr/sbin/munged (code=exited, status=1/FAILURE)
      CPU: 31ms

Dec 02 09:51:19 vpn.sc-magi.com systemd[1]: Starting MUNGE authentication service...
Dec 02 09:51:21 vpn.sc-magi.com munged[27526]: munged: Error: Keyfile is insecure: "/etc/munge/munge.key" should not be readable or writable by group or world
Dec 02 09:51:21 vpn.sc-magi.com systemd[1]: munge.service: Control process exited, code=exited, status=1/FAILURE
Dec 02 09:51:21 vpn.sc-magi.com systemd[1]: munge.service: Failed with result 'exit-code'.
Dec 02 09:51:21 vpn.sc-magi.com systemd[1]: Failed to start MUNGE authentication service.
[root@localhost munge]# systemctl start munge
[root@localhost munge]# systemctl status munge
● munge.service - MUNGE authentication service
   Loaded: loaded (/usr/lib/systemd/system/munge.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-12-02 09:52:16 EST; 3s ago
     Docs: man:munged(8)
  Process: 27563 ExecStart=/usr/sbin/munged (code=exited, status=0/SUCCESS)
 Main PID: 27565 (munged)
    Tasks: 4 (limit: 999)
   Memory: 1.1M
      CPU: 40ms
   CGroup: /system.slice/munge.service
           mq27565 /usr/sbin/munged

Dec 02 09:52:16 vpn.sc-magi.com systemd[1]: Starting MUNGE authentication service...
Dec 02 09:52:16 vpn.sc-magi.com systemd[1]: Started MUNGE authentication service.

これで、見事に動きました。
次にいよいよSLURMを立ち上げてみます。

[root@localhost munge]# systemctl start slurmctld
[root@localhost munge]# systemctl start slurmdbd
[root@localhost munge]# systemctl start slurmd
[root@localhost munge]# systemctl sttus slurmd
Unknown operation sttus.
[root@localhost munge]# systemctl status slurmd
● slurmd.service - Slurm node daemon
   Loaded: loaded (/usr/lib/systemd/system/slurmd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-12-02 09:44:48 EST; 7min ago
  Process: 27305 ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 27307 (slurmd)
    Tasks: 1
   Memory: 2.6M
      CPU: 668ms
   CGroup: /system.slice/slurmd.service
           mq27307 /usr/sbin/slurmd

Dec 02 09:52:08 vpn.sc-magi.com slurmd[27307]: error: authentication: Invalid authentication credential
Dec 02 09:52:08 vpn.sc-magi.com slurmd[27307]: error: Unable to register: Protocol authentication error
Dec 02 09:52:11 vpn.sc-magi.com slurmd[27307]: error: If munged is up, restart with --num-threads=10
Dec 02 09:52:11 vpn.sc-magi.com slurmd[27307]: error: Munge encode failed: Failed to access "/var/run/munge/munge.socket.2": No such file or directory
Dec 02 09:52:11 vpn.sc-magi.com slurmd[27307]: error: authentication: Invalid authentication credential
Dec 02 09:52:11 vpn.sc-magi.com slurmd[27307]: error: Unable to register: Protocol authentication error
Dec 02 09:52:14 vpn.sc-magi.com slurmd[27307]: error: If munged is up, restart with --num-threads=10
Dec 02 09:52:14 vpn.sc-magi.com slurmd[27307]: error: Munge encode failed: Failed to access "/var/run/munge/munge.socket.2": No such file or directory
Dec 02 09:52:14 vpn.sc-magi.com slurmd[27307]: error: authentication: Invalid authentication credential
Dec 02 09:52:14 vpn.sc-magi.com slurmd[27307]: error: Unable to register: Protocol authentication error

立ち上がっていはいますが、うまく動いていない模様。そこでこのプロセスをもう一度再起動

[root@localhost munge]# systemctl status slurmd
● slurmd.service - Slurm node daemon
   Loaded: loaded (/usr/lib/systemd/system/slurmd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-12-02 10:07:18 EST; 2s ago
  Process: 27641 ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 27643 (slurmd)
    Tasks: 1
   Memory: 1.4M
      CPU: 98ms
   CGroup: /system.slice/slurmd.service
           mq27643 /usr/sbin/slurmd

Dec 02 10:07:17 vpn.sc-magi.com systemd[1]: Starting Slurm node daemon...
Dec 02 10:07:17 vpn.sc-magi.com slurmd[27641]: error: Node configuration differs from hardware: Procs=1:4(hw) Boards=1:1(hw) SocketsPerBoard=1:1(hw) CoresPerSocket=1:4(hw) ThreadsPerCore=1:1(hw)
Dec 02 10:07:17 vpn.sc-magi.com slurmd[27641]: Message aggregation disabled
Dec 02 10:07:17 vpn.sc-magi.com slurmd[27643]: slurmd version 19.05.4 started
Dec 02 10:07:17 vpn.sc-magi.com systemd[1]: slurmd.service: Can't open PID file /run/slurm/slurmd.pid (yet?) after start: No such file or directory
Dec 02 10:07:17 vpn.sc-magi.com slurmd[27643]: slurmd started on Mon, 02 Dec 2019 10:07:17 -0500
Dec 02 10:07:17 vpn.sc-magi.com slurmd[27643]: CPUs=1 Boards=1 Sockets=1 Cores=1 Threads=1 Memory=951 TmpDisk=475 Uptime=9636 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)
Dec 02 10:07:18 vpn.sc-magi.com systemd[1]: Started Slurm node daemon.

あ、あとは設定ファイルの問題みたいですね。それは追々やっていきます。