はじめに
PowerVC が正常起動しないエラーを対応したログです。
[参考 Technote]
原因
PowerVC は、機能するためにメッセージ ブローカー サービスである RabbitMQ に依存しています。RabbitMQ でディスクがいっぱいになるイベントが発生すると、ディスクに書き込むことができず、実行中のトランザクションは失敗します。RabbitMQ は、それらのインフライトトランザクションがクリアされるまで動作しません。これにより、PowerVC の開始と正常な動作が妨げられる
とのことです。
壊れても影響が少ない環境であったため、とりあえず試してみることにしました。
当ログは一例としての参考です。問題解決には製品サポートへのお問い合わせが確実です。
環境
PowerVC 2.1.1 (RHEL 8.6 , ppc64le, on Power)
- シングル・ノード構成 (legacy install)
発生した事象
PowerVC の調子が良くなかった (APIの実行で認証エラーとなる) ので再起動を実施したところログインできなくなりました。
サービスを確認すると、一部サービスが起動していません。
個別にサービスを立ち上げようとしてもできませんでした。
# powervc-services status
● clerk-api.service - clerk API Server
Active: active (running) since Wed 2023-07-26 05:39:54 EDT; 12h ago
● ttv-validation-api.service - TTV API Server
Active: active (running) since Wed 2023-07-26 05:39:54 EDT; 12h ago
● panko-api.service - OpenStack Panko API Server
Active: active (running) since Wed 2023-07-26 05:39:54 EDT; 12h ago
● openstack-gnocchi-metricd.service - OpenStack Gnocchi metricd Server
Active: active (running) since Wed 2023-07-26 05:39:54 EDT; 12h ago
● openstack-swift-account.service - OpenStack Object Storage (swift) - Account Server
Active: active (running) since Wed 2023-07-26 05:39:53 EDT; 12h ago
● openstack-swift-object.service - OpenStack Object Storage (swift) - Object Server
Active: active (running) since Wed 2023-07-26 05:39:53 EDT; 12h ago
● openstack-swift-proxy.service - OpenStack Object Storage (swift) - Proxy Server
Active: active (running) since Wed 2023-07-26 05:39:53 EDT; 12h ago
● openstack-swift-container.service - OpenStack Object Storage (swift) - Container Server
Active: active (running) since Wed 2023-07-26 05:39:53 EDT; 12h ago
● openstack-swift-object-replicator.service - OpenStack Object Storage (swift) - Object Replicator
Active: active (running) since Wed 2023-07-26 05:39:53 EDT; 12h ago
● memcached.service - memcached daemon
Active: active (running) since Wed 2023-07-26 05:39:26 EDT; 12h ago
● placement-api.service - Placement API Server
Active: active (running) since Wed 2023-07-26 05:39:53 EDT; 12h ago
● mongod.service - MongoDB Database Server
Active: active (running) since Wed 2023-07-26 05:39:38 EDT; 12h ago
● powervc-ui-server.service - Powervc ui server
Active: active (running) since Wed 2023-07-26 05:39:53 EDT; 12h ago
● powervc-bumblebee.service - PowerVC Bumblebee
Active: active (running) since Wed 2023-07-26 05:39:36 EDT; 12h ago
● powervc-cinder-ibm-health.service - PowerVC cinder monitor service
Active: inactive (dead)
● powervc-nova-ibm-health.service - PowerVC nova monitor service
Active: inactive (dead)
● powervc-neutron-ibm-health.service - PowerVC neutron monitor service
Active: failed (Result: exit-code) since Wed 2023-07-26 05:40:01 EDT; 12h ago
● openstack-ceilometer-notification.service - OpenStack ceilometer notification agent
Active: active (running) since Wed 2023-07-26 05:40:22 EDT; 12h ago
● openstack-ceilometer-polling.service - OpenStack ceilometer polling agent
Active: active (running) since Wed 2023-07-26 05:40:22 EDT; 12h ago
● openstack-nova-api.service - OpenStack Nova API Server
Active: active (running) since Wed 2023-07-26 05:39:35 EDT; 12h ago
● openstack-nova-conductor.service - OpenStack Nova Conductor Server
Active: active (running) since Wed 2023-07-26 05:39:35 EDT; 12h ago
● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server
Active: active (running) since Wed 2023-07-26 18:23:29 EDT; 999ms ago
● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server
Active: active (running) since Wed 2023-07-26 05:39:35 EDT; 12h ago
● openstack-nova-compute-910541B_785CA21.service - OpenStack Nova Compute Server
Active: active (running) since Wed 2023-07-26 18:23:30 EDT; 31ms ago
● neutron-server.service - OpenStack Neutron Server
Active: active (running) since Wed 2023-07-26 05:39:35 EDT; 12h ago
● openstack-nova-ibm-notification.service - OpenStack Nova IBM Notification
Active: active (running) since Wed 2023-07-26 18:23:23 EDT; 7s ago
● openstack-nova-ibm-ego-ha-service.service - OpenStack Nova IBM EGO HA Service
Active: active (running) since Wed 2023-07-26 18:23:29 EDT; 1s ago
● openstack-nova-ibm-ego-resource-optimization.service - OpenStack Nova IBM EGO Resource Optimization
Active: active (running) since Wed 2023-07-26 18:23:20 EDT; 9s ago
● openstack-cinder-api.service - OpenStack Cinder API Server
Active: active (running) since Wed 2023-07-26 05:39:35 EDT; 12h ago
● openstack-cinder-scheduler.service - OpenStack Cinder Scheduler Server
Active: inactive (dead) since Wed 2023-07-26 05:39:50 EDT; 12h ago
● openstack-cinder-volume-xxx.xx.xxx.xxx.service - OpenStack Cinder Volume Server
Active: inactive (dead) since Wed 2023-07-26 05:40:06 EDT; 12h ago
● powervc-cinder-conductor.service - PowerVC Cinder Conductor Server
Active: inactive (dead) since Wed 2023-07-26 05:39:58 EDT; 12h ago
● openstack-glance-api.service - OpenStack Image Service (code-named Glance) API server
Active: active (running) since Wed 2023-07-26 05:39:35 EDT; 12h ago
● rabbitmq-server.service - RabbitMQ broker
Active: active (running) since Wed 2023-07-26 05:39:35 EDT; 12h ago
● httpd.service - The Apache HTTP Server
Active: active (running) since Wed 2023-07-26 05:39:26 EDT; 12h ago
● memcached.service - memcached daemon
Active: active (running) since Wed 2023-07-26 05:39:26 EDT; 12h ago
● mariadb.service - MariaDB 10.5 database server
Active: active (running) since Wed 2023-07-26 05:39:26 EDT; 12h ago
cinderサービスが停止している状態であるので、まずは /var/log/cinder でエラーを確認しました。
以下のエラー出力を確認
conductor.log:2023-07-26 03:33:30.774 64966 ERROR oslo_service.service raise error_for_code(reply_code, reply_text,
conductor.log:2023-07-26 03:33:30.774 64966 ERROR oslo_service.service amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
conductor.log:2023-07-26 03:33:30.774 64966 ERROR oslo_service.service
conductor.log:2023-07-26 04:30:05.127 1621 ERROR oslo_service.service [req-1axxxxa-c27a-4190-b94e-078xxxxbf - - - - -] Error starting thread.: amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
=> oslo_service.service、メッセージング・サービス周りでエラーが発生しているようです。
RabbitMQ サービスを再構築
色々とエラーを検索し、冒頭の Technote にたどり着きました。試してみます。
- Pacemaker を確認
# service corosync status
Redirecting to /bin/systemctl status corosync.service
Unit corosync.service could not be found.
# service pacemaker status
Redirecting to /bin/systemctl status pacemaker.service
Unit pacemaker.service could not be found.
=> この環境では Pacemaker を導入していないシングル構成のため "could not be found." の出力です。
手順には Pacemaker 周りの対応が記載してあり、この環境では無効ですが、一応実行しておきます。
# crm_mon -1r
bash: crm_mon: command not found...
# ps -ef |grep -i rabbit
rabbitmq 18438 1 10 05:39 ? 01:22:10 /usr/lib64/erlang/erts-13.1/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -sbwt none -sbwtdcpu none -sbwtdio none -- -root /usr/lib64/erlang -bindir /usr/lib64/erlang/erts-13.1/bin -progname erl -- -home /var/lib/rabbitmq -- -pa -noshell -noinput -s rabbit boot -boot start_sasl -syslog logger [] -syslog syslog_error_logger false -kernel prevent_overlapping_partitions false
rabbitmq 18464 18438 0 05:39 ? 00:00:00 erl_child_setup 63536
rabbitmq 18780 1 0 05:39 ? 00:00:00 /usr/lib64/erlang/erts-13.1/bin/epmd -daemon
rabbitmq 18863 18464 0 05:39 ? 00:00:13 /usr/lib64/erlang/erts-13.1/bin/inet_gethost 4
rabbitmq 18864 18863 0 05:39 ? 00:00:20 /usr/lib64/erlang/erts-13.1/bin/inet_gethost 4
rabbitmq 18950 18464 0 05:39 ? 00:00:03 /bin/sh -s rabbit_disk_monitor
root 56701 51181 0 18:34 pts/0 00:00:00 grep --color=auto -i rabbit
- rabbitmq-server の停止
# systemctl stop rabbitmq-server
# systemctl status rabbitmq-server
● rabbitmq-server.service - RabbitMQ broker
Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/rabbitmq-server.service.d
└─service.conf
Active: inactive (dead) since Wed 2023-07-26 18:34:12 EDT; 6s ago
Process: 56743 ExecStop=/usr/sbin/rabbitmqctl shutdown (code=exited, status=0/SUCCESS)
Process: 18438 ExecStart=/usr/sbin/rabbitmq-server (code=exited, status=0/SUCCESS)
Main PID: 18438 (code=exited, status=0/SUCCESS)
Status: "Standing by"
Jul 26 05:39:34 pvc211_test rabbitmq-server[18438]: Config file(s): /etc/rabbitmq/rabbitmq.config
Jul 26 05:39:35 pvc211_test rabbitmq-server[18438]: Starting broker... completed with 0 plugins.
Jul 26 05:39:35 pvc211_test systemd[1]: Started RabbitMQ broker.
Jul 26 18:34:09 pvc211_test systemd[1]: Stopping RabbitMQ broker...
Jul 26 18:34:10 pvc211_test rabbitmqctl[56743]: Shutting down RabbitMQ node rabbit@pvc211_test running at PID 18438
Jul 26 18:34:10 pvc211_test rabbitmq-server[18438]: Gracefully halting Erlang VM
Jul 26 18:34:10 pvc211_test rabbitmqctl[56743]: Waiting for PID 18438 to terminate
Jul 26 18:34:12 pvc211_test rabbitmqctl[56743]: RabbitMQ node rabbit@pvc211_test running at PID 18438 successfully shut>
Jul 26 18:34:12 pvc211_test systemd[1]: rabbitmq-server.service: Succeeded.
Jul 26 18:34:12 pvc211_test systemd[1]: Stopped RabbitMQ broker.
- mnesia ディレクトリの削除
# rm -fr /var/lib/rabbitmq/mnesia
# rm -rf /var/lib/rabbitmq/MnesiaCore*
- rabbitmq-server の起動
# systemctl start rabbitmq-server
# systemctl status rabbitmq-server
● rabbitmq-server.service - RabbitMQ broker
Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/rabbitmq-server.service.d
└─service.conf
Active: active (running) since Wed 2023-07-26 18:35:04 EDT; 10s ago
Process: 56743 ExecStop=/usr/sbin/rabbitmqctl shutdown (code=exited, status=0/SUCCESS)
Main PID: 56904 (beam.smp)
Tasks: 85 (limit: 202096)
Memory: 131.8M
CGroup: /system.slice/rabbitmq-server.service
├─56904 /usr/lib64/erlang/erts-13.1/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 51>
├─56917 erl_child_setup 63536
├─57005 /usr/lib64/erlang/erts-13.1/bin/epmd -daemon
├─57088 /usr/lib64/erlang/erts-13.1/bin/inet_gethost 4
├─57089 /usr/lib64/erlang/erts-13.1/bin/inet_gethost 4
└─57094 /bin/sh -s rabbit_disk_monitor
Jul 26 18:35:02 pvc211_test rabbitmq-server[56904]: Doc guides: https://rabbitmq.com/documentation.html
Jul 26 18:35:02 pvc211_test rabbitmq-server[56904]: Support: https://rabbitmq.com/contact.html
Jul 26 18:35:02 pvc211_test rabbitmq-server[56904]: Tutorials: https://rabbitmq.com/getstarted.html
Jul 26 18:35:02 pvc211_test rabbitmq-server[56904]: Monitoring: https://rabbitmq.com/monitoring.html
Jul 26 18:35:02 pvc211_test rabbitmq-server[56904]: Logs: /var/log/rabbitmq/rabbit@pvc211_test.log
Jul 26 18:35:02 pvc211_test rabbitmq-server[56904]: /var/log/rabbitmq/rabbit@pvc211_test_upgrade.log
Jul 26 18:35:02 pvc211_test rabbitmq-server[56904]: <stdout>
Jul 26 18:35:02 pvc211_test rabbitmq-server[56904]: Config file(s): /etc/rabbitmq/rabbitmq.config
Jul 26 18:35:04 pvc211_test rabbitmq-server[56904]: Starting broker... completed with 0 plugins.
Jul 26 18:35:04 pvc211_test systemd[1]: Started RabbitMQ broker.
- Rabbitmq のパスワードを取得
# rmq_pwd=$(cat /etc/nova/nova.conf | grep transport_url | grep aes | grep -v "#" | head -1 |awk '{print $3}'| awk '{$1=$1};1')
# rmq_passwd=$(python3 -c "from oslo_config.ibm.encrypthandler import EncryptHandler; print (EncryptHandler().decode('$rmq_pwd'))" | awk -F':' '{print $3}'|awk -F'@' '{print $1}' | awk '{$1=$1};1')
# echo $rmq_passwd
sxxxxxSaxxxxc
- powervc_mq ユーザーに適切なユーザー権限を設定
# rabbitmqctl add_user powervc_mq $rmq_passwd
Adding user "powervc_mq" ...
Done. Don't forget to grant the user permissions to some virtual hosts! See 'rabbitmqctl help set_permissions' to learn more.
# rabbitmqctl change_password powervc_mq $rmq_passwd
Changing password for user "powervc_mq" ...
# rabbitmqctl set_permissions -p / powervc_mq ".*" ".*" ".*"
Setting permissions for user "powervc_mq" in vhost "/" ...
- Rabbitmq のポリシーを設定
# rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}'
Setting policy "ha-all" for pattern "" to "{"ha-mode":"all","ha-sync-mode":"automatic"}" with priority "0" for vhost "/" ...
# rabbitmqctl cluster_status
Cluster status of node rabbit@pvc211_test ...
Basics
Cluster name: rabbit@pvc211_test
Disk Nodes
rabbit@pvc211_test
Running Nodes
rabbit@pvc211_test
Versions
rabbit@pvc211_test: RabbitMQ 3.10.8 on Erlang 25.1
Maintenance status
Node: rabbit@pvc211_test, status: not under maintenance
Alarms
(none)
Network Partitions
(none)
Listeners
Node: rabbit@pvc211_test, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@pvc211_test, interface: xxx.xx.xxx.xxx3.104, port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS
Feature flags
Flag: classic_mirrored_queue_version, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled
- ステータスとキューを確認
# rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name messages
ibm-notification_fanout_d4aab1d228704dfd88fb523961ca5a1f 0
conductor_fanout_f7ab41d88f794a9d82f5d9ad267a9125 0
q-plugin 0
conductor_fanout_a0b3e64dc093418ebcec62f116b83c30 0
q-reports-plugin_fanout_893abcac482a4b1486ea4dc3b0890526 0
q-l3-plugin_fanout_572158eb9f5e4321b19a82d7027d4e58 0
q-server-resource-versions_fanout_21210a1d111149ebaa2f22e718076bac 0
q-server-resource-versions_fanout_6787757510a045bd8443178fe321836f 0
q-l3-plugin_fanout_176a70fe4f154a9ea0e4fa5f942ce33f 0
ibm_notifications.error 0
q-server-resource-versions.pvc211_test 0
reply_f95c70a7119348e69f60771dd871cbcb 0
q-server-resource-versions_fanout_e2df1f51f9a848f381fcff87940ade44 0
q-l3-plugin_fanout_d86f05a645ac4a71848db2564cf4206c 0
q-plugin_fanout_11c386ed8d1f474bb20567365b90e676 0
q-reports-plugin_fanout_b5e8af10c9864c0593697e53b3ef3f44 0
q-reports-plugin_fanout_d868b73cc6484bdf94e1876285b31350 0
q-plugin_fanout_2e30f345cf144c61b934d64671bb9bc9 0
q-server-resource-versions_fanout_47c1888488ba4da8a2ebe9e44ee08c66 0
notifications.sample 0
q-l3-plugin_fanout_9356c113ac0841d994e626e4bb071900 0
hapolicy_fanout_0f50ad7726c04aa299932c9ba9d36cdb 0
notifications.error 0
conductor_fanout_3dd44d6345b0489c8c72934128c62a83 0
q-server-resource-versions_fanout_b345dd2c50384086b4dfa0cf4c135067 0
q-plugin.pvc211_test 0
q-reports-plugin_fanout_a648e63e74b143868800310b276b0598 0
q-reports-plugin 0
q-server-resource-versions_fanout_a79137f7b7f142fda174e4490669334f 0
q-l3-plugin.pvc211_test 0
q-l3-plugin_fanout_b2927b13f8b14c638a4bcfc82d2e78ef 0
conductor_fanout_ea192500f4784f46b4d35db23ccca267 0
scheduler 0
hapolicy.pvc211_test 0
q-plugin_fanout_868d75c1e31143fe9160d28f2f46b63c 0
q-l3-plugin_fanout_3676e0968dae4b1fb756a5c378d23879 0
q-plugin_fanout_e5c3a68483a343518cee2e88af8bb1fc 0
conductor_fanout_399f33c359c94385acd0a3ad40edbe06 0
rtpolicy_fanout_6c7ab3ef1a9641759bf71c54ca593b2c 0
rtpolicy.pvc211_test 0
q-reports-plugin_fanout_86cfb6d38c0747bf86d4d089fa4a834a 0
ibm_notifications.sample 0
hapolicy 0
q-plugin_fanout_2dd9bd9ad65b49fe96dd03cb75d5f82d 0
q-server-resource-versions_fanout_b64cbdc1a68e4302ad6aa5134cb521ab 0
notifications.info 0
q-plugin_fanout_bca04640d1304f68bb77baac5f9a61bc 0
notifications.warn 0
q-reports-plugin.pvc211_test 0
q-reports-plugin_fanout_aed864506e5b43c8b9b933efcbfdca17 0
q-reports-plugin_fanout_e5a68c45e3e24ab6b7b10f73b7215cd5 0
conductor_fanout_1618d106e2d84cc7a14ed82bb3c1ab27 0
q-plugin_fanout_5750d7ccdbc2400582f0b250ba0da638 0
ibm_notifications.debug 0
conductor.pvc211_test 0
ibm_notifications.info 0
q-reports-plugin_fanout_6614ed5c311745169c809785106fc81f 0
q-l3-plugin_fanout_19a2c5f9e9484c5daaa51e829e0d3269 0
ibm_notifications.audit 0
ibm-notification.pvc211_test 0
q-server-resource-versions_fanout_b92da8439dac43ac862d5b02810b4e38 0
rtpolicy 0
ibm-notification 0
q-l3-plugin 0
scheduler_fanout_8f88db8997344edda9eae5c4f558b2f1 0
q-reports-plugin_fanout_165df3290aec4d439ff370639635ae90 0
notifications.audit 0
q-reports-plugin_fanout_4d6573fbbf624000b34eb6f897c7d015 0
q-l3-plugin_fanout_3bcf25362d7645b3b383ab72a4cf4c2e 0
notifications.debug 0
conductor 0
q-plugin_fanout_4c711279f5a94c209dcaf9ce0517f3bf 0
ibm_notifications.warn 0
q-l3-plugin_fanout_c1a46bd6b62248d4a0a5ee88042f08c8 0
notifications.critical 0
q-plugin_fanout_ab98b0aaee1e4a9cae9bd20161a47360 0
q-server-resource-versions_fanout_ab59b7ff536c4b1894f2c4228cf4301c 0
scheduler.pvc211_test 0
q-server-resource-versions 0
ibm_notifications.critical 0
q-l3-plugin_fanout_729a8c70228044c8ac606ff13a5cd027 0
# pcs resource enable rabbitmq-server-clone
bash: pcs: command not found...
- PowerVC 再起動
# powervc-services stop
Stopping clerk services...
Stopping validator services...
Stopping panko services...
Stopping gnocchi services...
Stopping swift services...
Stopping placement services...
Stopping ui-server services...
Stopping bumblebee services...
Stopping health services...
Stopping ceilometer services...
Stopping nova services...
Stopping neutron services...
Stopping ego services...
Stopping cinder services...
Stopping glance services...
Stopping rabbitmq services...
Stopping httpd services...
Stopping db services...
# powervc-services start
Starting db services...
Starting httpd services...
Starting rabbitmq services...
Starting glance services...
Starting cinder services...
Starting ego services...
Starting neutron services...
Starting nova services...
Starting ceilometer services...
Starting health services...
Starting bumblebee services...
Starting ui-server services...
Starting placement services...
Starting swift services...
Starting gnocchi services...
Starting panko services...
Starting validator services...
Starting clerk services...
- 確認
# powervc-services status
● clerk-api.service - clerk API Server
Active: active (running) since Wed 2023-07-26 18:43:06 EDT; 46s ago
● ttv-validation-api.service - TTV API Server
Active: active (running) since Wed 2023-07-26 18:43:06 EDT; 46s ago
● panko-api.service - OpenStack Panko API Server
Active: active (running) since Wed 2023-07-26 18:43:06 EDT; 46s ago
● openstack-gnocchi-metricd.service - OpenStack Gnocchi metricd Server
Active: active (running) since Wed 2023-07-26 18:43:06 EDT; 46s ago
● openstack-swift-account.service - OpenStack Object Storage (swift) - Account Server
Active: active (running) since Wed 2023-07-26 18:43:05 EDT; 47s ago
● openstack-swift-object.service - OpenStack Object Storage (swift) - Object Server
Active: active (running) since Wed 2023-07-26 18:43:05 EDT; 47s ago
● openstack-swift-proxy.service - OpenStack Object Storage (swift) - Proxy Server
Active: active (running) since Wed 2023-07-26 18:43:05 EDT; 47s ago
● openstack-swift-container.service - OpenStack Object Storage (swift) - Container Server
Active: active (running) since Wed 2023-07-26 18:43:05 EDT; 47s ago
● openstack-swift-object-replicator.service - OpenStack Object Storage (swift) - Object Replicator
Active: active (running) since Wed 2023-07-26 18:43:05 EDT; 47s ago
● memcached.service - memcached daemon
Active: active (running) since Wed 2023-07-26 18:42:37 EDT; 1min 15s ago
● placement-api.service - Placement API Server
Active: active (running) since Wed 2023-07-26 18:43:05 EDT; 47s ago
● mongod.service - MongoDB Database Server
Active: active (running) since Wed 2023-07-26 18:42:50 EDT; 1min 2s ago
● powervc-ui-server.service - Powervc ui server
Active: active (running) since Wed 2023-07-26 18:43:05 EDT; 47s ago
● powervc-bumblebee.service - PowerVC Bumblebee
Active: active (running) since Wed 2023-07-26 18:42:47 EDT; 1min 5s ago
● powervc-cinder-ibm-health.service - PowerVC cinder monitor service
Active: active (running) since Wed 2023-07-26 18:42:47 EDT; 1min 5s ago
● powervc-nova-ibm-health.service - PowerVC nova monitor service
Active: active (running) since Wed 2023-07-26 18:42:47 EDT; 1min 5s ago
● powervc-neutron-ibm-health.service - PowerVC neutron monitor service
Active: active (running) since Wed 2023-07-26 18:42:47 EDT; 1min 5s ago
● openstack-ceilometer-notification.service - OpenStack ceilometer notification agent
Active: active (running) since Wed 2023-07-26 18:43:29 EDT; 23s ago
● openstack-ceilometer-polling.service - OpenStack ceilometer polling agent
Active: active (running) since Wed 2023-07-26 18:43:29 EDT; 24s ago
● openstack-nova-api.service - OpenStack Nova API Server
Active: active (running) since Wed 2023-07-26 18:42:47 EDT; 1min 6s ago
● openstack-nova-conductor.service - OpenStack Nova Conductor Server
Active: active (running) since Wed 2023-07-26 18:42:47 EDT; 1min 6s ago
● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server
Active: active (running) since Wed 2023-07-26 18:43:12 EDT; 40s ago
● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server
Active: active (running) since Wed 2023-07-26 18:42:47 EDT; 1min 6s ago
● openstack-nova-compute-910541B_785CA21.service - OpenStack Nova Compute Server
Active: active (running) since Wed 2023-07-26 18:42:47 EDT; 1min 6s ago
● neutron-server.service - OpenStack Neutron Server
Active: active (running) since Wed 2023-07-26 18:42:47 EDT; 1min 6s ago
● openstack-nova-ibm-notification.service - OpenStack Nova IBM Notification
Active: active (running) since Wed 2023-07-26 18:42:47 EDT; 1min 6s ago
● openstack-nova-ibm-ego-ha-service.service - OpenStack Nova IBM EGO HA Service
Active: active (running) since Wed 2023-07-26 18:42:47 EDT; 1min 6s ago
● openstack-nova-ibm-ego-resource-optimization.service - OpenStack Nova IBM EGO Resource Optimization
Active: active (running) since Wed 2023-07-26 18:42:47 EDT; 1min 6s ago
● openstack-cinder-api.service - OpenStack Cinder API Server
Active: active (running) since Wed 2023-07-26 18:42:46 EDT; 1min 7s ago
● openstack-cinder-scheduler.service - OpenStack Cinder Scheduler Server
Active: active (running) since Wed 2023-07-26 18:42:46 EDT; 1min 7s ago
● openstack-cinder-volume-xxx.xx.xxx.xxx.service - OpenStack Cinder Volume Server
Active: active (running) since Wed 2023-07-26 18:42:46 EDT; 1min 7s ago
● powervc-cinder-conductor.service - PowerVC Cinder Conductor Server
Active: active (running) since Wed 2023-07-26 18:42:46 EDT; 1min 7s ago
● openstack-glance-api.service - OpenStack Image Service (code-named Glance) API server
Active: active (running) since Wed 2023-07-26 18:42:46 EDT; 1min 7s ago
● rabbitmq-server.service - RabbitMQ broker
Active: active (running) since Wed 2023-07-26 18:42:46 EDT; 1min 7s ago
● httpd.service - The Apache HTTP Server
Active: active (running) since Wed 2023-07-26 18:42:37 EDT; 1min 16s ago
● memcached.service - memcached daemon
Active: active (running) since Wed 2023-07-26 18:42:37 EDT; 1min 16s ago
● mariadb.service - MariaDB 10.5 database server
Active: active (running) since Wed 2023-07-26 18:42:37 EDT; 1min 16s ago
無事サービスが起動しています。
おわりに
これで回復しなければ再インストールが必要かもと考えていたところでしたので正常に戻って一安心です。
この実施は一例です。繰り返しになりますが、問題判別や解決は製品サポートへのご確認をお勧めします。
以上です。