OpenShift 4.4を構築してアプリを動かしてみる - 踏み台サーバの構築編 -

1.ツールのインストール

1.1.yum の更新

パッケージ情報の更新と、パッケージの更新を行います。

$ sudo yum update -y

1.2.ビルドツール、ネットワークツールのインストール

$ sudo yum install -y bind-utils net-tools vim

2.ネットワークの設定

2.1変更前のネットワーク確認

ens192とens224の2つが認識されているか確認します。
認識されていない場合、CentOSのセットアップでネットワークの有効化に失敗しています。

nmtuiなどのコマンドで有効化が可能ですが、この後の内容と整合性が取れなくなるため
以下の通りでない場合、CentOSの再インストールをお勧めします。

$ nmcli

ens192: connected to ens192
        "VMware VMXNET3"
        ethernet (vmxnet3), 00:0C:29:18:D5:2C, hw, mtu 1500
        ip4 default, ip6 default
        inet4 192.168.11.35/24
        route4 0.0.0.0/0
        route4 192.168.11.0/24
        inet6 240b:10:89c0:c300:fc93:6d1e:dc82:623/64
        inet6 fe80::72eb:c375:628d:321a/64
        route6 240b:10:89c0:c300::/64
        route6 ::/0
        route6 ff00::/8
        route6 fe80::/64

ens224: disconnected
        "VMware VMXNET3"
        1 connection available
        ethernet (vmxnet3), 00:0C:29:18:D5:36, hw, mtu 1500

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

• ens192 : 外部ネットワーク
• ens224 : VMWare内の内部ネットワーク

2.2.内部ネットワークの手動設定

$ sudo nmcli con modify ens224 ipv4.method manual ipv4.address 172.16.0.254/24
$ sudo nmcli con down ens224
$ sudo nmcli con up ens224 

2.2.1.Failed to modify connection 'ens224': Insufficient privileges

一般ユーザでnmcliからネットワークを更新すると、権限がないため変更できないとエラーが表示されます。
sudoコマンドの付け忘れ、または、rootユーザで再実行してください。

Error: Failed to modify connection 'ens224': Insufficient privileges

2.3.内部ネットワークの自動接続設定

手動設定でIPを割り当てるため、再起動時に、内部ネットワークのens224がダウンして
他のサービスが起動しない状況が発生する場合があります。
そのため、手動割り当てでも自動接続を有効にします。

$ sudo nmcli con modify ens224 connection.autoconnect yes

2.4.変更後のネットワーク確認

• ens224が認識されていること
• ens224がens224がconnectedとなっていること

$ nmcli
ens192: connected to ens192
        "VMware VMXNET3"
        ethernet (vmxnet3), 00:0C:29:18:D5:2C, hw, mtu 1500
        ip4 default, ip6 default
        inet4 192.168.11.35/24
        route4 0.0.0.0/0
        route4 192.168.11.0/24
        inet6 240b:10:89c0:c300:fc93:6d1e:dc82:623/64
        inet6 fe80::72eb:c375:628d:321a/64
        route6 240b:10:89c0:c300::/64
        route6 ::/0
        route6 ff00::/8
        route6 fe80::/64

ens224: connected to ens224
        "VMware VMXNET3"
        ethernet (vmxnet3), 00:0C:29:18:D5:36, hw, mtu 1500
        inet4 172.16.0.254/24
        route4 172.16.0.0/24
        inet6 fe80::ee5c:3728:e10d:5215/64
        route6 fe80::/64
        route6 ff00::/8

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

DNS configuration:
        servers: 192.168.11.254
        interface: ens192

        servers: 240b:10:89c0:c300:5a52:8aff:fe6f:1c13
        interface: ens192

connection.autoconnectがyesになっていること

$ sudo nmcli con show ens224

connection.id:                          ens224
connection.uuid:                        79f644db-fa2a-410e-ae79-e88c007fa2b6
connection.stable-id:                   --
connection.type:                        802-3-ethernet
connection.interface-name:              ens224
connection.autoconnect:                 yes
connection.autoconnect-priority:        0
connection.autoconnect-retries:         -1 (default)
connection.multi-connect:               0 (default)
connection.auth-retries:                -1
connection.timestamp:                   1610544649
connection.read-only:                   no
connection.permissions:                 --
connection.zone:                        --

3.IPv6無効化、IPv4ルーティング

3.1.設定ファイルの新規作成

$ sudo vi /etc/sysctl.d/99-custom.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv4.ip_forward = 1  

3.2.設定ファイルの内容確認

$ sudo cat /etc/sysctl.d/99-custom.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv4.ip_forward = 1  

3.3.設定ファイルの適用

$ sudo sysctl -p /etc/sysctl.d/99-custom.conf

4.ファイヤーウォールの設定

4.1現在のゾーンを確認

$ sudo firewall-cmd --get-active-zones
public
  interfaces: ens192 ens224

4.2.デフォルトゾーンの変更

$ sudo firewall-cmd --set-default-zone=trusted
success

4.3.変更結果確認

$ sudo firewall-cmd --get-active-zones
trusted
  interfaces: ens192 ens224 

4.4.マスカレードの設定追加

$ sudo firewall-cmd --add-masquerade --zone=trusted --permanent
success

4.5.設定の反映

$ sudo firewall-cmd --reload
success

4.6.ゾーン設定の反映確認

$ sudo firewall-cmd --get-active-zones
trusted
  interfaces: ens192 ens224

4.7.マスカレード設定の反映確認

$ sudo firewall-cmd --list-all --permanent --zone=trusted
trusted
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: yes
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

5.DNSサーバの設定

5.1.dnsmasqのインストール

$ sudo yum install -y dnsmasq

Last metadata expiration check: 2:23:13 ago on Mon 11 Jan 2021 06:53:26 PM JST.
Dependencies resolved.
================================================================================
 Package          Architecture    Version              Repository          Size
================================================================================
Installing:
 dnsmasq          x86_64          2.79-13.el8          appstream          318 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 318 k
Installed size: 647 k
Downloading Packages:
dnsmasq-2.79-13.el8.x86_64.rpm                  220 kB/s | 318 kB     00:01
--------------------------------------------------------------------------------
Total                                           147 kB/s | 318 kB     00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1
  Running scriptlet: dnsmasq-2.79-13.el8.x86_64                             1/1
  Installing       : dnsmasq-2.79-13.el8.x86_64                             1/1
  Running scriptlet: dnsmasq-2.79-13.el8.x86_64                             1/1
  Verifying        : dnsmasq-2.79-13.el8.x86_64                             1/1

Installed:
  dnsmasq-2.79-13.el8.x86_64

Complete!

5.2.既存の設定ファイルを退避

$ sudo mv /etc/dnsmasq.conf /etc/dnsmasq.def

5.3.設定ファイルの編集

OpenShift 4.4以下の場合、master-0やetcd-1などを変更するとOpenShiftがインストール完了できない場合があるため
ホスト名の変更は避けた方が良いです。（ハマりました。）

ドメイン名は「lab.openshift.local」にしていますが、自由に変更して構いません。
ただし変更した場合、この後のresolveの設定ファイルも整合性をとってください。

$ sudo vi /etc/dnsmasq.conf

port=53
domain-needed
bogus-priv
resolv-file=/etc/resolv.dnsmasq
no-poll
address=/apps.lab.openshift.local/172.16.0.254 # bastionのIPアドレス
no-dhcp-interface=ens192
expand-hosts
domain=lab.openshift.local
dhcp-range=172.16.0.100,172.16.0.200,255.255.255.0,12h
dhcp-host=xx:xx:xx:xx:xx:xx,bootstrap,172.16.0.100 # bootstrapのMACアドレス
dhcp-host=xx:xx:xx:xx:xx:xx,master-0,172.16.0.110　# master-0のMACアドレス
dhcp-host=xx:xx:xx:xx:xx:xx,master-1,172.16.0.111  # master-1のMACアドレス
dhcp-host=xx:xx:xx:xx:xx:xx,master-2,172.16.0.112　# master-2のMACアドレス
dhcp-host=xx:xx:xx:xx:xx:xx,worker-0,172.16.0.120  # worker-0のMACアドレス
dhcp-host=xx:xx:xx:xx:xx:xx,worker-1,172.16.0.121  # worker-1のMACアドレス
dhcp-option=option:dns-server,172.16.0.254　　　　　# bastionのIPアドレス
dhcp-option=option:netmask,255.255.255.0
dhcp-leasefile=/var/lib/dnsmasq/dnsmasq.leases
srv-host=_etcd-server-ssl._tcp.lab.openshift.local,etcd-0.lab.openshift.local,2380,0,10 # 注意
srv-host=_etcd-server-ssl._tcp.lab.openshift.local,etcd-1.lab.openshift.local,2380,0,10 # 注意
srv-host=_etcd-server-ssl._tcp.lab.openshift.local,etcd-2.lab.openshift.local,2380,0,10 # 注意
log-dhcp
log-facility=/var/log/dnsmasq.log
conf-dir=/etc/dnsmasq.d,.rpmnew,.rpmsave,.rpmorig

5.4.外部ネットワークのDNSサーバ変更

$ sudo nmcli con modify ens192 ipv4.dns 127.0.0.1
$ sudo nmcli con down ens192
$ sudo nmcli con up ens192

5.5.DNSサーバの設定ファイル編集

$ sudo vi /etc/resolv.conf 
search lab.openshift.local
nameserver 127.0.0.1

5.6.NetworkManagerの設定変更

NetworkManagerからresolve.confの上書きを防ぐため、NetworkManagerの設定ファイルを変更します。

$ sudo vi /etc/NetworkManager/NetworkManager.conf

[main]
#plugins=ifcfg-rh
dns=none #追記

5.7.内部ネットワークのDNSサーバ設定を編集

• nameserverのアドレスはbastionのIPアドレスではありません。
• nameserverのアドレスは、外部ネットワークのルータのIPアドレスです。
  192.168.11.1は筆者のネットワーク環境のルータのIPアドレスのため、
  ご自身のネットワーク環境に合わせて、設定してください。

$ sudo vi /etc/resolv.dnsmasq 
search lab.openshift.local
nameserver 192.168.11.1 # 外部ネットワークのルータのIPアドレス

5.8.hostsの追記

行末に以下を追加
こちらも、master-0やetcd-1などのホスト名を変更すると正常に動作しない場合があるため避けてください。

IPアドレスは、boostrapが100番、masterが110番台、workerが120番台にした場合の例です。

$ sudo vi /etc/hosts
# OpenShift
192.168.11.35   api        # bastionの外部ネットワークアドレス
172.16.0.254    api-int　　# bastionの内部ネットワークアドレス
172.16.0.100    bootstrap
172.16.0.110    master-0
172.16.0.111    master-1
172.16.0.112    master-2
172.16.0.110    etcd-0
172.16.0.111    etcd-1
172.16.0.112    etcd-2
172.16.0.120    worker-0
172.16.0.121    worker-1

5.9.dnsmasqサービスの起動

$ sudo systemctl enable --now dnsmasq.service
Created symlink /etc/systemd/system/multi-user.target.wants/dnsmasq.service → /usr/lib/systemd/system/dnsmasq.service.

5.10.dnsmasqサービスのステータス確認

$ systemctl status dnsmasq
● dnsmasq.service - DNS caching server.
   Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; enabled; vendor pre>
   Active: active (running) since Mon 2021-01-11 21:33:02 JST; 38s ago
 Main PID: 62060 (dnsmasq)
    Tasks: 1 (limit: 11082)
   Memory: 2.0M
   CGroup: /system.slice/dnsmasq.service
           mq62060 /usr/sbin/dnsmasq -k

Jan 11 21:33:02 center.network.local systemd[1]: Started DNS caching server..

5.11.ログファイルの権限変更

再起動時に書き込み権限がなくてエラーとなるため、書き込み権限を付与します。

$ sudo chmod 666 /var/log/dnsmasq.log

6.SELinuxの設定変更

$ sudo setsebool -P httpd_read_user_content 1
$ sudo setsebool -P haproxy_connect_any 1

7.ロードバランサの設定

7.1.haproxyのインストール

$ sudo yum install -y haproxy

7.2.haproxyの設定

$ sudo vi cat /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# for ocp4
#---------------------------------------------------------------------
frontend K8s-api
    bind *:6443
    option tcplog
    mode tcp
    default_backend     api-6443

frontend Machine-config
    bind *:22623
    option tcplog
    mode tcp
    default_backend     config-22623

frontend Ingress-http
    bind *:80
    option tcplog
    mode tcp
    default_backend http-80

frontend Ingress-https
    bind *:443
    option tcplog
    mode tcp
    default_backend     https-443

backend api-6443
    mode tcp
    balance     roundrobin
    option  ssl-hello-chk 
    server  bootstrap bootstrap.lab.openshift.local:6443 check
    server  master-0 master-0.lab.openshift.local:6443 check
    server  master-1 master-1.lab.openshift.local:6443 check
    server  master-2 master-2.lab.openshift.local:6443 check

backend config-22623
    mode tcp
    balance     roundrobin
    server  bootstrap bootstrap.lab.openshift.local:22623 check
    server  master-0 master-0.lab.openshift.local:22623 check
    server  master-1 master-1.lab.openshift.local:22623 check
    server  master-2 master-2.lab.openshift.local:22623 check

backend http-80
    mode tcp
    balance     roundrobin
    server  worker-0 worker-0.lab.openshift.local:80 check
    server  worker-1 worker-1.lab.openshift.local:80 check

backend https-443
    mode tcp
    balance     roundrobin
    option      ssl-hello-chk
    server  worker-0 worker-0.lab.openshift.local:443 check
    server  worker-1 worker-1.lab.openshift.local:443 check 

7.3.haproxyサービスの起動

$ sudo systemctl enable --now haproxy
$ sudo systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-01-14 00:23:52 JST; 8min ago
  Process: 1858 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 1860 (haproxy)
    Tasks: 2 (limit: 23312)
   Memory: 2.8M
   CGroup: /system.slice/haproxy.service
           tq1860 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           mq1862 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

Jan 14 00:23:52 bastion haproxy[1860]:    | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Jan 14 00:23:52 bastion haproxy[1860]: [WARNING] 013/002352 (1860) : config : missing timeouts for backend 'http-80'.
Jan 14 00:23:52 bastion haproxy[1860]:    | While not properly invalid, you will certainly encounter various problems
Jan 14 00:23:52 bastion haproxy[1860]:    | with such a configuration. To fix this, please ensure that all following
Jan 14 00:23:52 bastion haproxy[1860]:    | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Jan 14 00:23:52 bastion haproxy[1860]: [WARNING] 013/002352 (1860) : config : missing timeouts for backend 'https-443'.
Jan 14 00:23:52 bastion haproxy[1860]:    | While not properly invalid, you will certainly encounter various problems
Jan 14 00:23:52 bastion haproxy[1860]:    | with such a configuration. To fix this, please ensure that all following
Jan 14 00:23:52 bastion haproxy[1860]:    | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Jan 14 00:23:52 bastion systemd[1]: Started HAProxy Load Balancer.

7.3.[ERROR] haproxyの起動に失敗する場合

$ sudo systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2021-01-13 22:30:49 JST; 1h 39min ago
  Process: 1259 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $OPTIONS (code=exited, status=1/FAILURE)

Jan 13 22:30:49 bastion haproxy[1259]: [ALERT] 012/223049 (1259) : parsing [/etc/haproxy/haproxy.cfg:40] : 'server master-1' : could not resolve address 'master-1.lab.openshift.local'.
Jan 13 22:30:49 bastion haproxy[1259]: [ALERT] 012/223049 (1259) : parsing [/etc/haproxy/haproxy.cfg:41] : 'server master-2' : could not resolve address 'master-2.lab.openshift.local'.
Jan 13 22:30:49 bastion haproxy[1259]: [ALERT] 012/223049 (1259) : parsing [/etc/haproxy/haproxy.cfg:46] : 'server worker-0' : could not resolve address 'worker-0.lab.openshift.local'.
Jan 13 22:30:49 bastion haproxy[1259]: [ALERT] 012/223049 (1259) : parsing [/etc/haproxy/haproxy.cfg:47] : 'server worker-1' : could not resolve address 'worker-1.lab.openshift.local'.
Jan 13 22:30:49 bastion haproxy[1259]: [ALERT] 012/223049 (1259) : parsing [/etc/haproxy/haproxy.cfg:53] : 'server worker-0' : could not resolve address 'worker-0.lab.openshift.local'.
Jan 13 22:30:49 bastion haproxy[1259]: [ALERT] 012/223049 (1259) : parsing [/etc/haproxy/haproxy.cfg:54] : 'server worker-1' : could not resolve address 'worker-1.lab.openshift.local'.
Jan 13 22:30:49 bastion haproxy[1259]: [ALERT] 012/223049 (1259) : Failed to initialize server(s) addr.
Jan 13 22:30:49 bastion systemd[1]: haproxy.service: Control process exited, code=exited status=1
Jan 13 22:30:49 bastion systemd[1]: haproxy.service: Failed with result 'exit-code'.
Jan 13 22:30:49 bastion systemd[1]: Failed to start HAProxy Load Balancer.

1. haproxyを再起動してみる

再起動に失敗した例

$sudo systemctl restart haproxy
Job for haproxy.service failed because the control process exited with error code.
See "systemctl status haproxy.service" and "journalctl -xe" for details.

2. dnsmasqを再起動してみる

haproxyがdnsmasqで改善しなかった例

$ sudo systemctl restart dnsmasq.service
$ sudo systemctl restart haproxy
Job for haproxy.service failed because the control process exited with error code.
See "systemctl status haproxy.service" and "journalctl -xe" for details.

3. hostsの設定を確認する

エラーとなった could not resolve address [hostname]が存在するか確認する。
確認したところ、いずれも存在していた場合の例

$ sudo cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.11.35   api
172.16.0.254    api-int
172.16.0.110    etcd-0
172.16.0.111    etcd-1
172.16.0.112    etcd-2

172.16.0.100    bootstrap
172.16.0.110    master-0
172.16.0.111    master-1
172.16.0.112    master-2
172.16.0.120    worker-0
172.16.0.121    worker-1

4. resolveを確認する

再起動時やネットワークの再接続で、勝手に設定が戻る場合があります。

• 設定が戻っていた例

$ sudo cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 192.168.11.254
nameserver 127.0.0.1
nameserver 240b:10:89c0:c300:5a52:8aff:fe6f:1c13

• 修正後の設定

$ sudo vi /etc/resolv.conf
search lab.openshift.local
nameserver 127.0.0.1

• 設定の反映とサービスの再起動

$ sudo systemctl restart dnsmasq.service
$ sudo systemctl restart haproxy

5. 内部ネットワークの接続状態を確認する

再起動時などで内部ネットワークのens224がダウンしている場合があります。

• ens224がダウンdisconnectedしている例

$ sudo nmcli
ens192: connected to ens192
        "VMware VMXNET3"
        ethernet (vmxnet3), 00:0C:29:4D:C3:BA, hw, mtu 1500
        ip4 default, ip6 default
        inet4 192.168.11.35/24
        route4 0.0.0.0/0
        route4 192.168.11.0/24
        inet6 240b:10:89c0:c300:44e2:485f:e549:628c/64
        inet6 fe80::c9c:8b64:33fd:60db/64
        route6 240b:10:89c0:c300::/64
        route6 ::/0
        route6 fe80::/64
        route6 ff00::/8

ens224: disconnected
        "VMware VMXNET3"
        1 connection available
        ethernet (vmxnet3), 00:0C:29:4D:C3:C4, hw, mtu 1500

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

DNS configuration:
        servers: 192.168.11.254 127.0.0.1
        interface: ens192

        servers: 240b:10:89c0:c300:5a52:8aff:fe6f:1c13
        interface: ens192

Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.

Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.

• ens224の接続をアップする

$ sudo nmcli con up ens224
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4)

• 接続状態を確認する

$ sudo nmcli 
ens192: connected to ens192
        "VMware VMXNET3"
        ethernet (vmxnet3), 00:0C:29:4D:C3:BA, hw, mtu 1500
        ip4 default, ip6 default
        inet4 192.168.11.35/24
        route4 0.0.0.0/0
        route4 192.168.11.0/24
        inet6 240b:10:89c0:c300:44e2:485f:e549:628c/64
        inet6 fe80::c9c:8b64:33fd:60db/64
        route6 240b:10:89c0:c300::/64
        route6 ::/0
        route6 fe80::/64
        route6 ff00::/8

ens224: connected to ens224
        "VMware VMXNET3"
        ethernet (vmxnet3), 00:0C:29:4D:C3:C4, hw, mtu 1500
        inet4 172.16.0.254/24
        route4 172.16.0.0/24
        inet6 fe80::4492:8a53:68c3:7d6c/64
        route6 fe80::/64
        route6 ff00::/8

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

DNS configuration:
        servers: 192.168.11.254 127.0.0.1
        interface: ens192

        servers: 240b:10:89c0:c300:5a52:8aff:fe6f:1c13
        interface: ens192

Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.

Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.

• ens224が接続後、resolve.confの設定を確認し、nameserverが127.0.0.1になっていることを確認する

$ sudo cat /etc/resolv.conf
# Generated by NetworkManager
# nameserver 192.168.11.254
nameserver 127.0.0.1
# nameserver 240b:10:89c0:c300:5a52:8aff:fe6f:1c13

• 各種サービスの再起動

$ sudo systemctl restart dnsmasq.service
$ sudo systemctl restart haproxy

8.httpサーバ(nginx)

8.1.nginxのインストール

インストールに必要なファイルを配布するために、Webサーバが必要で今回はnginxを利用します。

$ sudo yum install -y nginx

8.2.nginxの設定

80番ポートはhaproxyと重複するため、nginxは8080番ポートを利用します。

• 変更点のみ記載

$ sudo vi /etc/nginx/nginx.conf
http {

    server {
        listen       8080 default_server;   # ポート番号変更：80->8008
#        listen       [::]:80 default_server;   # コメントアウト

    }

    disable_symlinks off;   # 追加
}

8.3.nginxサービスの起動

$ sudo systemctl enable --now nginx

9.各種ステータスの確認

OpenShiftのインストールを進める前に、各種ステータスを確認します

9.1. ネットワーク状態の確認

ens192とens224が認識されていること
ens192とens224がconnectedとなってIPが割り当てられていること

$ sudo nmcli
ens192: connected to ens192
        "VMware VMXNET3"
        ethernet (vmxnet3), 00:0C:29:4D:C3:BA, hw, mtu 1500
        ip4 default, ip6 default
        inet4 192.168.11.35/24
        route4 0.0.0.0/0
        route4 192.168.11.0/24
        inet6 240b:10:89c0:c300:44e2:485f:e549:628c/64
        inet6 fe80::c9c:8b64:33fd:60db/64
        route6 240b:10:89c0:c300::/64
        route6 ::/0
        route6 ff00::/8
        route6 fe80::/64

ens224: connected to ens224
        "VMware VMXNET3"
        ethernet (vmxnet3), 00:0C:29:4D:C3:C4, hw, mtu 1500
        inet4 172.16.0.254/24
        route4 172.16.0.0/24
        inet6 fe80::4492:8a53:68c3:7d6c/64
        route6 fe80::/64
        route6 ff00::/8

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

DNS configuration:
        servers: 192.168.11.254 127.0.0.1
        interface: ens192

        servers: 240b:10:89c0:c300:5a52:8aff:fe6f:1c13
        interface: ens192

Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.

Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.

9.2. dnsmasqサービスのステータス確認

Activateがactivateであること

$ sudo systemctl status dnsmasq.service
● dnsmasq.service - DNS caching server.
   Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-01-14 09:49:21 JST; 13min ago
 Main PID: 1003 (dnsmasq)
    Tasks: 1 (limit: 23215)
   Memory: 1.4M
   CGroup: /system.slice/dnsmasq.service
           mq1003 /usr/sbin/dnsmasq -k

Jan 14 09:49:21 bastion systemd[1]: Started DNS caching server..

9.3. haproxyサービスのステータス確認

Activateがactivateであること

systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-01-14 09:49:23 JST; 15min ago
  Process: 1265 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 1322 (haproxy)
    Tasks: 2 (limit: 23215)
   Memory: 4.8M
   CGroup: /system.slice/haproxy.service
           tq1322 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           mq1324 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

Jan 14 09:49:23 bastion haproxy[1322]:    | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Jan 14 09:49:23 bastion haproxy[1322]: [WARNING] 013/094923 (1322) : config : missing timeouts for backend 'http-80'.
Jan 14 09:49:23 bastion haproxy[1322]:    | While not properly invalid, you will certainly encounter various problems
Jan 14 09:49:23 bastion haproxy[1322]:    | with such a configuration. To fix this, please ensure that all following
Jan 14 09:49:23 bastion haproxy[1322]:    | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Jan 14 09:49:23 bastion haproxy[1322]: [WARNING] 013/094923 (1322) : config : missing timeouts for backend 'https-443'.
Jan 14 09:49:23 bastion haproxy[1322]:    | While not properly invalid, you will certainly encounter various problems
Jan 14 09:49:23 bastion haproxy[1322]:    | with such a configuration. To fix this, please ensure that all following
Jan 14 09:49:23 bastion haproxy[1322]:    | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Jan 14 09:49:23 bastion systemd[1]: Started HAProxy Load Balancer.

9.4. nginxサービスのステータス確認

Activateがactivateであること

systemctl status nginx.service
● nginx.service - The nginx HTTP and reverse proxy server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-01-14 09:49:21 JST; 16min ago
  Process: 1034 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
  Process: 1009 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
  Process: 1005 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)
 Main PID: 1035 (nginx)
    Tasks: 2 (limit: 23215)
   Memory: 12.1M
   CGroup: /system.slice/nginx.service
           tq1035 nginx: master process /usr/sbin/nginx
           mq1036 nginx: worker process

Jan 14 09:49:21 bastion systemd[1]: Starting The nginx HTTP and reverse proxy server...
Jan 14 09:49:21 bastion nginx[1009]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
Jan 14 09:49:21 bastion nginx[1009]: nginx: configuration file /etc/nginx/nginx.conf test is successful
Jan 14 09:49:21 bastion systemd[1]: nginx.service: Failed to parse PID from file /run/nginx.pid: Invalid argument
Jan 14 09:49:21 bastion systemd[1]: Started The nginx HTTP and reverse proxy server.

9.5. WEBサーバの疎通確認

curlコマンドで、127.0.0.1、172.16.0.254、api-intで実行し
ローカルホスト、内部ネットワーク、DNS解決が正常であるか確認します。
curlコマンドの結果は、すべて同じです。

• 127.0.0.1で確認

$ curl http://127.0.0.1:8080/
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
    <head>
        <title>Test Page for the Nginx HTTP Server on Red Hat Enterprise Linux</title>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
        <style type="text/css">
            /*<![CDATA[*/
            body {
                background-color: #fff;
                color: #000;
                font-size: 0.9em;
                font-family: sans-serif,helvetica;
                margin: 0;
                padding: 0;
            }
            :link {
                color: #c00;
            }
            :visited {
                color: #c00;
            }
            a:hover {
                color: #f50;
            }
            h1 {
                text-align: center;
                margin: 0;
                padding: 0.6em 2em 0.4em;
                background-color: #900;
                color: #fff;
                font-weight: normal;
                font-size: 1.75em;
                border-bottom: 2px solid #000;
            }
            h1 strong {
                font-weight: bold;
                font-size: 1.5em;
            }
            h2 {
                text-align: center;
                background-color: #900;
                font-size: 1.1em;
                font-weight: bold;
                color: #fff;
                margin: 0;
                padding: 0.5em;
                border-bottom: 2px solid #000;
            }
            hr {
                display: none;
            }
            .content {
                padding: 1em 5em;
            }
            .alert {
                border: 2px solid #000;
            }

            img {
                border: 2px solid #fff;
                padding: 2px;
                margin: 2px;
            }
            a:hover img {
                border: 2px solid #294172;
            }
            .logos {
                margin: 1em;
                text-align: center;
            }
            /*]]>*/
        </style>
    </head>

    <body>
        <h1>Welcome to <strong>nginx</strong> on Red Hat Enterprise Linux!</h1>

        <div class="content">
            <p>This page is used to test the proper operation of the
            <strong>nginx</strong> HTTP server after it has been
            installed. If you can read this page, it means that the
            web server installed at this site is working
            properly.</p>

            <div class="alert">
                <h2>Website Administrator</h2>
                <div class="content">
                    <p>This is the default <tt>index.html</tt> page that
                    is distributed with <strong>nginx</strong> on
                    Red Hat Enterprise Linux.  It is located in
                    <tt>/usr/share/nginx/html</tt>.</p>

                    <p>You should now put your content in a location of
                    your choice and edit the <tt>root</tt> configuration
                    directive in the <strong>nginx</strong>
                    configuration file
                    <tt>/etc/nginx/nginx.conf</tt>.</p>

                    <p>For information on Red Hat Enterprise Linux, please visit the <a href="http://www.redhat.com/">Red Hat, Inc. website</a>. The documentation for Red Hat Enterprise Linux is <a href="http://www.redhat.com/docs/manuals/enterprise/">available on the Red Hat, Inc. website</a>.</p>

                </div>
            </div>

            <div class="logos">
                <a href="http://nginx.net/"><img
                    src="nginx-logo.png"
                    alt="[ Powered by nginx ]"
                    width="121" height="32" /></a>
                <a href="http://www.redhat.com/"><img
                    src="poweredby.png"
                    alt="[ Powered by Red Hat Enterprise Linux ]"
                    width="88" height="31" /></a>
            </div>
        </div>
    </body>
</html>

• 内部ネットワークアドレスで確認

$ curl http://172.16.0.254:8080/
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
    <head>
        <title>Test Page for the Nginx HTTP Server on Red Hat Enterprise Linux</title>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
        <style type="text/css">
            /*<![CDATA[*/
            body {
                background-color: #fff;
                color: #000;
                font-size: 0.9em;
                font-family: sans-serif,helvetica;
                margin: 0;
                padding: 0;
            }
            :link {
                color: #c00;
            }
            :visited {
                color: #c00;
            }
            a:hover {
                color: #f50;
            }
            h1 {
                text-align: center;
                margin: 0;
                padding: 0.6em 2em 0.4em;
                background-color: #900;
                color: #fff;
                font-weight: normal;
                font-size: 1.75em;
                border-bottom: 2px solid #000;
            }
            h1 strong {
                font-weight: bold;
                font-size: 1.5em;
            }
            h2 {
                text-align: center;
                background-color: #900;
                font-size: 1.1em;
                font-weight: bold;
                color: #fff;
                margin: 0;
                padding: 0.5em;
                border-bottom: 2px solid #000;
            }
            hr {
                display: none;
            }
            .content {
                padding: 1em 5em;
            }
            .alert {
                border: 2px solid #000;
            }

            img {
                border: 2px solid #fff;
                padding: 2px;
                margin: 2px;
            }
            a:hover img {
                border: 2px solid #294172;
            }
            .logos {
                margin: 1em;
                text-align: center;
            }
            /*]]>*/
        </style>
    </head>

    <body>
        <h1>Welcome to <strong>nginx</strong> on Red Hat Enterprise Linux!</h1>

        <div class="content">
            <p>This page is used to test the proper operation of the
            <strong>nginx</strong> HTTP server after it has been
            installed. If you can read this page, it means that the
            web server installed at this site is working
            properly.</p>

            <div class="alert">
                <h2>Website Administrator</h2>
                <div class="content">
                    <p>This is the default <tt>index.html</tt> page that
                    is distributed with <strong>nginx</strong> on
                    Red Hat Enterprise Linux.  It is located in
                    <tt>/usr/share/nginx/html</tt>.</p>

                    <p>You should now put your content in a location of
                    your choice and edit the <tt>root</tt> configuration
                    directive in the <strong>nginx</strong>
                    configuration file
                    <tt>/etc/nginx/nginx.conf</tt>.</p>

                    <p>For information on Red Hat Enterprise Linux, please visit the <a href="http://www.redhat.com/">Red Hat, Inc. website</a>. The documentation for Red Hat Enterprise Linux is <a href="http://www.redhat.com/docs/manuals/enterprise/">available on the Red Hat, Inc. website</a>.</p>

                </div>
            </div>

            <div class="logos">
                <a href="http://nginx.net/"><img
                    src="nginx-logo.png"
                    alt="[ Powered by nginx ]"
                    width="121" height="32" /></a>
                <a href="http://www.redhat.com/"><img
                    src="poweredby.png"
                    alt="[ Powered by Red Hat Enterprise Linux ]"
                    width="88" height="31" /></a>
            </div>
        </div>
    </body>
</html>

• 内部DNSの確認

$ curl http://api-int:8080/
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
    <head>
        <title>Test Page for the Nginx HTTP Server on Red Hat Enterprise Linux</title>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
        <style type="text/css">
            /*<![CDATA[*/
            body {
                background-color: #fff;
                color: #000;
                font-size: 0.9em;
                font-family: sans-serif,helvetica;
                margin: 0;
                padding: 0;
            }
            :link {
                color: #c00;
            }
            :visited {
                color: #c00;
            }
            a:hover {
                color: #f50;
            }
            h1 {
                text-align: center;
                margin: 0;
                padding: 0.6em 2em 0.4em;
                background-color: #900;
                color: #fff;
                font-weight: normal;
                font-size: 1.75em;
                border-bottom: 2px solid #000;
            }
            h1 strong {
                font-weight: bold;
                font-size: 1.5em;
            }
            h2 {
                text-align: center;
                background-color: #900;
                font-size: 1.1em;
                font-weight: bold;
                color: #fff;
                margin: 0;
                padding: 0.5em;
                border-bottom: 2px solid #000;
            }
            hr {
                display: none;
            }
            .content {
                padding: 1em 5em;
            }
            .alert {
                border: 2px solid #000;
            }

            img {
                border: 2px solid #fff;
                padding: 2px;
                margin: 2px;
            }
            a:hover img {
                border: 2px solid #294172;
            }
            .logos {
                margin: 1em;
                text-align: center;
            }
            /*]]>*/
        </style>
    </head>

    <body>
        <h1>Welcome to <strong>nginx</strong> on Red Hat Enterprise Linux!</h1>

        <div class="content">
            <p>This page is used to test the proper operation of the
            <strong>nginx</strong> HTTP server after it has been
            installed. If you can read this page, it means that the
            web server installed at this site is working
            properly.</p>

            <div class="alert">
                <h2>Website Administrator</h2>
                <div class="content">
                    <p>This is the default <tt>index.html</tt> page that
                    is distributed with <strong>nginx</strong> on
                    Red Hat Enterprise Linux.  It is located in
                    <tt>/usr/share/nginx/html</tt>.</p>

                    <p>You should now put your content in a location of
                    your choice and edit the <tt>root</tt> configuration
                    directive in the <strong>nginx</strong>
                    configuration file
                    <tt>/etc/nginx/nginx.conf</tt>.</p>

                    <p>For information on Red Hat Enterprise Linux, please visit the <a href="http://www.redhat.com/">Red Hat, Inc. website</a>. The documentation for Red Hat Enterprise Linux is <a href="http://www.redhat.com/docs/manuals/enterprise/">available on the Red Hat, Inc. website</a>.</p>

                </div>
            </div>

            <div class="logos">
                <a href="http://nginx.net/"><img
                    src="nginx-logo.png"
                    alt="[ Powered by nginx ]"
                    width="121" height="32" /></a>
                <a href="http://www.redhat.com/"><img
                    src="poweredby.png"
                    alt="[ Powered by Red Hat Enterprise Linux ]"
                    width="88" height="31" /></a>
            </div>
        </div>
    </body>
</html>

以上です。お疲れ様でした。