1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

@yasuda2000_q(安田)

Agent BaseのOpenShift構築記録

1
Posted at

#0 基本情報

ESXi上にUPIでインストールする方法となります。

サーバの役割 ホスト名 IPアドレス MACアドレス vCPU数 メモリ ストレージ
踏み台＆DNS＆ロードバランサー 005-001-rhel9jpsv 10.0.5.1 どうでもいい 4 8GB 100GB×1(OS用)、500GB×1(NFSサーバデータ用)
OpenShiftマスターノード① 005-011-ocpmstr01 10.0.5.11 00:50:56:00:05:11 12 24GB 200GB
OpenShiftマスターノード② 005-012-ocpmstr02 10.0.5.12 00:50:56:00:05:12 12 24GB 200GB
OpenShiftマスターノード③ 005-013-ocpmstr03 10.0.5.13 00:50:56:00:05:13 12 24GB 200GB
OpenShiftワーカーノード① 005-021-ocpwrkr01 10.0.5.21 00:50:56:00:05:21 12 24GB 200GB
OpenShiftワーカーノード② 005-021-ocpwrkr02 10.0.5.22 00:50:56:00:05:22 12 24GB 200GB
項目 設定値
サブネットマスク 255.255.0.0
デフォルトゲートウェイ 10.0.1.1
クラスタードメイン名 ocp01.example.com

※補足
・[踏み台＆DNS＆ロードバランサー]にrhel-9.6-x86_64-dvd.isoでOSインストールした状態から始めます
・コマンドはすべてsudo su -実行してrootユーザで行うこととします
・実行するコマンドの前は$、コメントアウトは#と表記します
・ESXi上のOpenShiftノードの仮想マシンは、詳細パラメータで[disk.EnableUUID=TRUE]を設定します

事前準備
・[https://access.redhat.com/downloads/content/290/ver=4.20/rhel---9/4.20.0/x86_64/product-software]から[OpenShift v4.20.0 Linux Installer(openshift-install-linux-4.20.0.tar.gz)]と[OpenShift v4.20.0 Linux Clients(openshift-client-linux-amd64-rhel9-4.20.0.tar.gz)]をダウンロードする
・[https://console.redhat.com/openshift/install/pull-secret]から[Pull secret]をダウンロードする
・OpenShiftの各ノードにSSHログインするために使用するSSHキーペアを作成する

#1. [踏み台＆DNS＆ロードバランサー]DNS設定

①

何らかの方法で[OpenShift v4.20.0 Linux Installer(openshift-install-linux-4.20.0.tar.gz)]を展開して得られる[openshift-install]を[/root/work/openshift]に配置します。また、[OpenShift v4.20.0 Linux Clients(openshift-client-linux-amd64-rhel9-4.20.0.tar.gz)]を展開して得られる[oc]と[kubectl]を/usr/local/binに配置します。自分はいつもESXiのデータストアをファイルサーバ代わりにしています。

$ ls /root/work/openshift
openshift-install
$ ls /usr/local/bin
kubectl  oc

②

踏み台＆DNS＆ロードバランサーでDNSの設定をします。

$ dnf check-update
省略
$ dnf upgrade -y
省略
$ systemctl stop dnsmasq
$ systemctl disable dnsmasq
$ dnf install bind bind-utils -y
省略
$ cp /etc/named.conf /etc/named.conf.bak
$ vi /etc/named.conf
##### 編集 #####
$ cat /etc/named.conf
options {
    listen-on port 53 { 10.0.5.1; 127.0.0.1; };
    listen-on-v6 port 53 { ::1; };
    directory "/var/named";
    dump-file "/var/named/data/cache_dump.db";
    statistics-file "/var/named/data/named_stats.txt";
    memstatistics-file "/var/named/data/named_mem_stats.txt";
    secroots-file "/var/named/data/named.secroots";
    recursing-file "/var/named/data/named.recursing";
    allow-query { 10.0.0.0/16; localhost; };
    recursion yes;
    dnssec-validation yes;
    forwarders { 8.8.8.8; 1.1.1.1; };
    managed-keys-directory "/var/named/dynamic";
    geoip-directory "/usr/share/GeoIP";
    pid-file "/run/named/named.pid";
    session-keyfile "/run/named/session.key";
    include "/etc/crypto-policies/back-ends/bind.config";
};

logging {
    channel default_debug {
        file "data/named.run";
        severity dynamic;
    };
};

zone "ocp01.example.com" IN {
    type master;
    file "ocp01.example.com.zone";
};

zone "5.0.10.in-addr.arpa" IN {
    type master;
    file "ocp01.example.com.rev";
};
$ vi /var/named/ocp01.example.com.zone
##### 編集 #####
$ cat /var/named/ocp01.example.com.zone
$TTL 86400
@ IN SOA 005-001-rhel9jpsv.ocp01.example.com. root.ocp01.example.com. (
    2025122001 ; Serial
    3600       ; Refresh
    1800       ; Retry
    604800     ; Expire
    86400 )    ; Minimum TTL
@ IN NS 005-001-rhel9jpsv.ocp01.example.com.

; Load Balancer / Helper
005-001-rhel9jpsv IN A 10.0.5.1
api IN A 10.0.5.1
api-int IN A 10.0.5.1
*.apps IN A 10.0.5.1

; Master Nodes
005-011-ocpmstr01 IN A 10.0.5.11
005-012-ocpmstr02 IN A 10.0.5.12
005-013-ocpmstr03 IN A 10.0.5.13

; Worker Nodes
005-021-ocpwrkr01 IN A 10.0.5.21
005-022-ocpwrkr02 IN A 10.0.5.22
$ vi /var/named/ocp01.example.com.rev
##### 編集 #####
$ cat /var/named/ocp01.example.com.rev
$TTL 86400
@ IN SOA 005-001-rhel9jpsv.ocp01.example.com. root.ocp01.example.com. (
    2025122001 ; Serial
    3600       ; Refresh
    1800       ; Retry
    604800     ; Expire
    86400 )    ; Minimum TTL
@ IN NS 005-001-rhel9jpsv.ocp01.example.com.

1 IN PTR 005-001-rhel9jpsv.ocp01.example.com.
1 IN PTR api.ocp01.example.com.
1 IN PTR api-int.ocp01.example.com.
11 IN PTR 005-011-ocpmstr01.ocp01.example.com.
12 IN PTR 005-012-ocpmstr02.ocp01.example.com.
13 IN PTR 005-013-ocpmstr03.ocp01.example.com.
21 IN PTR 005-021-ocpwrkr01.ocp01.example.com.
22 IN PTR 005-022-ocpwrkr02.ocp01.example.com.
$ chown -R named:named /var/named/
$ restorecon -Rv /var/named/
$ named-checkconf /etc/named.conf
$ named-checkzone ocp01.example.com /var/named/ocp01.example.com.zone
zone ocp01.example.com/IN: loaded serial 2025122001
OK
$ named-checkzone 5.0.10.in-addr.arpa /var/named/ocp01.example.com.rev
zone 5.0.10.in-addr.arpa/IN: loaded serial 2025122001
OK
$ systemctl enable --now named
Created symlink /etc/systemd/system/multi-user.target.wants/named.service → /usr/lib/systemd/system/named.service.
$ firewall-cmd --add-service=dns --permanent
success
$ firewall-cmd --reload
success
$ dig @10.0.5.1 api.ocp01.example.com +short
10.0.5.1
$ dig @10.0.5.1 api-int.ocp01.example.com +short
10.0.5.1
$ dig @10.0.5.1 test.apps.ocp01.example.com +short
10.0.5.1
$ dig @10.0.5.1 -x 10.0.5.1 +short
005-001-rhel9jpsv.ocp01.example.com.
api-int.ocp01.example.com.
api.ocp01.example.com.
$ dig @10.0.5.1 005-011-ocpmstr01.ocp01.example.com +short
10.0.5.11
$ dig @10.0.5.1 005-012-ocpmstr02.ocp01.example.com +short
10.0.5.12
$ dig @10.0.5.1 005-013-ocpmstr03.ocp01.example.com +short
10.0.5.13
$ dig @10.0.5.1 005-021-ocpwrkr01.ocp01.example.com +short
10.0.5.21
$ dig @10.0.5.1 005-022-ocpwrkr02.ocp01.example.com +short
10.0.5.22
$ dig @10.0.5.1 -x 10.0.5.11 +short
005-011-ocpmstr01.ocp01.example.com.
$ dig @10.0.5.1 -x 10.0.5.12 +short
005-012-ocpmstr02.ocp01.example.com.
$ dig @10.0.5.1 -x 10.0.5.13 +short
005-013-ocpmstr03.ocp01.example.com.
$ dig @10.0.5.1 -x 10.0.5.21 +short
005-021-ocpwrkr01.ocp01.example.com.
$ dig @10.0.5.1 -x 10.0.5.22 +short
005-022-ocpwrkr02.ocp01.example.com.

#2. [踏み台＆DNS＆ロードバランサー]HAProxy設定

$ dnf install haproxy -y
省略
$ cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak
$ vi /etc/haproxy/haproxy.cfg
##### 編集 #####
$ cat /etc/haproxy/haproxy.cfg
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats
    ssl-default-bind-ciphers PROFILE=SYSTEM
    ssl-default-server-ciphers PROFILE=SYSTEM

defaults
    mode                tcp
    log                 global
    option              tcplog
    option              redispatch
    retries             3
    timeout connect     10s
    timeout client      1m
    timeout server      1m
    timeout check       10s
    maxconn             3000

# ==========================
# OpenShift Load Balancer (10.0.5.x)
# ==========================

# --- 6443: API Server ---
listen api-server-6443
  bind *:6443
  mode tcp
  balance roundrobin
  server master0 005-011-ocpmstr01.ocp01.example.com:6443 check inter 1s
  server master1 005-012-ocpmstr02.ocp01.example.com:6443 check inter 1s
  server master2 005-013-ocpmstr03.ocp01.example.com:6443 check inter 1s

# --- 22623: Machine Config Server ---
listen machine-config-server-22623
  bind *:22623
  mode tcp
  balance roundrobin
  server master0 005-011-ocpmstr01.ocp01.example.com:22623 check inter 1s
  server master1 005-012-ocpmstr02.ocp01.example.com:22623 check inter 1s
  server master2 005-013-ocpmstr03.ocp01.example.com:22623 check inter 1s

# --- 443: Ingress Router (HTTPS) ---
listen ingress-router-443
  bind *:443
  mode tcp
  balance source
  server worker0 005-021-ocpwrkr01.ocp01.example.com:443 check inter 1s
  server worker1 005-022-ocpwrkr02.ocp01.example.com:443 check inter 1s

# --- 80: Ingress Router (HTTP) ---
listen ingress-router-80
  bind *:80
  mode tcp
  balance source
  server worker0 005-021-ocpwrkr01.ocp01.example.com:80 check inter 1s
  server worker1 005-022-ocpwrkr02.ocp01.example.com:80 check inter 1s
$ haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid
$ setsebool -P haproxy_connect_any 1
$ systemctl enable --now haproxy
$ ss -lntp | egrep ':(6443|22623|80|443)'
LISTEN 0      3000         0.0.0.0:80         0.0.0.0:*    users:(("haproxy",pid=113679,fd=10))
LISTEN 0      3000         0.0.0.0:443        0.0.0.0:*    users:(("haproxy",pid=113679,fd=9))
LISTEN 0      3000         0.0.0.0:22623      0.0.0.0:*    users:(("haproxy",pid=113679,fd=8))
LISTEN 0      3000         0.0.0.0:6443       0.0.0.0:*    users:(("haproxy",pid=113679,fd=7))
firewall-cmd --add-service=http --permanent
success
$ firewall-cmd --add-service=https --permanent
success
$ firewall-cmd --add-port=6443/tcp --permanent
success
$ firewall-cmd --add-port=22623/tcp --permanent
success
$ firewall-cmd --reload
success
$ nc -zv 10.0.5.1 6443
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 10.0.5.1:6443.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
$ nc -zv 10.0.5.1 22623
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 10.0.5.1:22623.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
$ nc -zv 10.0.5.1 80
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 10.0.5.1:80.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
$ nc -zv 10.0.5.1 443
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 10.0.5.1:443.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
$ nc -zv api.ocp01.example.com 6443
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 10.0.5.1:6443.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.

#3. agent.x86_64.isoの生成

$ dnf install -y nmstate
$ cat << EOF > ./install-config.yaml
apiVersion: v1
baseDomain: example.com
metadata:
  name: ocp01
fips: false
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 2
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  replicas: 3
networking:
  networkType: OVNKubernetes
  machineNetwork:
  - cidr: 10.0.0.0/16
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
pullSecret: '省略'
sshKey: '省略'  
EOF
$ cat > agent-config.yaml << EOF
apiVersion: v1alpha1
kind: AgentConfig
metadata:
  name: ocp01

# Rendezvous は Master-1
rendezvousIP: 10.0.5.11

hosts:
  # -------------------- Master 1 --------------------
  - hostname: 005-011-ocpmstr01
    role: master
    interfaces:
      - name: ens33
        macAddress: 00:50:56:00:05:11
    networkConfig:
      interfaces:
        - name: ens33
          type: ethernet
          state: up
          mac-address: 00:50:56:00:05:11
          ipv4:
            enabled: true
            dhcp: false
            address:
              - ip: 10.0.5.11
                prefix-length: 16
      dns-resolver:
        config:
          server:
            - 10.0.5.1
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 10.0.1.1
            next-hop-interface: ens33
            table-id: 254

  # -------------------- Master 2 --------------------
  - hostname: 005-012-ocpmstr02
    role: master
    interfaces:
      - name: ens33
        macAddress: 00:50:56:00:05:12
    networkConfig:
      interfaces:
        - name: ens33
          type: ethernet
          state: up
          mac-address: 00:50:56:00:05:12
          ipv4:
            enabled: true
            dhcp: false
            address:
              - ip: 10.0.5.12
                prefix-length: 16
      dns-resolver:
        config:
          server:
            - 10.0.5.1
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 10.0.1.1
            next-hop-interface: ens33
            table-id: 254

  # -------------------- Master 3 --------------------
  - hostname: 005-013-ocpmstr03
    role: master
    interfaces:
      - name: ens33
        macAddress: 00:50:56:00:05:13
    networkConfig:
      interfaces:
        - name: ens33
          type: ethernet
          state: up
          mac-address: 00:50:56:00:05:13
          ipv4:
            enabled: true
            dhcp: false
            address:
              - ip: 10.0.5.13
                prefix-length: 16
      dns-resolver:
        config:
          server:
            - 10.0.5.1
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 10.0.1.1
            next-hop-interface: ens33
            table-id: 254

  # -------------------- Worker 1 --------------------
  - hostname: 005-021-ocpwrkr01
    role: worker
    interfaces:
      - name: ens33
        macAddress: 00:50:56:00:05:21
    networkConfig:
      interfaces:
        - name: ens33
          type: ethernet
          state: up
          mac-address: 00:50:56:00:05:21
          ipv4:
            enabled: true
            dhcp: false
            address:
              - ip: 10.0.5.21
                prefix-length: 16
      dns-resolver:
        config:
          server:
            - 10.0.5.1
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 10.0.1.1
            next-hop-interface: ens33
            table-id: 254

  # -------------------- Worker 2 --------------------
  - hostname: 005-022-ocpwrkr02
    role: worker
    interfaces:
      - name: ens33
        macAddress: 00:50:56:00:05:22
    networkConfig:
      interfaces:
        - name: ens33
          type: ethernet
          state: up
          mac-address: 00:50:56:00:05:22
          ipv4:
            enabled: true
            dhcp: false
            address:
              - ip: 10.0.5.22
                prefix-length: 16
      dns-resolver:
        config:
          server:
            - 10.0.5.1
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 10.0.1.1
            next-hop-interface: ens33
            table-id: 254
EOF
$ ./openshift-install agent create image
INFO Configuration has 3 master replicas, 0 arbiter replicas, and 2 worker replicas
INFO The rendezvous host IP (node0 IP) is 10.0.5.11
INFO Extracting base ISO from release payload
INFO Base ISO obtained from release and cached at [/root/.cache/agent/image_cache/coreos-x86_64.iso]
INFO Consuming Agent Config from target directory
INFO Consuming Install Config from target directory
INFO Generated ISO at agent.x86_64.iso.
$ ls /root/work/openshift/
agent.x86_64.iso  auth  openshift-install  rendezvousIP

#4. OpenShift構築（自動）

得られた[agent.x86_64.iso]をOpenShiftのマスターノード、ワーカーノードにマウントして起動すると勝手に構築が進みます。
踏み台サーバで下記のコマンドを実行するとインストールのログが出ます。

$ ./openshift-install agent wait-for install-complete --log-level=debug
1
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?