OKDとは

Red Hat OpenShiftのコミュニティ版
https://github.com/openshift/okd

環境情報

OKD: 4.10.0-0.okd-2022-03-07-13121
OS：Fedra CoreOS (FCOS) 35
ノード：OpenStack(Victoria)のインスタンス
プロキシ環境

構成図

作業ログ

インストール

以下を参照して、インストールを進める。
https://docs.okd.io/latest/installing/index.html

OKDには、以下2種類のインストール方法がある。

Installer-provisioned infrastructure (IPI)：インストーラーがサーバー等のインフラも配備する方法
User-provisioned infrastructure (UPI)：サーバー等のインフラはユーザーがあらかじめ用意する方法

更に、インフラごとにインストール手順が用意されている。
OpenStack上にOKDをインストールする手順もあるが、RHOSPが前提のため今回は使用しない。

今回はインフラを問わない以下の方法でインストールを進める。
https://docs.okd.io/latest/installing/installing_platform_agnostic/installing-platform-agnostic.html

FCOSのイメージをopenstackに作成

fcosのqcow2は以下からダウンロードする。
https://getfedora.org/ja/coreos/download?tab=cloud_operators&stream=stable&arch=x86_64

[centos@rdo ~]$ export https_proxy=172.16.183.1:63128/
[centos@rdo ~]$ curl -O https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220213.3.0/x86_64/fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  640M  100  640M    0     0  10.0M      0  0:01:03  0:01:03 --:--:-- 10.9M

ブラウザからダウンロードしたチェックサムファイルを配置する。（curlでダウンロードできなかったため）

[centos@rdo ~]$ ll fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2.xz*
-rw-rw-r-- 1 centos centos 671742324 Mar 14 14:20 fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2.xz
-rw-rw-r-- 1 centos centos       131 Mar 14 14:35 fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2.xz-CHECKSUM

署名とチェックサムを検証する。

[centos@rdo ~]$ curl -O https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220213.3.0/x86_64/fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2.xz.sig
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   566  100   566    0     0    510      0  0:00:01  0:00:01 --:--:--   510
[centos@rdo ~]$ ll fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2.xz.sig 
-rw-rw-r-- 1 centos centos 566 Mar 14 14:38 fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2.xz.sig
[centos@rdo ~]$ curl https://getfedora.org/static/fedora.gpg | gpg --import

...

gpg: /home/centos/.gnupg/trustdb.gpg: trustdb created
gpg: key F55AD3FB5323552A: public key "Fedora (37) <fedora-37-primary@fedoraproject.org>" imported
gpg: key 999F7CBF38AB71F4: public key "Fedora (36) <fedora-36-primary@fedoraproject.org>" imported
gpg: key DB4639719867C58F: public key "Fedora (35) <fedora-35-primary@fedoraproject.org>" imported
gpg: key 1161AE6945719A39: public key "Fedora (34) <fedora-34-primary@fedoraproject.org>" imported
gpg: key 49FD77499570FF31: public key "Fedora (33) <fedora-33-primary@fedoraproject.org>" imported
gpg: key 7BB90722DBBDCF7C: public key "Fedora (iot 2019) <fedora-iot-2019@fedoraproject.org>" imported
gpg: key 8A3872BF3228467C: public key "Fedora (epel9) <epel@fedoraproject.org>" imported
gpg: key 21EA45AB2F86D6A1: public key "Fedora EPEL (8) <epel@fedoraproject.org>" imported
gpg: key 6A2FAEA2352C64E5: public key "Fedora EPEL (7) <epel@fedoraproject.org>" imported
gpg: Total number processed: 9
gpg:               imported: 9
[centos@rdo ~]$ gpg --verify fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2.xz.sig fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2.xz
gpg: Signature made Tue 01 Mar 2022 12:37:37 PM JST
gpg:                using RSA key 787EA6AE1147EEE56C40B30CDB4639719867C58F
gpg: Good signature from "Fedora (35) <fedora-35-primary@fedoraproject.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 787E A6AE 1147 EEE5 6C40  B30C DB46 3971 9867 C58F
[centos@rdo ~]$ sha256sum -c fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2.xz-CHECKSUM
fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2.xz: OK
[centos@rdo ~]$ unset https_proxy

解凍して、qcow2形式のファイルにする。

[centos@rdo ~(keystone_admin)]$ unxz fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2.xz
[centos@rdo ~(keystone_admin)]$ ll fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2
-rw-rw-r-- 1 centos centos 1590951936 Mar 14 14:20 fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2

openstackにイメージを登録

[centos@rdo ~(keystone_admin)]$ openstack image create \
> --disk-format qcow2 \
> --container-format bare \
> --public \
> --file ./fedora-coreos-35.20220213.3.0-openstack.x86_64.qcow2 \
> fcos-35
+------------------+---------------------------------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                                                       |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------------+
| container_format | bare                                                                                                                                        |
| created_at       | 2022-03-18T08:32:41Z                                                                                                                        |
| disk_format      | qcow2                                                                                                                                       |
| file             | /v2/images/193df775-6184-438e-973f-cd5cd76c56b6/file                                                                                        |
| id               | 193df775-6184-438e-973f-cd5cd76c56b6                                                                                                        |
| min_disk         | 0                                                                                                                                           |
| min_ram          | 0                                                                                                                                           |
| name             | fcos-35                                                                                                                                     |
| owner            | c2b7c1d87c204b5aa1913603ae926ac8                                                                                                            |
| properties       | os_hidden='False', owner_specified.openstack.md5='', owner_specified.openstack.object='images/fcos-35', owner_specified.openstack.sha256='' |
| protected        | False                                                                                                                                       |
| schema           | /v2/schemas/image                                                                                                                           |
| status           | queued                                                                                                                                      |
| tags             |                                                                                                                                             |
| updated_at       | 2022-03-18T08:32:41Z                                                                                                                        |
| visibility       | public                                                                                                                                      |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------------+

フレーバーの作成

ドキュメントに記載の最小スペックを使用。
https://docs.okd.io/latest/installing/installing_platform_agnostic/installing-platform-agnostic.html#installation-minimum-resource-requirements_installing-platform-agnostic

[centos@rdo ~(keystone_admin)]$ openstack flavor create bootstrap_controlplane --ram 16384 --disk 100 --vcpus 4
+----------------------------+--------------------------------------+
| Field                      | Value                                |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| disk                       | 100                                  |
| id                         | 33f64211-3a83-412a-a0c6-2e70e254717c |
| name                       | bootstrap_controlplane               |
| os-flavor-access:is_public | True                                 |
| properties                 |                                      |
| ram                        | 16384                                |
| rxtx_factor                | 1.0                                  |
| swap                       |                                      |
| vcpus                      | 4                                    |
+----------------------------+--------------------------------------+
[centos@rdo ~(keystone_admin)]$ openstack flavor create compute --ram 8192 --disk 100 --vcpus 2
+----------------------------+--------------------------------------+
| Field                      | Value                                |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| disk                       | 100                                  |
| id                         | 194065c1-ea33-4766-b88b-e204796f35f4 |
| name                       | compute                              |
| os-flavor-access:is_public | True                                 |
| properties                 |                                      |
| ram                        | 8192                                 |
| rxtx_factor                | 1.0                                  |
| swap                       |                                      |
| vcpus                      | 2                                    |
+----------------------------+--------------------------------------+

セキュリティグループ

今回はdefaultのセキュリティグループに、全てのIPv4のingressを許可して使用する。

DNSの構築

以下を参考に、ubuntu 20.04のインスタンスにdnsmasqをインストールする。

ubuntu@util:~$ sudo apt -y install dnsmasq

...

Job for dnsmasq.service failed because the control process exited with error code.
See "systemctl status dnsmasq.service" and "journalctl -xe" for details.
invoke-rc.d: initscript dnsmasq, action "start" failed.
● dnsmasq.service - dnsmasq - A lightweight DHCP and caching DNS server
     Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2022-03-18 05:54:59 UTC; 5ms ago
    Process: 2236 ExecStartPre=/usr/sbin/dnsmasq --test (code=exited, status=0/SUCCESS)
    Process: 2237 ExecStart=/etc/init.d/dnsmasq systemd-exec (code=exited, status=2)

Mar 18 05:54:59 util systemd[1]: Starting dnsmasq - A lightweight DHCP and caching DNS server...
Mar 18 05:54:59 util dnsmasq[2236]: dnsmasq: syntax check OK.
Mar 18 05:54:59 util dnsmasq[2237]: dnsmasq: failed to create listening socket for port 53: Address already in use
Mar 18 05:54:59 util systemd[1]: dnsmasq.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
Mar 18 05:54:59 util dnsmasq[2237]: failed to create listening socket for port 53: Address already in use
Mar 18 05:54:59 util systemd[1]: dnsmasq.service: Failed with result 'exit-code'.
Mar 18 05:54:59 util dnsmasq[2237]: FAILED to start up
Mar 18 05:54:59 util systemd[1]: Failed to start dnsmasq - A lightweight DHCP and caching DNS server.
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Processing triggers for systemd (245.4-4ubuntu3.15) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for dbus (1.12.16-2ubuntu2.1) ...

失敗した。
53番ポートが既に使用されているようである。

ubuntu@util:~$ ss -atnu|grep :53
udp   ESTAB   0       0            127.0.0.1:36704     127.0.0.53:53            
udp   ESTAB   0       0            127.0.0.1:53269     127.0.0.53:53            
udp   UNCONN  0       0           127.0.0.53:53           0.0.0.0:*             
tcp   LISTEN  0       4096     127.0.0.53%lo:53           0.0.0.0:*

systemd-resolvedが、localhost(127.0.0.53)にスタブDNSサーバを立てるためのようである。
https://moguno.hatenablog.jp/entry/2018/06/17/155921

スタブDNSは不要なため、無効にする。

ubuntu@util:~$ sudo sed s/#DNSStubListener=yes/DNSStubListener=no/ /etc/systemd/resolved.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details

[Resolve]
#DNS=
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSOverTLS=no
#Cache=no-negative
DNSStubListener=no
#ReadEtcHosts=yes
ubuntu@util:~$ systemctl status dnsmasq.service
● dnsmasq.service - dnsmasq - A lightweight DHCP and caching DNS server
     Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-03-18 06:35:16 UTC; 23s ago
    Process: 2737 ExecStartPre=/usr/sbin/dnsmasq --test (code=exited, status=0/SUCCESS)
    Process: 2738 ExecStart=/etc/init.d/dnsmasq systemd-exec (code=exited, status=0/SUCCESS)
    Process: 2750 ExecStartPost=/etc/init.d/dnsmasq systemd-start-resolvconf (code=exited, status=0/SUCCESS)
   Main PID: 2749 (dnsmasq)
      Tasks: 1 (limit: 4726)
     Memory: 1.0M
     CGroup: /system.slice/dnsmasq.service
             └─2749 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.pid -u dnsmasq -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new --local-service --trust-anchor>

Mar 18 06:35:16 util systemd[1]: Starting dnsmasq - A lightweight DHCP and caching DNS server...
Mar 18 06:35:16 util dnsmasq[2737]: dnsmasq: syntax check OK.
Mar 18 06:35:16 util dnsmasq[2749]: started, version 2.80 cachesize 150
Mar 18 06:35:16 util dnsmasq[2749]: DNS service limited to local subnets
Mar 18 06:35:16 util dnsmasq[2749]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth nettlehash DNSSEC loop->
Mar 18 06:35:16 util dnsmasq[2749]: reading /etc/resolv.conf
Mar 18 06:35:16 util dnsmasq[2749]: using nameserver 127.0.0.53#53
Mar 18 06:35:16 util dnsmasq[2749]: read /etc/hosts - 7 addresses
Mar 18 06:35:16 util systemd[1]: Started dnsmasq - A lightweight DHCP and caching DNS server.

dnsmasqが起動した。

/etc/dnsmasq.confに以下定義を追加する。

port=53
domain-needed
bogus-priv
no-poll
address=/apps.test.example.com/10.0.0.137
expand-hosts
domain=test.example.com

/etc/hostsに以下定義を追加する。

10.0.0.137 api.test.example.com api-int.test.example.com

10.0.0.21 bootstrap.test.example.com

10.0.0.31 master0.test.example.com
10.0.0.32 master1.test.example.com
10.0.0.33 master2.test.example.com

10.0.0.41 worker0.test.example.com
10.0.0.42 worker1.test.example.com

dnsmasqを再起動して、設定を反映

ubuntu@util:~$ sudo systemctl restart dnsmasq
ubuntu@util:~$

動作確認

ubuntu@util:~$ dig @10.0.0.137 master0.test.example.com

; <<>> DiG 9.16.1-Ubuntu <<>> @10.0.0.137 master0.test.example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39444
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;master0.test.example.com.      IN      A

;; ANSWER SECTION:
master0.test.example.com. 0     IN      A       10.0.0.31

;; Query time: 0 msec
;; SERVER: 10.0.0.137#53(10.0.0.137)
;; WHEN: Thu Mar 24 05:43:18 UTC 2022
;; MSG SIZE  rcvd: 69

ubuntu@util:~$ dig @10.0.0.137 -x 10.0.0.31

; <<>> DiG 9.16.1-Ubuntu <<>> @10.0.0.137 -x 10.0.0.31
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7676
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;31.0.0.10.in-addr.arpa.                IN      PTR

;; ANSWER SECTION:
31.0.0.10.in-addr.arpa. 0       IN      PTR     master0.test.example.com.

;; Query time: 0 msec
;; SERVER: 10.0.0.137#53(10.0.0.137)
;; WHEN: Thu Mar 24 05:43:34 UTC 2022
;; MSG SIZE  rcvd: 89

ロードバランサーの構築

以下を参考に、dnsmasqをインストールしたマシンにhaproxyをインストールする。

ubuntu@util:~$ sudo apt update

...

ubuntu@util:~$ sudo apt -y install haproxy

...

/etc/haproxy/haproxy.cfgに以下定義を追加

frontend stats
    bind *:1936
    mode            http
    log             global
    maxconn 10
    stats enable
    stats hide-version
    stats refresh 30s
    stats show-node
    stats show-desc Stats for test cluster
    stats auth admin:test
    stats uri /stats

listen api-server-6443
    bind *:6443
    mode tcp
    server bootstrap bootstrap.test.example.com:6443 check inter 1s backup
    server master0   master0.test.example.com:6443   check inter 1s
    server master1   master1.test.example.com:6443   check inter 1s
    server master2   master2.test.example.com:6443   check inter 1s

listen machine-config-server-22623
    bind *:22623
    mode tcp
    server bootstrap bootstrap.test.example.com:22623 check inter 1s backup
    server master0   master0.test.example.com:22623 check inter 1s
    server master1   master1.test.example.com:22623 check inter 1s
    server master2   master2.test.example.com:22623 check inter 1s

listen ingress-router-443
    bind *:443
    mode tcp
    balance source
    server worker0 worker0.test.example.com:443 check inter 1s
    server worker1 worker1.test.example.com:443 check inter 1s

listen ingress-router-80
    bind *:80
    mode tcp
    balance source
    server worker0 worker0.test.example.com:80 check inter 1s
    server worker1 worker1.test.example.com:80 check inter 1s

設定を反映させるためにhaproxyを再起動

ubuntu@util:~$ sudo systemctl restart haproxy.service 
ubuntu@util:~$

OKDインストールモジュールのダウンロード

ubuntu@util:~$ https_proxy=172.16.184.1:63128/ curl -LO https://github.com/openshift/okd/releases/download/4.10.0-0.okd-2022-03-07-131213/openshift-install-linux-4.10.0-0.okd-2022-03-07-131213.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   700  100   700    0     0   2811      0 --:--:-- --:--:-- --:--:--  2811
100  124M  100  124M    0     0  8355k      0  0:00:15  0:00:15 --:--:-- 8069k
ubuntu@util:~$ https_proxy=172.16.184.1:63128/ curl -LO https://github.com/openshift/okd/releases/download/4.10.0-0.okd-2022-03-07-131213/sha256sum.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   652  100   652    0     0   2608      0 --:--:-- --:--:-- --:--:--  2608
100   710  100   710    0     0   1085      0 --:--:-- --:--:-- --:--:--  3169
ubuntu@util:~$ sha256sum -c --ignore-missing sha256sum.txt
openshift-install-linux-4.10.0-0.okd-2022-03-07-131213.tar.gz: OK
ubuntu@util:~$ tar -xvf openshift-install-linux-4.10.0-0.okd-2022-03-07-131213.tar.gz 
README.md
openshift-install
ubuntu@util:~$ ls
README.md  openshift-install  openshift-install-linux-4.10.0-0.okd-2022-03-07-131213.tar.gz  sha256sum.txt

OCコマンドのインストール

ubuntu@util:~$ https_proxy=172.16.184.1:63128/ curl -O https://mirror.openshift.com/pub/openshift-v4/clients/oc/latest/linux/oc.tar.gz

...

ubuntu@util:~$ tar xvzf oc.tar.gz 
README.md
oc
kubectl
ubuntu@util:~$ sudo mv kubectl /usr/local/bin/
ubuntu@util:~$ sudo mv oc /usr/local/bin/
ubuntu@util:~$ oc version
Client Version: 4.8.11

インストール用コンフィグファイルの作成

ubuntu@util:~$ mkdir okd

install-config.yamlの作成

apiVersion: v1
baseDomain: example.com
proxy:
  httpProxy: http://172.16.184.1:63128/
  httpsProxy: http://172.16.184.1:63128/
  noProxy: example.com
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: test
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
pullSecret: '{"auths":{"fake":{"auth":"aWQ6cGFzcwo="}}}'
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAA...'

UPIの場合、computeのreplicasは0に指定する必要がある。

マニフェストの作成

ubuntu@util:~$ ./openshift-install create manifests --dir okd
INFO Consuming Install Config from target directory 
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings 
INFO Manifests created in: okd/manifests and okd/openshift 
ubuntu@util:~$ tree okd
okd
├── manifests
│   ├── cluster-config.yaml
│   ├── cluster-dns-02-config.yml
│   ├── cluster-infrastructure-02-config.yml
│   ├── cluster-ingress-02-config.yml
│   ├── cluster-network-01-crd.yml
│   ├── cluster-network-02-config.yml
│   ├── cluster-proxy-01-config.yaml
│   ├── cluster-scheduler-02-config.yml
│   ├── cvo-overrides.yaml
│   ├── kube-cloud-config.yaml
│   ├── kube-system-configmap-root-ca.yaml
│   ├── machine-config-server-tls-secret.yaml
│   └── openshift-config-secret-pull-secret.yaml
└── openshift
    ├── 99_kubeadmin-password-secret.yaml
    ├── 99_openshift-cluster-api_master-user-data-secret.yaml
    ├── 99_openshift-cluster-api_worker-user-data-secret.yaml
    ├── 99_openshift-machineconfig_99-master-ssh.yaml
    ├── 99_openshift-machineconfig_99-worker-ssh.yaml
    └── openshift-install-manifests.yaml

2 directories, 19 files

mastersSchedulableの確認

ubuntu@util:~$ cat okd/manifests/cluster-scheduler-02-config.yml 
apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
  creationTimestamp: null
  name: cluster
spec:
  mastersSchedulable: true
  policy:
    name: ""
status: {}

trueが指定されているので、falseに変更する。

ignitionファイルを作成

ubuntu@util:~$ ./openshift-install create ignition-configs --dir okd
INFO Consuming Master Machines from target directory 
INFO Consuming Common Manifests from target directory 
INFO Consuming OpenShift Install (Manifests) from target directory 
INFO Consuming Openshift Manifests from target directory 
INFO Consuming Worker Machines from target directory 
INFO Ignition-Configs created in: okd and okd/auth 
ubuntu@util:~$ tree okd/
okd/
├── auth
│   ├── kubeadmin-password
│   └── kubeconfig
├── bootstrap.ign
├── master.ign
├── metadata.json
└── worker.ign

1 directory, 6 files

FCOSのインストール

openstackでFCOSのインスタンスををインストールすると、ホスト名にはインスタンス名ではなくIPアドレスが設定される。
そのため、ignitionでホスト名を設定する。

参考：https://coreos.github.io/ignition/examples/#set-the-hostname

以下の定義をignitionに追加する。

{
  "storage": {
    "files": [{
      "path": "/etc/hostname",
      "mode": 420,
      "overwrite": true,
      "contents": { "source": "data:,bootstrap.test.example.com" }
    }]
  }
}

ネットワークのポート作成

あらかじめサブネットのDNSにdnsmasqを指定しておく必要がある。

[centos@rdo ~(keystone_admin)]$ openstack port create --network 4507ad6f-361d-4b4e-bc20-2327dca81756 --fixed-ip subnet=e87d8830-c075-4359-9907-182d7d17fadd,ip-address=10.0.0.21 bootstrap
+-------------------------+--------------------------------------------------------------------------+
| Field                   | Value                                                                    |
+-------------------------+--------------------------------------------------------------------------+
| admin_state_up          | UP                                                                       |
| allowed_address_pairs   |                                                                          |
| binding_host_id         |                                                                          |
| binding_profile         |                                                                          |
| binding_vif_details     |                                                                          |
| binding_vif_type        | unbound                                                                  |
| binding_vnic_type       | normal                                                                   |
| created_at              | 2022-03-23T04:42:50Z                                                     |
| data_plane_status       | None                                                                     |
| description             |                                                                          |
| device_id               |                                                                          |
| device_owner            |                                                                          |
| dns_assignment          | None                                                                     |
| dns_domain              | None                                                                     |
| dns_name                | None                                                                     |
| extra_dhcp_opts         |                                                                          |
| fixed_ips               | ip_address='10.0.0.21', subnet_id='e87d8830-c075-4359-9907-182d7d17fadd' |
| id                      | 61fed2e0-1e82-46ab-a164-1bd1e0390aa6                                     |
| ip_allocation           | None                                                                     |
| mac_address             | fa:16:3e:08:34:f8                                                        |
| name                    | bootstrap                                                                |
| network_id              | 4507ad6f-361d-4b4e-bc20-2327dca81756                                     |
| numa_affinity_policy    | None                                                                     |
| port_security_enabled   | True                                                                     |
| project_id              | c2b7c1d87c204b5aa1913603ae926ac8                                         |
| propagate_uplink_status | None                                                                     |
| qos_network_policy_id   | None                                                                     |
| qos_policy_id           | None                                                                     |
| resource_request        | None                                                                     |
| revision_number         | 1                                                                        |
| security_group_ids      | 4f975a13-0638-45ce-ad80-213be2ce275d                                     |
| status                  | DOWN                                                                     |
| tags                    |                                                                          |
| trunk_details           | None                                                                     |
| updated_at              | 2022-03-23T04:42:50Z                                                     |
+-------------------------+--------------------------------------------------------------------------+

以下を参考にインスタンスを作成
https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-openstack/#_launching_a_vm_instance

[centos@rdo ~(keystone_admin)]$ openstack server create \
> --port=61fed2e0-1e82-46ab-a164-1bd1e0390aa6 \
> --flavor=33f64211-3a83-412a-a0c6-2e70e254717c \
> --image=193df775-6184-438e-973f-cd5cd76c56b6 \
> --security-group 4f975a13-0638-45ce-ad80-213be2ce275d \
> --user-data ./ignition/bootstrap.ign \
> bootstrap.test.example.com

Request is too large. Larger than max_request_body_size (114688).<br /><br />


 (HTTP 413) (Request-ID: req-94be941b-e04c-41fd-8421-6bfe27255dc9)

ignitionファイルのサイズが大き過ぎて失敗した。
novaのパラメータチューニングが必要そう。

ignitionファイルのサイズを確認

[centos@rdo ~(keystone_admin)]$ du ignition/bootstrap.ign -h
264K    ignition/bootstrap.ign

nova.confを確認すると、以下のサイズがデフォルトのようである。

# The maximum body size for each  request, in bytes (integer value)
# Deprecated group;name - DEFAULT;osapi_max_request_body_size
# Deprecated group;name - DEFAULT;max_request_body_size
#max_request_body_size=114688

以下のように変更する。

max_request_body_size=1000000

設定変更の反映

[centos@rdo ~(keystone_admin)]$ sudo systemctl restart httpd

インスタンス作成を再実施したところ、413のエラーは解消したようだが、今度は400のエラーが発生。

Invalid input for field/attribute user_data. Value: eyJp ... dfX0K' is too long (HTTP 400) (Request-ID: req-13954903-17bc-4da0-b16c-3a8f8ecd47b9)

以下によると、openstackのuser dataの上限は65,535B（base64エンコード後）とのこと。
https://docs.openstack.org/nova/latest/user/metadata.html#user-data

user dataのサイズ上限を上げることはできなそう。
ISOからのインストールであれば、ignitionをwebサーバー経由で渡せることができるため、bootstrapノードだけはISOからインストールする。

まずはdnsmasqとhaproxyをインストールしたマシンにnginxをインストールして、bootstrap nodeのignitionファイルを公開する。

ubuntu@util:~$ sudo apt install nginx

...

ubuntu@util:~$ systemctl status nginx.service 
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2022-03-24 03:05:39 UTC; 1h 18min ago
       Docs: man:nginx(8)
    Process: 35946 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 35947 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)

Mar 24 03:05:36 util systemd[1]: Starting A high performance web server and a reverse proxy server...
Mar 24 03:05:36 util nginx[35947]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Mar 24 03:05:37 util nginx[35947]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Mar 24 03:05:37 util nginx[35947]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Mar 24 03:05:38 util nginx[35947]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Mar 24 03:05:38 util nginx[35947]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Mar 24 03:05:39 util nginx[35947]: nginx: [emerg] still could not bind()
Mar 24 03:05:39 util systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Mar 24 03:05:39 util systemd[1]: nginx.service: Failed with result 'exit-code'.
Mar 24 03:05:39 util systemd[1]: Failed to start A high performance web server and a reverse proxy server.
ubuntu@util:~$ sudo lsof -i :80
COMMAND   PID    USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
haproxy 34102 haproxy   12u  IPv4 1057064      0t0  TCP *:http (LISTEN)

ポート80番はhaproxyに使われているので、別のポートを使用する。
/etc/nginx/sites-enabled/defaultを以下のように修正

server {
        #listen 80 default_server;
        #listen [::]:80 default_server;
        listen 60080 default_server;
        listen [::]:60080 default_server;

ubuntu@util:~$ sudo systemctl start nginx.service 
ubuntu@util:~$ systemctl status nginx.service 
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-03-24 04:28:26 UTC; 5s ago

...

ubuntu@util:~$ curl http://127.0.0.1:60080/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

/var/www/html/に公開するignitionファイルを配置する。

ubuntu@util:~$ sudo cp okd/bootstrap.ign /var/www/html/
ubuntu@util:~$ curl 127.0.0.1:60080/bootstrap.ign
ubuntu@util:~$ curl 127.0.0.1:60080/bootstrap.ign
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
ubuntu@util:~$ sudo ls -l /var/www/html/bootstrap.ign 
-rw-r----- 1 root root 268862 Mar 24 04:34 /var/www/html/bootstrap.ign
ubuntu@util:~$ sudo chmod +r /var/www/html/bootstrap.ign

...

次に、FCOSのisoをopenstackに登録する。

isoのダウンロード

[centos@rdo ~]$ https_proxy=http://172.16.183.1:63128/ curl -O https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220227.3.0/x86_64/fedor
a-coreos-35.20220227.3.0-live.x86_64.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  772M  100  772M    0     0  9691k      0  0:01:21  0:01:21 --:--:-- 8166k

isoの検証（チェックサムとシグネイチャをダウンロードしておく）

[centos@rdo ~]$ gpg --verify fedora-coreos-35.20220227.3.0-live.x86_64.iso.sig fedora-coreos-35.20220227.3.0-live.x86_64.iso
gpg: Signature made Tue 15 Mar 2022 06:01:18 AM JST
gpg:                using RSA key 787EA6AE1147EEE56C40B30CDB4639719867C58F
gpg: Good signature from "Fedora (35) <fedora-35-primary@fedoraproject.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 787E A6AE 1147 EEE5 6C40  B30C DB46 3971 9867 C58F
[centos@rdo ~]$ sha256sum -c fedora-coreos-35.20220227.3.0-live.x86_64.iso-CHECKSUM
fedora-coreos-35.20220227.3.0-live.x86_64.iso: OK

openstackにイメージを登録

[centos@rdo ~(keystone_admin)]$ openstack image create \
> --disk-format iso \
> --container-format bare \
> --public \
> --file ./fedora-coreos-35.20220227.3.0-live.x86_64.iso \
> fcos-35-iso
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                                                           |
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| container_format | bare                                                                                                                                            |
| created_at       | 2022-03-24T02:52:35Z                                                                                                                            |
| disk_format      | iso                                                                                                                                             |
| file             | /v2/images/1410641e-92b9-465d-8ef2-2261a145c18e/file                                                                                            |
| id               | 1410641e-92b9-465d-8ef2-2261a145c18e                                                                                                            |
| min_disk         | 0                                                                                                                                               |
| min_ram          | 0                                                                                                                                               |
| name             | fcos-35-iso                                                                                                                                     |
| owner            | c2b7c1d87c204b5aa1913603ae926ac8                                                                                                                |
| properties       | os_hidden='False', owner_specified.openstack.md5='', owner_specified.openstack.object='images/fcos-35-iso', owner_specified.openstack.sha256='' |
| protected        | False                                                                                                                                           |
| schema           | /v2/schemas/image                                                                                                                               |
| status           | queued                                                                                                                                          |
| tags             |                                                                                                                                                 |
| updated_at       | 2022-03-24T02:52:35Z                                                                                                                            |
| visibility       | public                                                                                                                                          |
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+

以下を参考に、isoイメージからインスタンスを作成する。
https://docs.openstack.org/ja/user-guide/cli-nova-launch-instance-using-ISO-image.html

[centos@rdo ~(keystone_admin)]$ openstack server create \
> --network=openstack_net \
> --flavor=33f64211-3a83-412a-a0c6-2e70e254717c \
> --image=fcos-35-iso \
> --security-group 4f975a13-0638-45ce-ad80-213be2ce275d \
> bootstrap-temp
+-------------------------------------+---------------------------------------------------------------+
| Field                               | Value                                                         |
+-------------------------------------+---------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                        |
| OS-EXT-AZ:availability_zone         |                                                               |
| OS-EXT-SRV-ATTR:host                | None                                                          |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                                          |
| OS-EXT-SRV-ATTR:instance_name       |                                                               |
| OS-EXT-STS:power_state              | NOSTATE                                                       |
| OS-EXT-STS:task_state               | scheduling                                                    |
| OS-EXT-STS:vm_state                 | building                                                      |
| OS-SRV-USG:launched_at              | None                                                          |
| OS-SRV-USG:terminated_at            | None                                                          |
| accessIPv4                          |                                                               |
| accessIPv6                          |                                                               |
| addresses                           |                                                               |
| adminPass                           | YS4nYfbgVug7                                                  |
| config_drive                        |                                                               |
| created                             | 2022-03-24T06:23:12Z                                          |
| flavor                              | bootstrap_controlplane (33f64211-3a83-412a-a0c6-2e70e254717c) |
| hostId                              |                                                               |
| id                                  | ae8e2a3b-ded8-4987-b77d-3595a46ff9d2                          |
| image                               | fcos-35-iso (1410641e-92b9-465d-8ef2-2261a145c18e)            |
| key_name                            | None                                                          |
| name                                | bootstrap-temp                                                |
| progress                            | 0                                                             |
| project_id                          | c2b7c1d87c204b5aa1913603ae926ac8                              |
| properties                          |                                                               |
| security_groups                     | name='4f975a13-0638-45ce-ad80-213be2ce275d'                   |
| status                              | BUILD                                                         |
| updated                             | 2022-03-24T06:23:12Z                                          |
| user_id                             | 973053e4bb1f4bae95f1ad4264f73487                              |
| volumes_attached                    |                                                               |
+-------------------------------------+---------------------------------------------------------------+
[centos@rdo ~(keystone_admin)]$ openstack volume create \
>   --size 20 \
>   --bootable bootstrap-temp
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2022-03-24T06:23:54.000000           |
| description         | None                                 |
| encrypted           | False                                |
| id                  | 77b92d9a-82a9-4f58-a6ea-2bac028d8b33 |
| migration_status    | None                                 |
| multiattach         | False                                |
| name                | bootstrap-temp                       |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 20                                   |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | iscsi                                |
| updated_at          | None                                 |
| user_id             | 973053e4bb1f4bae95f1ad4264f73487     |
+---------------------+--------------------------------------+
[centos@rdo ~(keystone_admin)]$ openstack server add volume \
> bootstrap-temp \
> bootstrap-temp \
> --device /dev/vda

horizonのコンソールからfcosのインストールを進める。

[centos@rdo ~(keystone_admin)]$ openstack server delete bootstrap-temp
[centos@rdo ~(keystone_admin)]$ openstack image create --volume bootstrap-temp bootstrap
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| container_format    | bare                                 |
| disk_format         | raw                                  |
| display_description | None                                 |
| id                  | 77b92d9a-82a9-4f58-a6ea-2bac028d8b33 |
| image_id            | 6ef09886-88e3-41be-bdf2-c8a8858a7e72 |
| image_name          | bootstrap                            |
| protected           | False                                |
| size                | 20                                   |
| status              | uploading                            |
| updated_at          | 2022-03-24T06:29:12.000000           |
| visibility          | shared                               |
| volume_type         | iscsi                                |
+---------------------+--------------------------------------+
[centos@rdo ~(keystone_admin)]$ openstack server create \
> --port=61fed2e0-1e82-46ab-a164-1bd1e0390aa6 \
> --flavor=33f64211-3a83-412a-a0c6-2e70e254717c \
> --image=bootstrap \
> --security-group 4f975a13-0638-45ce-ad80-213be2ce275d \
> bootstrap.test.example.com
+-------------------------------------+---------------------------------------------------------------+
| Field                               | Value                                                         |
+-------------------------------------+---------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                        |
| OS-EXT-AZ:availability_zone         |                                                               |
| OS-EXT-SRV-ATTR:host                | None                                                          |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                                          |
| OS-EXT-SRV-ATTR:instance_name       |                                                               |
| OS-EXT-STS:power_state              | NOSTATE                                                       |
| OS-EXT-STS:task_state               | scheduling                                                    |
| OS-EXT-STS:vm_state                 | building                                                      |
| OS-SRV-USG:launched_at              | None                                                          |
| OS-SRV-USG:terminated_at            | None                                                          |
| accessIPv4                          |                                                               |
| accessIPv6                          |                                                               |
| addresses                           |                                                               |
| adminPass                           | 7gRMHjrnVoWp                                                  |
| config_drive                        |                                                               |
| created                             | 2022-03-24T06:31:00Z                                          |
| flavor                              | bootstrap_controlplane (33f64211-3a83-412a-a0c6-2e70e254717c) |
| hostId                              |                                                               |
| id                                  | 45bc184f-b759-498b-91ae-22cbad006774                          |
| image                               | bootstrap (6ef09886-88e3-41be-bdf2-c8a8858a7e72)              |
| key_name                            | None                                                          |
| name                                | bootstrap.test.example.com                                    |
| progress                            | 0                                                             |
| project_id                          | c2b7c1d87c204b5aa1913603ae926ac8                              |
| properties                          |                                                               |
| security_groups                     | name='4f975a13-0638-45ce-ad80-213be2ce275d'                   |
| status                              | BUILD                                                         |
| updated                             | 2022-03-24T06:31:00Z                                          |
| user_id                             | 973053e4bb1f4bae95f1ad4264f73487                              |
| volumes_attached                    |                                                               |
+-------------------------------------+---------------------------------------------------------------+

controlplaneの3ノードはqcow2のfcosから作成する。

[centos@rdo ~(keystone_admin)]$ openstack server create \
> --port=2068243d-0039-4cd3-9663-6a869f62ccef \
> --flavor=33f64211-3a83-412a-a0c6-2e70e254717c \
> --image=193df775-6184-438e-973f-cd5cd76c56b6 \
> --security-group 4f975a13-0638-45ce-ad80-213be2ce275d \
> --user-data ./ignition/master0.ign \
> master0.test.example.com
+-------------------------------------+---------------------------------------------------------------+
| Field                               | Value                                                         |
+-------------------------------------+---------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                        |
| OS-EXT-AZ:availability_zone         |                                                               |
| OS-EXT-SRV-ATTR:host                | None                                                          |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                                          |
| OS-EXT-SRV-ATTR:instance_name       |                                                               |
| OS-EXT-STS:power_state              | NOSTATE                                                       |
| OS-EXT-STS:task_state               | scheduling                                                    |
| OS-EXT-STS:vm_state                 | building                                                      |
| OS-SRV-USG:launched_at              | None                                                          |
| OS-SRV-USG:terminated_at            | None                                                          |
| accessIPv4                          |                                                               |
| accessIPv6                          |                                                               |
| addresses                           |                                                               |
| adminPass                           | B8ung9mr6CoZ                                                  |
| config_drive                        |                                                               |
| created                             | 2022-03-24T06:31:34Z                                          |
| flavor                              | bootstrap_controlplane (33f64211-3a83-412a-a0c6-2e70e254717c) |
| hostId                              |                                                               |
| id                                  | 7fbd80e0-deec-4330-a394-6b7e0569f5f8                          |
| image                               | fcos-35 (193df775-6184-438e-973f-cd5cd76c56b6)                |
| key_name                            | None                                                          |
| name                                | master0.test.example.com                                      |
| progress                            | 0                                                             |
| project_id                          | c2b7c1d87c204b5aa1913603ae926ac8                              |
| properties                          |                                                               |
| security_groups                     | name='4f975a13-0638-45ce-ad80-213be2ce275d'                   |
| status                              | BUILD                                                         |
| updated                             | 2022-03-24T06:31:34Z                                          |
| user_id                             | 973053e4bb1f4bae95f1ad4264f73487                              |
| volumes_attached                    |                                                               |
+-------------------------------------+---------------------------------------------------------------+

...

ブートストラッププロセスの完了を待つ

ubuntu@util:~$ ./openshift-install --dir okd wait-for bootstrap-complete --log-level=info
INFO Waiting up to 20m0s (until 6:52AM) for the Kubernetes API at https://api.test.example.com:6443... 
INFO API v1.23.3-2003+e419edff267ffa-dirty up     
INFO Waiting up to 30m0s (until 7:08AM) for bootstrapping to complete... 
INFO It is now safe to remove the bootstrap resources 
INFO Time elapsed: 30m2s

完了したようである。

controlplne nodeの確認

ubuntu@util:~$ echo "export KUBECONFIG=~/okd/auth/kubeconfig" >> ~/.bashrc 
ubuntu@util:~$ source ~/.bashrc
ubuntu@util:~$ oc whoami
system:admin

ubuntu@util:~$ oc get csr
NAME                                             AGE   SIGNERNAME                                    REQUESTOR                                                                         REQUESTEDDURATION   CONDITION
csr-f42fn                                        24m   kubernetes.io/kubelet-serving                 system:node:master1.test.example.com                                              <none>              Approved,Issued
csr-k87lm                                        24m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
csr-lvnl9                                        23m   kubernetes.io/kubelet-serving                 system:node:master0.test.example.com                                              <none>              Approved,Issued
csr-lwgzc                                        24m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
csr-mfzn7                                        23m   kubernetes.io/kubelet-serving                 system:node:master2.test.example.com                                              <none>              Approved,Issued
csr-r9hhv                                        25m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
csr-rnm7k                                        24m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
csr-tqj6v                                        23m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
csr-w7f8f                                        23m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
system:openshift:openshift-authenticator-h2bpt   13m   kubernetes.io/kube-apiserver-client           system:serviceaccount:openshift-authentication-operator:authentication-operator   <none>              Approved,Issued
system:openshift:openshift-monitoring-zh4dr      12m   kubernetes.io/kube-apiserver-client           system:serviceaccount:openshift-monitoring:cluster-monitoring-operator            <none>              Approved,Issued

全てのCSRが承認されている。

worker nodeの追加

作成したignitionには、ssh公開鍵の設定が入っていないため、以下のような定義を追加しておく。

{
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD..."
        ]
      }
    ]
  },
}

[centos@rdo ~(keystone_admin)]$ openstack port create --network 4507ad6f-361d-4b4e-bc20-2327dca81756 --fixed-ip subnet=e87d8830-c075-4359-9907-182d7d17fadd,ip-address=10.0.0.41 worker0
+-------------------------+--------------------------------------------------------------------------+
| Field                   | Value                                                                    |
+-------------------------+--------------------------------------------------------------------------+
| admin_state_up          | UP                                                                       |
| allowed_address_pairs   |                                                                          |
| binding_host_id         |                                                                          |
| binding_profile         |                                                                          |
| binding_vif_details     |                                                                          |
| binding_vif_type        | unbound                                                                  |
| binding_vnic_type       | normal                                                                   |
| created_at              | 2022-03-24T07:23:03Z                                                     |
| data_plane_status       | None                                                                     |
| description             |                                                                          |
| device_id               |                                                                          |
| device_owner            |                                                                          |
| dns_assignment          | None                                                                     |
| dns_domain              | None                                                                     |
| dns_name                | None                                                                     |
| extra_dhcp_opts         |                                                                          |
| fixed_ips               | ip_address='10.0.0.41', subnet_id='e87d8830-c075-4359-9907-182d7d17fadd' |
| id                      | 525f121f-3673-405e-bca5-3f120eede580                                     |
| ip_allocation           | None                                                                     |
| mac_address             | fa:16:3e:d3:60:6f                                                        |
| name                    | worker0                                                                  |
| network_id              | 4507ad6f-361d-4b4e-bc20-2327dca81756                                     |
| numa_affinity_policy    | None                                                                     |
| port_security_enabled   | True                                                                     |
| project_id              | c2b7c1d87c204b5aa1913603ae926ac8                                         |
| propagate_uplink_status | None                                                                     |
| qos_network_policy_id   | None                                                                     |
| qos_policy_id           | None                                                                     |
| resource_request        | None                                                                     |
| revision_number         | 1                                                                        |
| security_group_ids      | 4f975a13-0638-45ce-ad80-213be2ce275d                                     |
| status                  | DOWN                                                                     |
| tags                    |                                                                          |
| trunk_details           | None                                                                     |
| updated_at              | 2022-03-24T07:23:03Z                                                     |
+-------------------------+--------------------------------------------------------------------------+
[centos@rdo ~(keystone_admin)]$ openstack server create \
> --port=525f121f-3673-405e-bca5-3f120eede580 \
> --flavor=compute \
> --image=193df775-6184-438e-973f-cd5cd76c56b6 \
> --security-group 4f975a13-0638-45ce-ad80-213be2ce275d \
> --user-data ./ignition/worker0.ign \
> worker0.test.example.com
+-------------------------------------+------------------------------------------------+
| Field                               | Value                                          |
+-------------------------------------+------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                         |
| OS-EXT-AZ:availability_zone         |                                                |
| OS-EXT-SRV-ATTR:host                | None                                           |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                           |
| OS-EXT-SRV-ATTR:instance_name       |                                                |
| OS-EXT-STS:power_state              | NOSTATE                                        |
| OS-EXT-STS:task_state               | scheduling                                     |
| OS-EXT-STS:vm_state                 | building                                       |
| OS-SRV-USG:launched_at              | None                                           |
| OS-SRV-USG:terminated_at            | None                                           |
| accessIPv4                          |                                                |
| accessIPv6                          |                                                |
| addresses                           |                                                |
| adminPass                           | BZJYr7YPt787                                   |
| config_drive                        |                                                |
| created                             | 2022-03-24T07:23:45Z                           |
| flavor                              | compute (194065c1-ea33-4766-b88b-e204796f35f4) |
| hostId                              |                                                |
| id                                  | 5e8f6fbc-34d3-45ce-a88a-42fe926ee006           |
| image                               | fcos-35 (193df775-6184-438e-973f-cd5cd76c56b6) |
| key_name                            | None                                           |
| name                                | worker0.test.example.com                       |
| progress                            | 0                                              |
| project_id                          | c2b7c1d87c204b5aa1913603ae926ac8               |
| properties                          |                                                |
| security_groups                     | name='4f975a13-0638-45ce-ad80-213be2ce275d'    |
| status                              | BUILD                                          |
| updated                             | 2022-03-24T07:23:46Z                           |
| user_id                             | 973053e4bb1f4bae95f1ad4264f73487               |
| volumes_attached                    |                                                |
+-------------------------------------+------------------------------------------------+

しばらくするとpendingのCSRが作成された。

ubuntu@util:~$ oc get csr
NAME                                             AGE   SIGNERNAME                                    REQUESTOR                                                                         REQUESTEDDURATION   CONDITION
csr-7v7xj                                        11s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Pending

...

ubuntu@util:~$ oc get node
NAME                       STATUS   ROLES    AGE   VERSION
master0.test.example.com   Ready    master   43m   v1.23.3+759c22b
master1.test.example.com   Ready    master   44m   v1.23.3+759c22b
master2.test.example.com   Ready    master   43m   v1.23.3+759c22b
ubuntu@util:~$ oc adm certificate approve csr-7v7xj
certificatesigningrequest.certificates.k8s.io/csr-7v7xj approved
ubuntu@util:~$ oc get csr csr-7v7xj
NAME        AGE    SIGNERNAME                                    REQUESTOR                                                                   REQUESTEDDURATION   CONDITION
csr-7v7xj   116s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   <none>              Approved,Issued
ubuntu@util:~$ oc get node
NAME                       STATUS     ROLES    AGE   VERSION
master0.test.example.com   Ready      master   44m   v1.23.3+759c22b
master1.test.example.com   Ready      master   45m   v1.23.3+759c22b
master2.test.example.com   Ready      master   44m   v1.23.3+759c22b
worker0.test.example.com   NotReady   worker   5s    v1.23.3+759c22b

...

ubuntu@util:~$ oc get node
NAME                       STATUS   ROLES    AGE     VERSION
master0.test.example.com   Ready    master   50m     v1.23.3+759c22b
master1.test.example.com   Ready    master   51m     v1.23.3+759c22b
master2.test.example.com   Ready    master   50m     v1.23.3+759c22b
worker0.test.example.com   Ready    worker   5m22s   v1.23.3+759c22b

同じようにworker nodeをもう一台追加。

ubuntu@util:~$ oc get node
NAME                       STATUS   ROLES    AGE     VERSION
master0.test.example.com   Ready    master   7d18h   v1.23.3+759c22b
master1.test.example.com   Ready    master   7d18h   v1.23.3+759c22b
master2.test.example.com   Ready    master   7d18h   v1.23.3+759c22b
worker0.test.example.com   Ready    worker   7d17h   v1.23.3+759c22b
worker1.test.example.com   Ready    worker   4m25s   v1.23.3+759c22b

webコンソールへのアクセス

URLの確認

ubuntu@util:~$ oc whoami --show-console
https://console-openshift-console.apps.test.example.com

ブラウザを実行するマシンに以下のhostsを設定

172.16.184.197 console-openshift-console.apps.test.example.com

172.16.184.197はhaproxyをインストールしたマシンに付与したfloating IP

ブラウザからアクセスすると、以下にリダイレクトされた。

https://oauth-openshift.apps.test.example.com/oauth/authorize?client_id=console&redirect_uri=https%3A%2F%2Fconsole-openshift-console.apps.test.example.com%2Fauth%2Fcallback&response_type=code&scope=user%3Afull&state=4e6e8409

以下をhostsに追加して再トライ。

172.16.184.197 oauth-openshift.apps.test.example.com

以下のようにログイン画面が開いた。

初期ユーザーとしてkubeadminが使えるようだが、パスワードがわからない。
インストールディレクトリのauth/kubeadmin-passwordに記載された文字列を入れたらログインできた！

アプリのデプロイ

以下を参考にアプリをデプロイする。
https://docs.okd.io/latest/applications/creating_applications/creating-applications-using-cli.html#applications-create-using-cli-image_creating-applications-using-cli

ubuntu@util:~$ oc new-app docker.io/nginx
--> Found container image 12766a6 (2 days old) from docker.io for "docker.io/nginx"

    * An image stream tag will be created as "nginx:latest" that will track this image

--> Creating resources ...
    imagestream.image.openshift.io "nginx" created
    deployment.apps "nginx" created
    service "nginx" created
--> Success
    WARNING: No container image registry has been configured with the server. Automatic builds and deployments may not function.
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose service/nginx' 
    Run 'oc status' to view your app.
ubuntu@util:~$ oc get pod
NAME                     READY   STATUS    RESTARTS   AGE
nginx-6778f7495f-qwn8f   1/1     Running   0          34s

クラスタ外にserviceを公開する。

ubuntu@util:~$ oc expose service nginx 
route.route.openshift.io/nginx exposed
ubuntu@util:~$ oc get routes.route.openshift.io 
NAME    HOST/PORT                             PATH   SERVICES   PORT     TERMINATION   WILDCARD
nginx   nginx-default.apps.test.example.com          nginx      80-tcp                 None
ubuntu@util:~$ curl http://nginx-default.apps.test.example.com
curl: (6) Could not resolve host: nginx-default.apps.test.example.com
ubuntu@util:~$ curl http://nginx-default.apps.test.example.com --resolv nginx-default.apps.test.example.com:80:127.0.0.1
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

クラスタ外からopenshift上にデプロイしたnginxにアクセスすることができた！

本記事の内容は以上です。