bootstrapのVMを起動した後、masterのVMとworkerのVMを順次起動させて、OpenShiftを構築します。
1. bootstrapの起動
bootstrapのVMを起動します。
正常に起動すると、 OpenShiftのVersion 4.4とipアドレス:172.16.0.100が表示されます。
SELinux : mount invalid
放置すると、このように大量のSELinuxのメッセージが表示されますが無視して大丈夫です。
2. クラスターの作成
2.1 bootstrapにsshログイン
踏み台(bastion)からbootsrapにログインします。
sshのidentityファイルは秘密鍵のnew_rsaを使用します。
(公開鍵のnew_rsa.pubは使用しません.)
[root@bastion ~]# ssh -i ~/.ssh/new_rsa core@bootstrap
Red Hat Enterprise Linux CoreOS 44.82.202008011133-0
Part of OpenShift 4.4, RHCOS is a Kubernetes native operating system
managed by the Machine Config Operator (`clusteroperator/machine-config`).
WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
https://docs.openshift.com/container-platform/4.4/architecture/architecture-rhcos.html
---
This is the bootstrap node; it will be destroyed when the master is fully up.
The primary service is "bootkube.service". To watch its status, run e.g.
journalctl -b -f -u bootkube.service
2.2 クラスター作成状況
ssh ログイン後のメッセージに書かれているコマンドをそのまま実行します。
(監視したまま、3.master起動、4.worker起動を実施してください。)
[core@bootstrap ~]$ journalctl -b -f -u bootkube.service
3. master起動
master-0、master-1、master-2を順次起動していきます。
-
master-0
-
master-1
-
master-2
起動時のconnection refused
以下の画面のように起動時にconnection refusedと表示され#6で止まる場合がありますが
気長に待っていると起動します。
4. worker起動
worker-0、worker-1を順次起動していきます。
-
worker-0
-
worker-1
補足
-
bootkube.service complete完了後のworker起動
数時間たっても、nodesに追加されない場合は、workerのVMの起動が遅かった可能性があります。
残念ですが、bootstrap、master、workerすべてのVMを削除しbastionの端末でからマニフェストとingitionファイルの再生成からやり直すのが確実です。 -
workerの起動が早い場合
workerの初回起動中に500エラーが発生します。
Workerの起動するタイミングは、以下のbootstrapのjournalctlでモニタリングログでfailed to create some manifests:のメッセージが表示されなくなったタイミングが目安です。
workerを起動タイミングは、bastionの端末から以下のコマンドで、正確に確認することもできます。
[root@bastion #] curl -k https://bootstrap:22623/config/worker
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: [#701] failed to create some manifests:
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: Pod Status:openshift-cluster-version/cluster-version-operator Ready
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-apiserver/kube-apiserver DoesNotExist
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-scheduler/openshift-kube-scheduler DoesNotExist
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-controller-manager/kube-controller-manager DoesNotExist
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: [#702] failed to create some manifests:
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:53 bootstrap bootkube.sh[2577]: [#703] failed to create some manifests:
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: [#723] failed to create some manifests:
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: [#724] failed to create some manifests:
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: [#725] failed to create some manifests:
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: [#726] failed to create some manifests:
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: Created "99_openshift-machineconfig_99-master-ssh.yaml" machineconfigs.v1.machineconfiguration.openshift.io/99-ma ster-ssh -n
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: Created "99_openshift-machineconfig_99-worker-ssh.yaml" machineconfigs.v1.machineconfiguration.openshift.io/99-wo rker-ssh -n
Feb 26 05:43:27 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-apiserver/kube-apiserver DoesNotExist
Feb 26 05:43:27 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-scheduler/openshift-kube-scheduler Pending
Feb 26 05:43:27 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-controller-manager/kube-controller-manager DoesNotExist
Feb 26 05:43:27 bootstrap bootkube.sh[2577]: Pod Status:openshift-cluster-version/cluster-version-operator Ready
Feb 26 05:43:42 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-controller-manager/kube-controller-manager DoesNotExist
Feb 26 05:43:42 bootstrap bootkube.sh[2577]: Pod Status:openshift-cluster-version/cluster-version-operator Ready
Feb 26 05:43:42 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-apiserver/kube-apiserver DoesNotExist
Feb 26 05:43:42 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-scheduler/openshift-kube-scheduler RunningNotReady
Feb 26 05:43:52 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-apiserver/kube-apiserver DoesNotExist
Feb 26 05:43:52 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-scheduler/openshift-kube-scheduler Pending
Feb 26 05:43:52 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-controller-manager/kube-controller-manager DoesNotExist
Feb 26 05:43:52 bootstrap bootkube.sh[2577]: Pod Status:openshift-cluster-version/cluster-version-operator Ready
Feb 26 05:43:57 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-apiserver/kube-apiserver DoesNotExist
Feb 26 05:43:57 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-scheduler/openshift-kube-scheduler Pending
Feb 26 05:43:57 bootstrap bootkube.sh[2577]: Pod Status:openshift-kube-controller-manager/kube-controller-manager Pending
5. クラスター作成完了
5.1. bootkube.service complete
すべてのmasterとworkerを起動後、2.2のbootstrapのbootkube.serviceを確認し、bootkube.service completeと表示されればクラスターの作成が完了です。
5.2. bootsrapからsshログアウト
bootsrapのsshからexistしてsshログアウトし、bastionに戻ります。
[core@bootstrap ~]$ exit
logout
Connection to bootstrap closed.
[root@bastion ~]#
5.3. openshift-installer から確認
# cd /opt/ocp
# /usr/local/bin/openshift-install --dir=bare-metal wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.4.31
DEBUG Built from commit 0227b5f653786d8d58312cd08a2e924e72ae646f
INFO Waiting up to 20m0s for the Kubernetes API at https://api.lab.openshift.local:6443...
INFO API v1.17.1+40d7dbd up
INFO Waiting up to 40m0s for bootstrapping to complete...
DEBUG Bootstrap status: complete
INFO It is now safe to remove the bootstrap resources
6. マシンの認証
クラスターが作成された後は、各マシーンのCSRを承認します。
6.1 OpenShiftにログイン
踏み台サーバ(bastion)からOpenShiftにログインします。
system:adminと表示されれば成功です。
# export KUBECONFIG=/opt/ocp/bare-metal/auth/kubeconfig
# /usr/local/bin/oc whoami
system:admin
6.2. nodesの確認
以下のコマンドで、master-0,master-1,master-2,worker-0,worker-1の5台が認識されているか確認してください。
# /usr/local/bin/oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master-0 Ready master 10m v1.17.1+40d7dbd 172.16.0.110 <none> Red Hat Enterprise Linux CoreOS 44.82.202011202131-0 (Ootpa) 4.18.0-193.29.1.el8_2.x86_64 cri-o://1.17.5-11.rhaos4.4.git7f979af.el8
master-1 Ready master 9m1s v1.17.1+40d7dbd 172.16.0.111 <none> Red Hat Enterprise Linux CoreOS 44.82.202011202131-0 (Ootpa) 4.18.0-193.29.1.el8_2.x86_64 cri-o://1.17.5-11.rhaos4.4.git7f979af.el8
master-2 Ready master 4m6s v1.17.1+40d7dbd 172.16.0.112 <none> Red Hat Enterprise Linux CoreOS 44.82.202011202131-0 (Ootpa) 4.18.0-193.29.1.el8_2.x86_64 cri-o://1.17.5-11.rhaos4.4.git7f979af.el8
worker-0 Ready worker 5m53s v1.17.1+40d7dbd 172.16.0.120 <none> Red Hat Enterprise Linux CoreOS 44.82.202011202131-0 (Ootpa) 4.18.0-193.29.1.el8_2.x86_64 cri-o://1.17.5-11.rhaos4.4.git7f979af.el8
worker-1 Ready worker 3m46s v1.17.1+40d7dbd 172.16.0.121 <none> Red Hat Enterprise Linux CoreOS 44.82.202011202131-0 (Ootpa) 4.18.0-193.29.1.el8_2.x86_64 cri-o://1.17.5-11.rhaos4.4.git7f979af.el8
7.マシンCSR承認
7.1 CSRの確認
PendingになっているCSRを確認します。
# /usr/local/bin/oc get csr
NAME AGE REQUESTOR CONDITION
csr-78qq4 24m system:node:worker-0 Approved,Issued
csr-9dnw9 27m system:node:master-1 Approved,Issued
csr-bnvd7 29m system:node:master-0 Approved,Issued
csr-dvcww 28m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending
csr-lqc5j 22m system:node:worker-1 Approved,Issued
csr-mqt2m 22m system:node:master-2 Approved,Issued
csr-n7s7d 23m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-rhhkt 29m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending
csr-t5p5j 25m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-xwfvc 22m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
7.2 CSRの承認
pendingになっているCSRのNameを指定して、承認します。
# /usr/local/bin/oc adm certificate approve csr-dvcww
certificatesigningrequest.certificates.k8s.io/csr-dvcww approved
# /usr/local/bin/oc adm certificate approve csr-rhhkt
certificatesigningrequest.certificates.k8s.io/csr-rhhkt approved
7.3 CSRの確認
すべて、Approved, Issuedになったことを確認します。
# /usr/local/bin/oc get csr
NAME AGE REQUESTOR CONDITION
csr-78qq4 24m system:node:worker-0 Approved,Issued
csr-9dnw9 27m system:node:master-1 Approved,Issued
csr-bnvd7 29m system:node:master-0 Approved,Issued
csr-dvcww 28m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-lqc5j 22m system:node:worker-1 Approved,Issued
csr-mqt2m 22m system:node:master-2 Approved,Issued
csr-n7s7d 23m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-rhhkt 29m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-t5p5j 25m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-xwfvc 22m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
8. Operator の初期設定
8.1 Operatorの確認
image-registry 以外は AVAILABLE Trueになっていることを確認します。
# /usr/local/bin/oc get clusteroperator
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.4.31 True False False 14m
cloud-credential 4.4.31 True False False 36m
cluster-autoscaler 4.4.31 True False False 23m
console 4.4.31 True False False 21m
csi-snapshot-controller 4.4.31 True False False 26m
dns 4.4.31 True False False 33m
etcd 4.4.31 True False False 29m
image-registry False False True 26m
ingress 4.4.31 True False False 24m
insights 4.4.31 True False False 26m
kube-apiserver 4.4.31 True False False 28m
kube-controller-manager 4.4.31 True False False 32m
kube-scheduler 4.4.31 True False False 32m
kube-storage-version-migrator 4.4.31 True False False 30m
machine-api 4.4.31 True False False 26m
machine-config 4.4.31 True False False 27m
marketplace 4.4.31 True False False 26m
monitoring 4.4.31 True False False 22m
network 4.4.31 True False False 33m
node-tuning 4.4.31 True False False 34m
openshift-apiserver 4.4.31 True False False 25m
openshift-controller-manager 4.4.31 True False False 25m
openshift-samples 4.4.31 True False False 22m
operator-lifecycle-manager 4.4.31 True False False 33m
operator-lifecycle-manager-catalog 4.4.31 True False False 33m
operator-lifecycle-manager-packageserver 4.4.31 True False False 25m
service-ca 4.4.31 True False False 34m
service-catalog-apiserver 4.4.31 True False False 34m
service-catalog-controller-manager 4.4.31 True False False 34m
storage 4.4.31 True False False 26m
8.2 image-registryのパッチ適用
storage にemptyDir:{}のパラメータを追加します。
# /usr/local/bin/oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
config.imageregistry.operator.openshift.io/cluster patched
# /usr/local/bin/oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'
config.imageregistry.operator.openshift.io/cluster patched
# /usr/local/bin/oc get configs.imageregistry.operator.openshift.io -o yaml
apiVersion: v1
items:
- apiVersion: imageregistry.operator.openshift.io/v1
kind: Config
metadata:
creationTimestamp: "2021-02-26T05:51:03Z"
finalizers:
- imageregistry.operator.openshift.io/finalizer
generation: 2
name: cluster
resourceVersion: "30551"
selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
uid: fd18386a-774a-401d-924e-65101804229f
spec:
httpSecret: 93134baefb52122548a08f508fbde6c1253c9c9fa1ba10da00a2549922a52d17b5503502495b8730bd18f28cef86b40aaac3377f9e3c45463548048518968fff
logging: 2
managementState: Managed
proxy: {}
replicas: 1
requests:
read:
maxWaitInQueue: 0s
write:
maxWaitInQueue: 0s
rolloutStrategy: RollingUpdate
storage:
emptyDir: {}
status:
conditions:
- lastTransitionTime: "2021-02-26T06:27:07Z"
message: All registry resources are removed
reason: Removed
status: "False"
type: Progressing
- lastTransitionTime: "2021-02-26T05:51:03Z"
message: The registry is removed
reason: Removed
status: "True"
type: Available
- lastTransitionTime: "2021-02-26T05:51:03Z"
status: "False"
type: Degraded
- lastTransitionTime: "2021-02-26T05:51:03Z"
message: The registry is removed
reason: Removed
status: "True"
type: Removed
observedGeneration: 2
readyReplicas: 0
storage: {}
storageManaged: false
kind: List
metadata:
resourceVersion: ""
selfLink: ""
8.3 修正後のOperatorの確認
すべて AVAILABLE Trueになっていることを確認します。
# /usr/local/bin/oc get clusteroperator
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.4.31 True False False 26m
cloud-credential 4.4.31 True False False 48m
cluster-autoscaler 4.4.31 True False False 35m
console 4.4.31 True False False 33m
csi-snapshot-controller 4.4.31 True False False 39m
dns 4.4.31 True False False 45m
etcd 4.4.31 True False False 41m
image-registry 4.4.31 True False False 38m
ingress 4.4.31 True False False 36m
insights 4.4.31 True False False 38m
kube-apiserver 4.4.31 True False False 40m
kube-controller-manager 4.4.31 True False False 44m
kube-scheduler 4.4.31 True False False 44m
kube-storage-version-migrator 4.4.31 True False False 42m
machine-api 4.4.31 True False False 38m
machine-config 4.4.31 True False False 39m
marketplace 4.4.31 True False False 38m
monitoring 4.4.31 True False False 34m
network 4.4.31 True False False 46m
node-tuning 4.4.31 True False False 46m
openshift-apiserver 4.4.31 True False False 37m
openshift-controller-manager 4.4.31 True False False 37m
openshift-samples 4.4.31 True False False 35m
operator-lifecycle-manager 4.4.31 True False False 45m
operator-lifecycle-manager-catalog 4.4.31 True False False 46m
operator-lifecycle-manager-packageserver 4.4.31 True False False 37m
service-ca 4.4.31 True False False 46m
service-catalog-apiserver 4.4.31 True False False 46m
service-catalog-controller-manager 4.4.31 True False False 46m
storage 4.4.31 True False False 38m
9. インストール完了の確認
Install complete!と表示され、パスワード等の情報が表示されればOKです。
# /usr/local/bin/openshift-install --dir=bare-metal wait-for install-complete --log-level=debug
DEBUG OpenShift Installer 4.4.31
DEBUG Built from commit 0227b5f653786d8d58312cd08a2e924e72ae646f
DEBUG Fetching Install Config...
DEBUG Loading Install Config...
DEBUG Loading SSH Key...
DEBUG Loading Base Domain...
DEBUG Loading Platform...
DEBUG Loading Cluster Name...
DEBUG Loading Base Domain...
DEBUG Loading Platform...
DEBUG Loading Pull Secret...
DEBUG Loading Platform...
DEBUG Using Install Config loaded from state file
DEBUG Reusing previously-fetched Install Config
INFO Waiting up to 30m0s for the cluster at https://api.lab.openshift.local:6443 to initialize...
DEBUG Cluster is initialized
INFO Waiting up to 10m0s for the openshift-console route to be created...
DEBUG Route found in openshift-console namespace: console
DEBUG Route found in openshift-console namespace: downloads
DEBUG OpenShift console route is created
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/opt/ocp/bare-metal/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.lab.openshift.local
INFO Login to the console with user: kubeadmin, password: XXXXX-XXXXX-XXXXX-XXXXX
お疲れ様でした