Go to Qiita Advent Calendar 2024 Top
LoginSignup
12
1

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 1 year has passed since last update.

TUNA-JPAdvent Calendar 2023

Day 18
@tom7

TKG の構築を観察しながら Cluster API を理解する

Last updated at Posted at 2023-12-17

モチベーション

  • Cluster API がどのように動いて TKG の k8sクラスタが構築されているのかを知りたい
    • TKG は Cluster API を活用した製品
    • Cluster API の作法を知らなくても、環境とyamlを1つ用意すれば k8s クラスタが作れてしまう、しかしどのように??

今回は Management Cluster のデプロイの様子を観察することで Cluster API がどのように動作して k8s クラスタを作成しているかを理解してみる.

結論

  • TKG は kind を起点にして Cluster API で Management Clusterを構築している
    • Tanzu CLIと kindにあるCluster API の共同作業で実現
  1. Tanzu CLI が kind で種火となる Tiny single k8s クラスタを作成
  2. Tanzu CLI が kind に Cluster API をインストール
  3. kind の Cluster API が Management Cluster の node VM を作成
  4. Tanzu CLI が kind クラスタのオブジェクトを Management Cluster にコピー
  5. Tanzu CLI が Management Cluster 稼働後に kind を削除する

Cluster API の復習

  • k8s のノードを k8s の pod のように宣言的に扱うことができる
  • メリット
    • 複数のクラスタのアップグレードを管理できる
    • ノードのスケールアウト、スケールインが容易
    • Machine Helath Check によるオートヒーリングが可能
  • Cluster API で追加されたコンポーネント
    • Management Cluster (本記事フォーカス対象)
      • Workload クラスタを作成、削除、管理するためのクラスタ
    • Workload Cluster (今回は対象外)
      • 実際のサービスpod が稼働するクラスタ

環境

  • TKG v2.4.0
  • Load Balancer: NSX-ALB
  • Air-gap 環境 (インターネットアクセスなし)
    • プライベートコンテナレジストリ(Harbor)を用意

確認方法

以下のコマンドを実行完了後に出力されたログ ~.config/tanzu/tkg/logs/${mgmt-cluster-name}.log を元に Cluster API の動きを観察する.

以下、ログは適度に省略しながら記載する.

# Management Cluster の作成
tanzu cluster create -f mgmt-v240.yaml -v9

1. bootstrap VM が kind クラスタを作成

  • YAML の validation チェック
  • bootstrap VM が kind のイメージを docker で取得
  • プライベートコンテナレジストリ(192.168.13.28) のCA証明書が kind 用に作成
Setting up management cluster...
I1217 14:05:34.130814 init.go:122] Validating configuration...
I1217 14:05:34.141650 init.go:181] Using infrastructure provider vsphere:v1.7.1
I1217 14:05:34.141715 init.go:183] Generating cluster configuration...
I1217 14:05:34.141734 init.go:186] Setting up bootstrapper...
I1217 14:05:34.148466 client.go:125] Fetching configuration for kind node image...
I1217 14:05:34.149242 client.go:227] kindConfig: 
[plugins.'io.containerd.grpc.v1.cri'.registry.configs.'192.168.13.28']
insecure_skip_verify = false
ca_file = '/etc/containerd/tkg-registry-ca.crt'
] []}
  • bootstrap VM 内に kind クラスタを作成
I1217 14:05:34.149267 client.go:133] Creating kind cluster: tkg-kind-clvg0bmjs5c8n6ba3en0
I1217 14:05:34.165056 logger.go:115] Creating cluster "tkg-kind-clvg0bmjs5c8n6ba3en0" ...
I1217 14:05:34.165114 logger.go:115] Ensuring node image (192.168.13.28/v2.4.0/kind/node:v1.27.5_vmware.1-tkg.1_v0.17.0) ...
I1217 14:05:34.172035 logger.go:115] Image: 192.168.13.28/v2.4.0/kind/node:v1.27.5_vmware.1-tkg.1_v0.17.0 present locally
I1217 14:05:34.178936 logger.go:115] Preparing nodes ...
I1217 14:05:35.072268 logger.go:115] Writing configuration ...
  • 各種コンポーネントのインストール
    • kind の kubeconfig (root/.kube-tkg/tmp/config_xxxx) がこのタイミングで生成されている
# Install CNI, StorageClass
I1217 14:05:35.237381 logger.go:115] Starting control-plane ...
I1217 14:05:43.612627 logger.go:115] Installing CNI ...
I1217 14:05:44.034368 logger.go:115] Installing StorageClass ...
I1217 14:05:44.326737 logger.go:115] Waiting 2m0s for control-plane = Ready ...
I1217 14:05:59.924954 logger.go:115] Ready after 16s 

# kubeconfig for kind cluster
I1217 14:06:00.076917 init.go:196] Bootstrapper created. Kubeconfig: /root/.kube-tkg/tmp/config_CWg6rjGm

# Install kapp-controller
I1217 14:06:00.085079 init.go:213] Installing kapp-controller on bootstrap cluster...
I1217 14:06:00.091789 management_components.go:375] User ConfigValues File: /tmp/2047343267.yaml
I1217 14:06:00.101771 management_components.go:398] Kapp-controller values-file: /tmp/3561350697.yaml
I1217 14:06:00.240969 management_components.go:414] Kapp-controller configuration file: /tmp/3898824935
I1217 14:06:00.909286 clusterclient.go:1481] waiting for resource kapp-controller of type *v1.Deployment to be up and running
  • Cluster API の主要コンポーネントの情報を取得
    • この図にあるように Cluster API のコアとなるyaml が取得されている
      • core-components - Provider: cluster-api
      • bootstrap-components - Provider: kubeadm
      • control-plane-components - Provider: kubeadm
      • infrastructure-components - Provider: vsphere
I1217 14:06:10.920405 init.go:228] Installing providers on bootstrapper...
I1217 14:06:10.928278 inventory.go:175] Installing the clusterctl inventory CRD
I1217 14:06:10.928499 inventory.go:187] Creating CustomResourceDefinition="providers.clusterctl.cluster.x-k8s.io"
I1217 14:06:11.208330 init.go:111] Fetching providers
I1217 14:06:11.224784 components_client.go:92] Fetching File="core-components.yaml" Provider="cluster-api" Type="CoreProvider" Version="v1.4.5"
I1217 14:06:11.344903 components_client.go:92] Fetching File="bootstrap-components.yaml" Provider="kubeadm" Type="BootstrapProvider" Version="v1.4.5"
I1217 14:06:11.423734 components_client.go:92] Fetching File="control-plane-components.yaml" Provider="kubeadm" Type="ControlPlaneProvider" Version="v1.4.5"
I1217 14:06:11.501407 components_client.go:92] Fetching File="infrastructure-components.yaml" Provider="vsphere" Type="InfrastructureProvider" Version="v1.7.1"
  • cert-manager のインストール
    • (推測) 後続のコンポーネントで使用する証明書を作成
    • 最後にDelete されている
I1217 14:06:11.626481 cert_manager.go:495] Creating Namespace="cert-manager-test"
I1217 14:06:11.635217 cert_manager.go:175] Installing cert-manager Version="v1.10.2"
I1217 14:06:11.635283 components_client.go:92] Fetching File="cert-manager.yaml" Provider="cert-manager" Type="" Version="v1.10.2"
I1217 14:06:11.654415 cert_manager.go:495] Creating Namespace="cert-manager"
I1217 14:06:23.181308 cert_manager.go:495] Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
I1217 14:06:23.203256 cert_manager.go:495] Creating Certificate="selfsigned-cert" Namespace="cert-manager-test"
...
I1217 14:06:23.207018 cert_manager.go:513] Deleting Namespace="cert-manager-test"
I1217 14:06:23.213140 cert_manager.go:513] Deleting Issuer="test-selfsigned" Namespace="cert-manager-test"
I1217 14:06:23.221836 cert_manager.go:513] Deleting Certificate="selfsigned-cert" Namespace="cert-manager-test"
  • cluster-api のインストール
    • capi-system の名前空間にcapi-controller-manager Deployment が作成される
      • これはnode作成やスケールアウトのトラブル時に見る重要な pod の一つ
    • CRD も同時に作成されている
    • 証明書も作成
  • 以下の CLuster API のコンポーネントも同様にインストールされるので省略
    • capi-kubeadm-bootstrap-controller-manager
    • capi-kubeadm-control-plane-controller-manager
    • capv-controller-manager (capi-controller-manager と同じく重要なpod)
I1217 14:06:23.228592 installer.go:111] Installing Provider="cluster-api" Version="v1.4.5" TargetNamespace="capi-system"
I1217 14:06:23.228661 installer.go:115] Creating objects Provider="cluster-api" Version="v1.4.5" TargetNamespace="capi-system"
I1217 14:06:23.232487 components.go:113] Creating Namespace="capi-system"
I1217 14:06:23.238852 components.go:113] Creating CustomResourceDefinition="clusterclasses.cluster.x-k8s.io"
I1217 14:06:23.256934 components.go:113] Creating CustomResourceDefinition="clusterresourcesetbindings.addons.cluster.x-k8s.io"
I1217 14:06:23.268994 components.go:113] Creating CustomResourceDefinition="clusterresourcesets.addons.cluster.x-k8s.io"
I1217 14:06:23.290954 components.go:113] Creating CustomResourceDefinition="clusters.cluster.x-k8s.io"
I1217 14:06:23.358393 components.go:113] Creating CustomResourceDefinition="extensionconfigs.runtime.cluster.x-k8s.io"
I1217 14:06:23.398020 components.go:113] Creating CustomResourceDefinition="ipaddressclaims.ipam.cluster.x-k8s.io"
I1217 14:06:23.425307 components.go:113] Creating CustomResourceDefinition="ipaddresses.ipam.cluster.x-k8s.io"
I1217 14:06:23.433684 components.go:113] Creating CustomResourceDefinition="machinedeployments.cluster.x-k8s.io"
I1217 14:06:23.460223 components.go:113] Creating CustomResourceDefinition="machinehealthchecks.cluster.x-k8s.io"
I1217 14:06:23.494358 components.go:113] Creating CustomResourceDefinition="machinepools.cluster.x-k8s.io"
I1217 14:06:23.523024 components.go:113] Creating CustomResourceDefinition="machines.cluster.x-k8s.io"
I1217 14:06:23.651131 components.go:113] Creating CustomResourceDefinition="machinesets.cluster.x-k8s.io"
I1217 14:06:23.754641 components.go:113] Creating Deployment="capi-controller-manager" Namespace="capi-system"
I1217 14:06:23.762583 components.go:113] Creating Certificate="capi-serving-cert" Namespace="capi-system"
I1217 14:06:23.798481 components.go:113] Creating Issuer="capi-selfsigned-issuer" Namespace="capi-system"
I1217 14:06:23.762583 components.go:113] Creating Certificate="capi-serving-cert" Namespace="capi-system"
I1217 14:06:23.798481 components.go:113] Creating Issuer="capi-selfsigned-issuer" Namespace="capi-system"
  • 全ての Cluster API のコンポーネントが動作しているか確認されている
I1217 14:06:25.891464 init.go:731] installed  Component=="cluster-api"  Type=="CoreProvider"  Version=="v1.4.5"
I1217 14:06:25.891510 init.go:731] installed  Component=="kubeadm"  Type=="BootstrapProvider"  Version=="v1.4.5"
I1217 14:06:25.891533 init.go:731] installed  Component=="kubeadm"  Type=="ControlPlaneProvider"  Version=="v1.4.5"
I1217 14:06:25.891546 init.go:731] installed  Component=="vsphere"  Type=="InfrastructureProvider"  Version=="v1.7.1"
I1217 14:06:31.342782 init.go:952] Passed waiting on provider infrastructure-vsphere after 5.348194854s
I1217 14:06:31.343665 init.go:952] Passed waiting on provider bootstrap-kubeadm after 5.348868066s
I1217 14:06:31.465910 init.go:952] Passed waiting on provider control-plane-kubeadm after 5.459816169s
I1217 14:06:31.466129 init.go:952] Passed waiting on provider cluster-api after 5.436981484s
I1217 14:06:41.173868 init.go:952] Passed waiting on provider ipam-in-cluster after 15.167454637s
I1217 14:06:41.174029 init.go:963] Success waiting on all providers.
  • Auto Managed Package がインストール
    • kapp-controller により実施
I1217 14:07:36.252178 management_component_install.go:452] successfully reconciled package: tkr-vsphere-resolver
I1217 14:07:36.252206 management_component_install.go:452] successfully reconciled package: tkr-source-controller
I1217 14:07:36.252325 management_component_install.go:452] successfully reconciled package: tanzu-framework
I1217 14:07:36.252326 management_component_install.go:452] successfully reconciled package: tanzu-featuregates
I1217 14:07:36.254214 management_component_install.go:452] successfully reconciled package: tkr-service
I1217 14:07:36.254220 management_component_install.go:452] successfully reconciled package: tkg-clusterclass
I1217 14:07:36.254494 management_component_install.go:452] successfully reconciled package: ako-operator
I1217 14:07:36.254555 management_component_install.go:452] successfully reconciled package: tanzu-core-management-plugins
I1217 14:07:36.254819 management_component_install.go:452] successfully reconciled package: tkg-pkg
I1217 14:07:36.254827 management_component_install.go:452] successfully reconciled package: tanzu-cliplugins
I1217 14:07:36.254983 management_component_install.go:452] successfully reconciled package: tanzu-auth
I1217 14:07:36.254989 management_component_install.go:452] successfully reconciled package: tanzu-addons-manager
  • TKr イメージの確認
    • vCenter 上のインベントリで対象オブジェクトがテンプレート化されていないとここでエラーになる
I1217 14:07:36.255036 get_tkrs.go:53] Checking Tkr v1.27.5---vmware.1-tkg.1 is ready...
I1217 14:07:36.259897 get_tkrs.go:58] Checking Tkr v1.27.5---vmware.1-tkg.1 package is installed successfully...
I1217 14:07:36.264855 clusterclass.go:234] Generated user config to /tmp/2568409535.yaml
I1217 14:07:36.265028 clusterclass.go:257] Rendering ClusterClass manifests with YTT templates: [/root/.config/tanzu/tkg/providers/infrastructure-vsphere/v1.7.1/cconly /root/.config/tanzu/tkg/providers/config_default.yaml /tmp/2568409535.yaml]
I1217 14:07:36.373441 clusterclass.go:269] Writing ClusterClass manifests to: /root/.config/tanzu/tkg/clusterclassconfigs/tkg-vsphere-default-v1.1.1.yaml
I1217 14:07:36.373803 clusterclass.go:273] Applying ClusterClass manifests /root/.config/tanzu/tkg/clusterclassconfigs/tkg-vsphere-default-v1.1.1.yaml
  • AKO pod の起動 (NSX-ALBを使用している場合のみ)
    • kind クラスタ上の ako-0 pod が Management Cluster の Control-plane VIP を作成している
I1217 14:07:37.819441 init.go:1064] Get AVI_CONTROL_PLANE_HA_PROVIDER from user config 
I1217 14:07:37.819507 init.go:265] Installing AKO on bootstrapper...

2. Management Cluster のコントロールプレーンノード 1台目が作成

  • ako-0 が NSX-ALB上にVIPを作成したタイミングで Management Cluster のコントロールプレーンノードが1台作成される
  • コントロールプレーンノード1台だけ作成されてスタックする際に確認する点
    • NSX-ALB の設定や状態を疑う
    • コントロールプレーンノードから vCenter:443 に通信できるかどうか
I1217 14:07:38.325790 template_client.go:91] Fetching File="cluster-template-definition-prodcc.yaml" Provider="vsphere" Type="InfrastructureProvider" Version="v1.7.1"
I1217 14:07:39.010952 init.go:283] Management cluster config file has been generated and stored at: '/root/.config/tanzu/tkg/clusterconfigs/mgmt-v240.yaml'
I1217 14:07:39.010983 init.go:293] Start creating management cluster...
I1217 14:07:40.590348 clusterclient.go:967] patch cluster object with operation status: 
	{
		"metadata": {
			"annotations": {
				"TKGOperationInfo" : "{\"Operation\":\"Create\",\"OperationStartTimestamp\":\"2023-12-17 14:07:40.590298793 +0000 UTC\",\"OperationTimeout\":1800}",
				"TKGOperationLastObservedTimestamp" : "2023-12-17 14:07:40.590298793 +0000 UTC"
			}
		}
	}
I1217 14:07:40.590406 resource.go:98] Applying patch to resource mgmt-v240 of type *v1beta1.Cluster ...
I1217 14:07:40.798562 poller.go:63] zero or multiple KCP objects found for the given cluster, 0 mgmt-v240 tkg-system, retrying
I1217 14:07:50.811426 poller.go:63] control plane is not available yet, retrying

3. Management Cluster の Worker ノードが作成

  • NSX-ALB 上の VIP が利用可能になった (VIP --> Management Cluster CP1台目と疎通完了になった) 時点で Worker node の作成が開始される
    • Worker node を3台構成で定義したなら3台まとめて同時に作成される
I1217 14:14:00.803806 init.go:313] Management cluster control plane is available, means API server is ready to receive requests
I1217 14:14:00.803999 clusterclient.go:1688] getting secret for cluster
I1217 14:14:00.805832 init.go:331] Saving management cluster kubeconfig into /root/.kube/config

4. Management Cluster の Control-plane ノードが Scale-out

  • Management Cluster の Control-plane ノードは Worker ノードと異なり、1台ずつスケールアウトする
    • (1台ずつのほうがETCD を追加する作業がやりやすいのかもしれない)
  • スケールアウト中も kapp-controller のインストールが進行中
I1217 14:14:00.820196 init.go:359] Installing kapp-controller on management cluster...
I1217 14:14:00.825069 management_components.go:375] User ConfigValues File: /tmp/1713834471.yaml
I1217 14:14:00.834882 management_components.go:398] Kapp-controller values-file: /tmp/2154323558.yaml
I1217 14:14:00.952402 management_components.go:414] Kapp-controller configuration file: /tmp/2316970380
I1217 14:14:01.782896 clusterclient.go:1481] waiting for resource kapp-controller of type *v1.Deployment to be up and running
I1217 14:14:01.788703 poller.go:63] pods are not yet running for deployment 'kapp-controller' in namespace 'tkg-system', retrying
I1217 14:14:06.804011 clusterclient.go:651] cluster state is unchanged 1
I1217 14:14:06.804062 clusterclient.go:656] cluster control plane is still being initialized: ScalingUp
I1217 14:14:06.804074 poller.go:92] cluster control plane is still being initialized: ScalingUp, retrying
I1217 14:14:21.827830 resource.go:98] Applying patch to resource mgmt-v240 of type *v1beta1.Cluster ...
I1217 14:14:21.869756 poller.go:92] cluster control plane is still being initialized: ScalingUp, retrying

5. Management Cluster に Cluster APIのコンポーネントをインストール

  • Management Cluster の ノードが3台とも Running になった時点で Cluster API のコンポーネントのインストールが開始
    • kind クラスタの時と同様の挙動
# Install cert-manager-test
I1217 14:17:37.593462 cert_manager.go:495] Creating Namespace="cert-manager-test"
I1217 14:17:37.872717 cert_manager.go:175] Installing cert-manager Version="v1.10.2"
I1217 14:17:48.841042 cert_manager.go:495] Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
I1217 14:17:49.891069 cert_manager.go:495] Creating Certificate="selfsigned-cert" Namespace="cert-manager-test"
I1217 14:17:49.897006 cert_manager.go:513] Deleting Namespace="cert-manager-test"
I1217 14:17:49.909803 cert_manager.go:513] Deleting Issuer="test-selfsigned" Namespace="cert-manager-test"
I1217 14:17:49.920604 cert_manager.go:513] Deleting Certificate="selfsigned-cert" Namespace="cert-manager-test"

# Install Cluster API components
I1217 14:17:53.734900 init.go:731] installed  Component=="cluster-api"  Type=="CoreProvider"  Version=="v1.4.5"
I1217 14:17:53.734998 init.go:731] installed  Component=="kubeadm"  Type=="BootstrapProvider"  Version=="v1.4.5"
I1217 14:17:53.735031 init.go:731] installed  Component=="kubeadm"  Type=="ControlPlaneProvider"  Version=="v1.4.5"
I1217 14:17:53.735061 init.go:731] installed  Component=="vsphere"  Type=="InfrastructureProvider"  Version=="v1.7.1"
I1217 14:17:53.735133 init.go:731] installed  Component=="in-cluster"  Type=="IPAMProvider"  Version=="v0.1.0"
I1217 14:17:54.023929 init.go:952] Passed waiting on provider cluster-api after 205.040755ms
I1217 14:17:58.932774 init.go:952] Passed waiting on provider bootstrap-kubeadm after 5.15724766s
I1217 14:17:58.966855 init.go:952] Passed waiting on provider infrastructure-vsphere after 5.191380192s
I1217 14:17:58.968796 init.go:952] Passed waiting on provider control-plane-kubeadm after 5.179215749s
I1217 14:18:08.862362 init.go:952] Passed waiting on provider ipam-in-cluster after 15.072771307s
I1217 14:18:08.862452 init.go:963] Success waiting on all providers.

# Install Auto Managed Packages
I1217 14:19:14.023302 management_component_install.go:452] successfully reconciled package: ako-operator
I1217 14:19:14.023302 management_component_install.go:452] successfully reconciled package: tkg-pkg
I1217 14:19:14.027707 management_component_install.go:452] successfully reconciled package: tanzu-framework
I1217 14:19:14.027888 management_component_install.go:452] successfully reconciled package: tkr-vsphere-resolver
I1217 14:19:14.028011 management_component_install.go:452] successfully reconciled package: tanzu-auth
I1217 14:19:14.028110 management_component_install.go:452] successfully reconciled package: tkr-service
I1217 14:19:14.028138 management_component_install.go:452] successfully reconciled package: tkr-source-controller
I1217 14:19:14.028254 management_component_install.go:452] successfully reconciled package: tanzu-cliplugins
I1217 14:19:14.028395 management_component_install.go:452] successfully reconciled package: tanzu-core-management-plugins
I1217 14:19:14.028384 management_component_install.go:452] successfully reconciled package: tanzu-addons-manager
I1217 14:19:14.028559 management_component_install.go:452] successfully reconciled package: tkg-clusterclass
I1217 14:19:14.028701 management_component_install.go:452] successfully reconciled package: tanzu-featuregates

# TKR
I1217 14:19:14.028823 get_tkrs.go:53] Checking Tkr v1.27.5---vmware.1-tkg.1 is ready...
I1217 14:19:14.039118 get_tkrs.go:58] Checking Tkr v1.27.5---vmware.1-tkg.1 package is installed successfully...
I1217 14:19:14.136661 init.go:576] Checking if TKr v1.27.5---vmware.1-tkg.1 is created on management cluster

6. kind の Cluster API Objects を Management Cluster にコピーする

I1217 14:19:14.140991 init.go:581] Applying ClusterBootstrap: apiVersion: v1
I1217 14:19:15.296342 init.go:427] Moving all Cluster API objects from bootstrap cluster to management cluster...
I1217 14:19:15.348083 mover.go:68] Performing move...
I1217 14:19:15.476603 objectgraph.go:416] Discovering Cluster API objects
I1217 14:19:15.488997 objectgraph.go:458] KubeadmControlPlaneTemplate Count=1
I1217 14:19:15.500434 objectgraph.go:458] Machine Count=6
I1217 14:19:15.508698 objectgraph.go:458] VSphereMachineTemplate Count=6
I1217 14:19:15.514835 objectgraph.go:458] Issuer Count=3
I1217 14:19:15.529617 objectgraph.go:458] CertificateRequest Count=4
I1217 14:19:15.575147 objectgraph.go:458] VSphereMachine Count=6
I1217 14:19:15.584545 objectgraph.go:458] VSphereVM Count=6
I1217 14:19:15.606432 objectgraph.go:458] Secret Count=64
I1217 14:19:15.654300 objectgraph.go:458] ClusterClass Count=1
I1217 14:19:15.676077 objectgraph.go:458] KubeadmConfig Count=6
I1217 14:19:15.683198 objectgraph.go:458] KubeadmControlPlane Count=1
I1217 14:19:15.711457 objectgraph.go:458] MachineSet Count=3
I1217 14:19:15.718075 objectgraph.go:458] MachineHealthCheck Count=4
I1217 14:19:15.724115 objectgraph.go:458] VSphereClusterTemplate Count=1
I1217 14:19:15.739943 objectgraph.go:458] Cluster Count=1
I1217 14:19:15.761110 objectgraph.go:458] KubeadmConfigTemplate Count=4
I1217 14:19:15.780327 objectgraph.go:458] ConfigMap Count=49
I1217 14:19:15.789373 objectgraph.go:458] Certificate Count=4
I1217 14:19:15.796596 objectgraph.go:458] MachineDeployment Count=3
I1217 14:19:15.805316 objectgraph.go:458] VSphereCluster Count=1
I1217 14:19:15.805374 objectgraph.go:467] Total objects Count=195
I1217 14:19:15.858612 mover.go:321] Pausing the source cluster
I1217 14:19:15.901762 mover.go:326] Pausing the source ClusterClasses
I1217 14:19:33.459045 mover.go:361] Resuming the target ClusterClasses
I1217 14:19:33.459100 mover.go:578] Remove Paused annotation ClusterClass="tkg-vsphere-default-v1.1.1" Namespace="tkg-system"
I1217 14:19:33.614196 mover.go:367] Resuming the target cluster
I1217 14:19:33.614270 mover.go:552] Set Cluster.Spec.Paused paused=false Cluster={"name":"mgmt-v240","namespace":"tkg-system"}
I1217 14:19:34.576443 resource.go:98] Applying patch to resource mgmt-v240 of type *v1beta1.Cluster ...
I1217 14:19:34.821799 init.go:486] Creating tkg-bom versioned ConfigMaps...

7. Management Cluster の確認

  • 不要になった kind クラスタが bootstrap VM から削除される
I1217 14:19:34.913253 init.go:492] You can now access the management cluster mgmt-v240 by running 'kubectl config use-context mgmt-v240-admin@mgmt-v240'

# Delete kind
I1217 14:19:34.919837 client.go:163] Deleting kind cluster: tkg-kind-clvg0bmjs5c8n6ba3en0
I1217 14:19:35.671198 logger.go:115] Deleted nodes: ["tkg-kind-clvg0bmjs5c8n6ba3en0-control-plane"]
Management cluster created!

8. Management Cluster の削除

  • (意識していなかったが) Management Cluster を削除するのも kind クラスタ
  • kind が新規に立ち上がり, Cluster API のコンポーネントをインストールした後に Management Cluster を削除している
  • Management Cluster 削除後に kind もTanzu CLI により削除される
# Management Cluster の削除
tanzu mc delete

# Log
Deleting management cluster 'mgmt-v240'. Are you sure? [y/N]: y
Verifying management cluster...
Warning: unable to find component 'kube_rbac_proxy' under BoM
Setting up cleanup cluster...
Installing providers to cleanup cluster...
Fetching providers
Installing cert-manager Version="v1.10.2"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v1.4.5" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v1.4.5" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v1.4.5" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-vsphere" Version="v1.7.1" TargetNamespace="capv-system"
Installing Provider="ipam-in-cluster" Version="v0.1.0" TargetNamespace="caip-in-cluster-system"
Moving TKR and Cluster API objects from management cluster to cleanup cluster...
Performing move...
Discovering Cluster API objects
Moving Cluster API objects Clusters=1
Moving Cluster API objects ClusterClasses=1
Creating objects in the target cluster
Deleting objects from the source cluster
Waiting for the Cluster API objects to get ready after move...
Cleaning up AVI Resources...
Deleting management cluster...
Management cluster 'mgmt-v240' deleted.
Deleting the management cluster context from the kubeconfig file '/root/.kube/config'

Management cluster deleted!

おわりに

知識不足で読み解けない箇所も多かったが Cluster API によってTKGクラスタ(Management Cluster) がどのようにして構築されるかを確認することができた.

参考資料

12
1
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
12
1

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?