vSphere with TanzuのWorkload Cluster上にHarborを立てる方法がTKG1.6ベースの方法にupdateされていたので、それを試した時のメモ。
こちらによると、’22/10/31時点ではTKG1.6のパッケージを使えば良いことになっているので、リンク先のこちらの手順に従う。
前提条件
以下が前提条件となっている。
- vSphere7.0u2以上のvSphere with Tanzuでworkload clusterを構築済み
- kubectlとtanzu-cli(v1.4以上)をインストール済み
また、今回はtype: LoadBalancerを使うためにNSX Advanced Load Balancerも構築済みとしている。（MetalLBなどでもOK）
kapp-controllerのインストール
tanzu cliにManagement Clusterを追加する。
export VCENTER_IP=xxxxx
export SUPERVISOR_IP=xxxxx
export KUBECTL_VSPHERE_PASSWORD='xxxxx'
export SUPERVISOR_NAME=xxx
kubectl vsphere login --vsphere-username=administrator@vsphere.local --server=$SUPERVISOR_IP --insecure-skip-tls-verify
kubectl config use-context $VCENTER_IP
tanzu login --kubeconfig ~/.kube/config --context $SUPERVISOR_IP --name $SUPERVISOR_NAME
Workload Clusterに切り替える。
export WORKLOAD_CLUSTER_NAME=tkgs-cluster-1
export WORKLOAD_CLUSTER_NAMESPACE=ns1
kubectl vsphere login --vsphere-username=administrator@vsphere.local \
--server=$SUPERVISOR_IP --insecure-skip-tls-verify \
--tanzu-kubernetes-cluster-name=$WORKLOAD_CLUSTER_NAME \
--tanzu-kubernetes-cluster-namespace=$WORKLOAD_CLUSTER_NAMESPACE
kubectl config use-context $WORKLOAD_CLUSTER_NAME
Cluster作成直後だとdefaultのStorageClassが設定されていないので、defaultを設定する。
export STORAGE_CLASS=xxxxx
kubectl patch storageclass $STORAGE_CLASS -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
kapp-controllerのPSPを作成する。
cat <<EOF > ./tanzu-system-kapp-ctrl-restricted.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: tanzu-system-kapp-ctrl-restricted
spec:
privileged: false
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
volumes:
- configMap
- emptyDir
- projected
- secret
- downwardAPI
- persistentVolumeClaim
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
rule: MustRunAsNonRoot
seLinux:
rule: RunAsAny
supplementalGroups:
rule: MustRunAs
ranges:
- min: 1
max: 65535
fsGroup:
rule: MustRunAs
ranges:
- min: 1
max: 65535
readOnlyRootFilesystem: false
EOF
kubectl apply -f tanzu-system-kapp-ctrl-restricted.yaml
kapp-controllerをapplyする。
kapp-controllerのmanifest
cat << EOF > ./kapp-controller.yaml
apiVersion: v1
kind: Namespace
metadata:
name: tkg-system
---
apiVersion: v1
kind: Namespace
metadata:
name: tanzu-package-repo-global
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1alpha1.data.packaging.carvel.dev
spec:
group: data.packaging.carvel.dev
groupPriorityMinimum: 100
service:
name: packaging-api
namespace: tkg-system
version: v1alpha1
versionPriority: 100
---
apiVersion: v1
kind: Service
metadata:
name: packaging-api
namespace: tkg-system
spec:
ports:
- port: 443
protocol: TCP
targetPort: api
selector:
app: kapp-controller
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: internalpackagemetadatas.internal.packaging.carvel.dev
spec:
group: internal.packaging.carvel.dev
names:
kind: InternalPackageMetadata
listKind: InternalPackageMetadataList
plural: internalpackagemetadatas
singular: internalpackagemetadata
scope: Namespaced
versions:
- name: v1alpha1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
properties:
categories:
items:
type: string
type: array
displayName:
type: string
iconSVGBase64:
type: string
longDescription:
type: string
maintainers:
items:
properties:
name:
type: string
type: object
type: array
providerName:
type: string
shortDescription:
type: string
supportDescription:
type: string
type: object
required:
- spec
type: object
served: true
storage: true
subresources:
status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: internalpackages.internal.packaging.carvel.dev
spec:
group: internal.packaging.carvel.dev
names:
kind: InternalPackage
listKind: InternalPackageList
plural: internalpackages
singular: internalpackage
scope: Namespaced
versions:
- name: v1alpha1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
properties:
capacityRequirementsDescription:
type: string
licenses:
items:
type: string
type: array
refName:
type: string
releaseNotes:
type: string
releasedAt:
format: date-time
nullable: true
type: string
template:
properties:
spec:
properties:
canceled:
description: Canceled when set to true will stop all active changes
type: boolean
cluster:
properties:
kubeconfigSecretRef:
properties:
key:
type: string
name:
type: string
type: object
namespace:
type: string
type: object
deploy:
items:
properties:
kapp:
properties:
delete:
properties:
rawOptions:
items:
type: string
type: array
type: object
inspect:
properties:
rawOptions:
items:
type: string
type: array
type: object
intoNs:
type: string
mapNs:
items:
type: string
type: array
rawOptions:
items:
type: string
type: array
type: object
type: object
type: array
fetch:
items:
properties:
git:
description: TODO implement git
properties:
lfsSkipSmudge:
type: boolean
ref:
type: string
secretRef:
description: 'Secret may include one or more keys: ssh-privatekey, ssh-knownhosts'
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
subPath:
type: string
url:
type: string
type: object
helmChart:
properties:
name:
description: 'Example: stable/redis'
type: string
repository:
properties:
secretRef:
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
url:
type: string
type: object
version:
type: string
type: object
http:
properties:
secretRef:
description: 'Secret may include one or more keys: username, password'
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
sha256:
type: string
subPath:
type: string
url:
description: 'URL can point to one of following formats: text, tgz, zip'
type: string
type: object
image:
properties:
secretRef:
description: 'Secret may include one or more keys: username, password, token. By default anonymous access is used for authentication. TODO support docker config formated secret'
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
subPath:
type: string
url:
description: 'Example: username/app1-config:v0.1.0'
type: string
type: object
imgpkgBundle:
properties:
image:
type: string
secretRef:
description: 'Secret may include one or more keys: username, password, token. By default anonymous access is used for authentication. TODO support docker config formated secret'
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
type: object
inline:
properties:
paths:
additionalProperties:
type: string
type: object
pathsFrom:
items:
properties:
configMapRef:
properties:
directoryPath:
type: string
name:
type: string
type: object
secretRef:
properties:
directoryPath:
type: string
name:
type: string
type: object
type: object
type: array
type: object
type: object
type: array
noopDelete:
description: When NoopDeletion set to true, App deletion should delete App CR but preserve App's associated resources
type: boolean
paused:
description: Paused when set to true will ignore all pending changes, once it set back to false, pending changes will be applied
type: boolean
serviceAccountName:
type: string
syncPeriod:
description: Controls frequency of app reconciliation
type: string
template:
items:
properties:
helmTemplate:
properties:
name:
type: string
namespace:
type: string
path:
type: string
valuesFrom:
items:
properties:
configMapRef:
properties:
name:
type: string
type: object
path:
type: string
secretRef:
properties:
name:
type: string
type: object
type: object
type: array
type: object
jsonnet:
description: TODO implement jsonnet
type: object
kbld:
properties:
paths:
items:
type: string
type: array
type: object
kustomize:
description: TODO implement kustomize
type: object
sops:
properties:
paths:
items:
type: string
type: array
pgp:
properties:
privateKeysSecretRef:
properties:
name:
type: string
type: object
type: object
type: object
ytt:
properties:
fileMarks:
items:
type: string
type: array
ignoreUnknownComments:
type: boolean
inline:
properties:
paths:
additionalProperties:
type: string
type: object
pathsFrom:
items:
properties:
configMapRef:
properties:
directoryPath:
type: string
name:
type: string
type: object
secretRef:
properties:
directoryPath:
type: string
name:
type: string
type: object
type: object
type: array
type: object
paths:
items:
type: string
type: array
strict:
type: boolean
valuesFrom:
items:
properties:
configMapRef:
properties:
name:
type: string
type: object
path:
type: string
secretRef:
properties:
name:
type: string
type: object
type: object
type: array
type: object
type: object
type: array
type: object
required:
- spec
type: object
valuesSchema:
description: valuesSchema can be used to show template values that can be configured by users when a Package is installed in an OpenAPI schema format.
properties:
openAPIv3:
nullable: true
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
version:
type: string
type: object
required:
- spec
type: object
served: true
storage: true
subresources:
status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: apps.kappctrl.k14s.io
spec:
group: kappctrl.k14s.io
names:
kind: App
listKind: AppList
plural: apps
singular: app
scope: Namespaced
versions:
- additionalPrinterColumns:
- description: Friendly description
jsonPath: .status.friendlyDescription
name: Description
type: string
- description: Last time app started being deployed. Does not mean anything was changed.
jsonPath: .status.deploy.startedAt
name: Since-Deploy
type: date
- description: Time since creation
jsonPath: .metadata.creationTimestamp
name: Age
type: date
name: v1alpha1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
properties:
canceled:
description: Canceled when set to true will stop all active changes
type: boolean
cluster:
properties:
kubeconfigSecretRef:
properties:
key:
type: string
name:
type: string
type: object
namespace:
type: string
type: object
deploy:
items:
properties:
kapp:
properties:
delete:
properties:
rawOptions:
items:
type: string
type: array
type: object
inspect:
properties:
rawOptions:
items:
type: string
type: array
type: object
intoNs:
type: string
mapNs:
items:
type: string
type: array
rawOptions:
items:
type: string
type: array
type: object
type: object
type: array
fetch:
items:
properties:
git:
description: TODO implement git
properties:
lfsSkipSmudge:
type: boolean
ref:
type: string
secretRef:
description: 'Secret may include one or more keys: ssh-privatekey, ssh-knownhosts'
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
subPath:
type: string
url:
type: string
type: object
helmChart:
properties:
name:
description: 'Example: stable/redis'
type: string
repository:
properties:
secretRef:
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
url:
type: string
type: object
version:
type: string
type: object
http:
properties:
secretRef:
description: 'Secret may include one or more keys: username, password'
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
sha256:
type: string
subPath:
type: string
url:
description: 'URL can point to one of following formats: text, tgz, zip'
type: string
type: object
image:
properties:
secretRef:
description: 'Secret may include one or more keys: username, password, token. By default anonymous access is used for authentication. TODO support docker config formated secret'
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
subPath:
type: string
url:
description: 'Example: username/app1-config:v0.1.0'
type: string
type: object
imgpkgBundle:
properties:
image:
type: string
secretRef:
description: 'Secret may include one or more keys: username, password, token. By default anonymous access is used for authentication. TODO support docker config formated secret'
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
type: object
inline:
properties:
paths:
additionalProperties:
type: string
type: object
pathsFrom:
items:
properties:
configMapRef:
properties:
directoryPath:
type: string
name:
type: string
type: object
secretRef:
properties:
directoryPath:
type: string
name:
type: string
type: object
type: object
type: array
type: object
type: object
type: array
noopDelete:
description: When NoopDeletion set to true, App deletion should delete App CR but preserve App's associated resources
type: boolean
paused:
description: Paused when set to true will ignore all pending changes, once it set back to false, pending changes will be applied
type: boolean
serviceAccountName:
type: string
syncPeriod:
description: Controls frequency of app reconciliation
type: string
template:
items:
properties:
helmTemplate:
properties:
name:
type: string
namespace:
type: string
path:
type: string
valuesFrom:
items:
properties:
configMapRef:
properties:
name:
type: string
type: object
path:
type: string
secretRef:
properties:
name:
type: string
type: object
type: object
type: array
type: object
jsonnet:
description: TODO implement jsonnet
type: object
kbld:
properties:
paths:
items:
type: string
type: array
type: object
kustomize:
description: TODO implement kustomize
type: object
sops:
properties:
paths:
items:
type: string
type: array
pgp:
properties:
privateKeysSecretRef:
properties:
name:
type: string
type: object
type: object
type: object
ytt:
properties:
fileMarks:
items:
type: string
type: array
ignoreUnknownComments:
type: boolean
inline:
properties:
paths:
additionalProperties:
type: string
type: object
pathsFrom:
items:
properties:
configMapRef:
properties:
directoryPath:
type: string
name:
type: string
type: object
secretRef:
properties:
directoryPath:
type: string
name:
type: string
type: object
type: object
type: array
type: object
paths:
items:
type: string
type: array
strict:
type: boolean
valuesFrom:
items:
properties:
configMapRef:
properties:
name:
type: string
type: object
path:
type: string
secretRef:
properties:
name:
type: string
type: object
type: object
type: array
type: object
type: object
type: array
type: object
status:
properties:
conditions:
items:
description: TODO rename to Condition
properties:
message:
description: Human-readable message indicating details about last transition.
type: string
reason:
description: Unique, this should be a short, machine understandable string that gives the reason for condition's last transition. If it reports "ResizeStarted" that means the underlying persistent volume is being resized.
type: string
status:
type: string
type:
type: string
required:
- status
- type
type: object
type: array
consecutiveReconcileFailures:
type: integer
consecutiveReconcileSuccesses:
type: integer
deploy:
properties:
error:
type: string
exitCode:
type: integer
finished:
type: boolean
startedAt:
format: date-time
type: string
stderr:
type: string
stdout:
type: string
updatedAt:
format: date-time
type: string
type: object
fetch:
properties:
error:
type: string
exitCode:
type: integer
startedAt:
format: date-time
type: string
stderr:
type: string
stdout:
type: string
updatedAt:
format: date-time
type: string
type: object
friendlyDescription:
type: string
inspect:
properties:
error:
type: string
exitCode:
type: integer
stderr:
type: string
stdout:
type: string
updatedAt:
format: date-time
type: string
type: object
managedAppName:
type: string
observedGeneration:
format: int64
type: integer
template:
properties:
error:
type: string
exitCode:
type: integer
stderr:
type: string
updatedAt:
format: date-time
type: string
type: object
usefulErrorMessage:
type: string
type: object
required:
- spec
type: object
served: true
storage: true
subresources:
status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: packageinstalls.packaging.carvel.dev
spec:
group: packaging.carvel.dev
names:
kind: PackageInstall
listKind: PackageInstallList
plural: packageinstalls
shortNames:
- pkgi
singular: packageinstall
scope: Namespaced
versions:
- additionalPrinterColumns:
- description: PackageMetadata name
jsonPath: .spec.packageRef.refName
name: Package name
type: string
- description: PackageMetadata version
jsonPath: .status.version
name: Package version
type: string
- description: Friendly description
jsonPath: .status.friendlyDescription
name: Description
type: string
- description: Time since creation
jsonPath: .metadata.creationTimestamp
name: Age
type: date
name: v1alpha1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
properties:
canceled:
description: Canceled when set to true will stop all active changes
type: boolean
cluster:
properties:
kubeconfigSecretRef:
properties:
key:
type: string
name:
type: string
type: object
namespace:
type: string
type: object
noopDelete:
description: When NoopDelete set to true, PackageInstall deletion should delete PackageInstall/App CR but preserve App's associated resources.
type: boolean
packageRef:
properties:
refName:
type: string
versionSelection:
properties:
constraints:
type: string
prereleases:
properties:
identifiers:
items:
type: string
type: array
type: object
type: object
type: object
paused:
description: Paused when set to true will ignore all pending changes, once it set back to false, pending changes will be applied
type: boolean
serviceAccountName:
type: string
syncPeriod:
description: Controls frequency of App reconciliation in time + unit format. Always >= 30s. If value below 30s is specified, 30s will be used.
type: string
values:
items:
properties:
secretRef:
properties:
key:
type: string
name:
type: string
type: object
type: object
type: array
type: object
status:
properties:
conditions:
items:
description: TODO rename to Condition
properties:
message:
description: Human-readable message indicating details about last transition.
type: string
reason:
description: Unique, this should be a short, machine understandable string that gives the reason for condition's last transition. If it reports "ResizeStarted" that means the underlying persistent volume is being resized.
type: string
status:
type: string
type:
type: string
required:
- status
- type
type: object
type: array
friendlyDescription:
type: string
observedGeneration:
format: int64
type: integer
usefulErrorMessage:
type: string
version:
description: TODO this is desired resolved version (not actually deployed)
type: string
type: object
required:
- spec
type: object
served: true
storage: true
subresources:
status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
packaging.carvel.dev/global-namespace: tanzu-package-repo-global
name: packagerepositories.packaging.carvel.dev
spec:
group: packaging.carvel.dev
names:
kind: PackageRepository
listKind: PackageRepositoryList
plural: packagerepositories
shortNames:
- pkgr
singular: packagerepository
scope: Namespaced
versions:
- additionalPrinterColumns:
- description: Time since creation
jsonPath: .metadata.creationTimestamp
name: Age
type: date
- description: Friendly description
jsonPath: .status.friendlyDescription
name: Description
type: string
name: v1alpha1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
properties:
fetch:
properties:
git:
description: TODO implement git
properties:
lfsSkipSmudge:
type: boolean
ref:
type: string
secretRef:
description: 'Secret may include one or more keys: ssh-privatekey, ssh-knownhosts'
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
subPath:
type: string
url:
type: string
type: object
http:
properties:
secretRef:
description: 'Secret may include one or more keys: username, password'
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
sha256:
type: string
subPath:
type: string
url:
description: 'URL can point to one of following formats: text, tgz, zip'
type: string
type: object
image:
properties:
secretRef:
description: 'Secret may include one or more keys: username, password, token. By default anonymous access is used for authentication. TODO support docker config formated secret'
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
subPath:
type: string
url:
description: 'Example: username/app1-config:v0.1.0'
type: string
type: object
imgpkgBundle:
properties:
image:
type: string
secretRef:
description: 'Secret may include one or more keys: username, password, token. By default anonymous access is used for authentication. TODO support docker config formated secret'
properties:
name:
description: Object is expected to be within same namespace
type: string
type: object
type: object
type: object
paused:
description: Paused when set to true will ignore all pending changes, once it set back to false, pending changes will be applied
type: boolean
syncPeriod:
description: Controls frequency of PackageRepository reconciliation
type: string
required:
- fetch
type: object
status:
properties:
conditions:
items:
description: TODO rename to Condition
properties:
message:
description: Human-readable message indicating details about last transition.
type: string
reason:
description: Unique, this should be a short, machine understandable string that gives the reason for condition's last transition. If it reports "ResizeStarted" that means the underlying persistent volume is being resized.
type: string
status:
type: string
type:
type: string
required:
- status
- type
type: object
type: array
consecutiveReconcileFailures:
type: integer
consecutiveReconcileSuccesses:
type: integer
deploy:
properties:
error:
type: string
exitCode:
type: integer
finished:
type: boolean
startedAt:
format: date-time
type: string
stderr:
type: string
stdout:
type: string
updatedAt:
format: date-time
type: string
type: object
fetch:
properties:
error:
type: string
exitCode:
type: integer
startedAt:
format: date-time
type: string
stderr:
type: string
stdout:
type: string
updatedAt:
format: date-time
type: string
type: object
friendlyDescription:
type: string
observedGeneration:
format: int64
type: integer
template:
properties:
error:
type: string
exitCode:
type: integer
stderr:
type: string
updatedAt:
format: date-time
type: string
type: object
usefulErrorMessage:
type: string
type: object
required:
- spec
type: object
served: true
storage: true
subresources:
status: {}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: kapp-controller-config
namespace: tkg-system
annotations:
kapp.k14s.io/change-group: apps.kappctrl.k14s.io/kapp-controller-config
data:
caCerts: ""
httpProxy: ""
httpsProxy: ""
noProxy: ""
dangerousSkipTLSVerify: ""
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kapp-controller.carvel.dev/version: v0.30.0
kapp.k14s.io/change-rule: upsert after upserting apps.kappctrl.k14s.io/kapp-controller-config
name: kapp-controller
namespace: tkg-system
spec:
replicas: 1
revisionHistoryLimit: 0
selector:
matchLabels:
app: kapp-controller
template:
metadata:
labels:
app: kapp-controller
spec:
containers:
- args:
- -packaging-global-namespace=tanzu-package-repo-global
- -concurrency=4
env:
- name: KAPPCTRL_MEM_TMP_DIR
value: /etc/kappctrl-mem-tmp
- name: KAPPCTRL_SYSTEM_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: KAPPCTRL_API_PORT
value: "10350"
image: projects.registry.vmware.com/tkg/kapp-controller:v0.30.0_vmware.1
name: kapp-controller
ports:
- containerPort: 10350
name: api
protocol: TCP
resources:
requests:
cpu: 120m
memory: 100Mi
securityContext:
runAsGroup: 2000
runAsUser: 1000
volumeMounts:
- mountPath: /etc/kappctrl-mem-tmp
name: template-fs
securityContext:
fsGroup: 3000
serviceAccount: kapp-controller-sa
volumes:
- emptyDir:
medium: Memory
name: template-fs
priorityClassName: system-cluster-critical
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/master
- effect: NoSchedule
key: node.kubernetes.io/not-ready
- effect: NoSchedule
key: node.cloudprovider.kubernetes.io/uninitialized
value: "true"
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kapp-controller-sa
namespace: tkg-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kapp-controller-cluster-role
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- secrets
verbs:
- create
- get
- list
- watch
- apiGroups:
- ""
resources:
- serviceaccounts
verbs:
- get
- apiGroups:
- kappctrl.k14s.io
resources:
- apps
- apps/status
verbs:
- '*'
- apiGroups:
- packaging.carvel.dev
resources:
- packageinstalls
- packageinstalls/status
verbs:
- '*'
- apiGroups:
- packaging.carvel.dev
resources:
- packagerepositories
- packagerepositories/status
verbs:
- '*'
- apiGroups:
- internal.packaging.carvel.dev
resources:
- internalpackagemetadatas
verbs:
- '*'
- apiGroups:
- data.packaging.carvel.dev
resources:
- packagemetadatas
- packagemetadatas/status
verbs:
- '*'
- apiGroups:
- internal.packaging.carvel.dev
resources:
- internalpackages
verbs:
- '*'
- apiGroups:
- data.packaging.carvel.dev
resources:
- packages
- packages/status
verbs:
- '*'
- apiGroups:
- ""
resources:
- configmaps
verbs:
- '*'
- apiGroups:
- apiregistration.k8s.io
resources:
- apiservices
verbs:
- update
- get
- apiGroups:
- ""
resources:
- namespaces
verbs:
- list
- watch
- get
- update
- apiGroups:
- admissionregistration.k8s.io
resources:
- mutatingwebhookconfigurations
verbs:
- list
- watch
- apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- list
- watch
- apiGroups:
- policy
resources:
- podsecuritypolicies
resourceNames:
- tanzu-system-kapp-ctrl-restricted
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kapp-controller-cluster-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kapp-controller-cluster-role
subjects:
- kind: ServiceAccount
name: kapp-controller-sa
namespace: tkg-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: pkg-apiserver:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: kapp-controller-sa
namespace: tkg-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pkgserver-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: kapp-controller-sa
namespace: tkg-system
EOF
kubectl apply -f kapp-controller.yaml
PackageRepositoryの導入
Harborはtanzu packageコマンドでインストールしていくため、packageの配布元となるPackageRepositoryを導入する。名前は適当で問題ない。
export REPOSITORY_NAME=tanzu-repo
tanzu package repository add $REPOSITORY_NAME -n tanzu-package-repo-global --url projects.registry.vmware.com/tkg/packages/standard/repo:v1.6.0
これでHarborを展開できる状態になっている。
$ tanzu package available list
：（省略）
harbor.tanzu.vmware.com harbor OCI Registry 2.5.3+vmware.1-tkg.1
cert-managerのインストール
次にHarborが利用する証明書の管理のためにcert managerをインストールする。
最初に利用可能なバージョンを取得する。
tanzu package available list cert-manager.tanzu.vmware.com -A
利用可能なバージョンを指定してインストールする。
export NAMESPACE=my-packages
export CERT_MANAGER_VER='1.7.2+vmware.1-tkg.1'
tanzu package install cert-manager --package-name cert-manager.tanzu.vmware.com --namespace $NAMESPACE --version $CERT_MANAGER_VER --create-namespace
Contourのインストール
HarborにはIngress経由でアクセスしたいため、Ingress Controllerもインストールする。
cert-managerと同じようにバージョンを調べる。
tanzu package available list contour.tanzu.vmware.com -A
export CONTOUR_VER=1.20.2+vmware.1-tkg.1
Contourの設定ファイルを抽出する手順がマニュアルには書かれているが、その下にあるManifestをコピーした方が早いので、こちらをコピーして利用する。ただし、今回はLoadBalancerを用意しているので、typeをLoadBalancerに変更する。
cat << EOF > ./contour-default-values.yaml
---
infrastructure_provider: vsphere
namespace: tanzu-system-ingress
contour:
configFileContents: {}
useProxyProtocol: false
replicas: 2
pspNames: "vmware-system-restricted"
logLevel: info
envoy:
service:
type: LoadBalancer
annotations: {}
nodePorts:
http: null
https: null
externalTrafficPolicy: Cluster
disableWait: false
hostPorts:
enable: true
http: 80
https: 443
hostNetwork: false
terminationGracePeriodSeconds: 300
logLevel: info
pspNames: null
certificates:
duration: 8760h
renewBefore: 360h
EOF
hostPortsを将来的に使いそうなので、以下のPSP設定も入れる。
kubectl create clusterrolebinding envoy-tkg-admin-privileged-binding --clusterrole=psp:vmware-system-privileged --serviceaccount=tanzu-system-ingress:envoy
Contourをインストールする。
tanzu package install contour \
--package-name contour.tanzu.vmware.com \
--version $CONTOUR_VER \
--values-file contour-default-values.yaml \
--namespace $NAMESPACE
Harborのインストール
いつものごとくバージョンを取得する。
tanzu package available list harbor.tanzu.vmware.com -A
export HARBOR_VER=2.5.3+vmware.1-tkg.1
設定ファイルに関しては、生成せずにドキュメントに記載があるものを使う。なお、そのままコピーすると先頭に2つスペースが入っているので、多分取り除いて使った方がいい。（自分はそうした）
Harborの設定ファイル(harbor-default-values.yaml)
cat <<'EOF' > ./harbor-default-values.yaml
#@data/values
#@overlay/match-child-defaults missing_ok=True
---
#! The namespace to install Harbor
namespace: tanzu-system-registry
#! The FQDN for accessing Harbor admin UI and Registry service.
hostname: harbor.yourdomain.com
#! The network port of the Envoy service in Contour or other Ingress Controller.
port:
https: 443
#! The log level of core, exporter, jobservice, registry. Its value is debug, info, warning, error or fatal.
logLevel: info
#! [Optional] The certificate for harbor FQDN if you want to use your own TLS certificate.
#! We will issue the certificate by cert-manager when it's empty and the tlsCertificateSecretName is empty.
tlsCertificate:
#! [Required] the certificate
tls.crt:
#! [Required] the private key
tls.key:
#! [Optional] the certificate of CA, this enables the download
#! link on portal to download the certificate of CA
ca.crt:
#! [Optional] The name of the secret for harbor FQDN if you want to use your own TLS certificate,
#! which contains keys named:
#! "tls.crt" - the certificate
#! "tls.key" - the private key
#! We will issue the certificate by cert-manager when it's empty and the tlsCertificate is empty.
tlsCertificateSecretName:
#! Use contour http proxy instead of the ingress when it's true
enableContourHttpProxy: true
#! [Required] The initial password of Harbor admin.
harborAdminPassword:
#! [Required] The secret key used for encryption. Must be a string of 16 chars.
secretKey:
database:
#! [Required] The initial password of the postgres database.
password:
shmSizeLimit:
maxIdleConns:
maxOpenConns:
exporter:
cacheDuration:
core:
replicas: 1
#! [Required] Secret is used when core server communicates with other components.
secret:
#! [Required] The XSRF key. Must be a string of 32 chars.
xsrfKey:
jobservice:
replicas: 1
#! [Required] Secret is used when job service communicates with other components.
secret:
registry:
replicas: 1
#! [Required] Secret is used to secure the upload state from client
#! and registry storage backend.
#! See: https://github.com/docker/distribution/blob/master/docs/configuration.md#http
secret:
notary:
#! Whether to install Notary
enabled: true
trivy:
#! enabled the flag to enable Trivy scanner
enabled: true
replicas: 1
#! gitHubToken the GitHub access token to download Trivy DB
gitHubToken: ""
#! skipUpdate the flag to disable Trivy DB downloads from GitHub
#
#! You might want to set the value of this flag to `true` in test or CI/CD environments to avoid GitHub rate limiting issues.
#! If the value is set to `true` you have to manually download the `trivy.db` file and mount it in the
#! `/home/scanner/.cache/trivy/db/trivy.db` path.
skipUpdate: false
#! The persistence is always enabled and a default StorageClass
#! is needed in the k8s cluster to provision volumes dynamically.
#! Specify another StorageClass in the "storageClass" or set "existingClaim"
#! if you have already existing persistent volumes to use
#
#! For storing images and charts, you can also use "azure", "gcs", "s3",
#! "swift" or "oss". Set it in the "imageChartStorage" section
persistence:
persistentVolumeClaim:
registry:
#! Use the existing PVC which must be created manually before bound,
#! and specify the "subPath" if the PVC is shared with other components
existingClaim: ""
#! Specify the "storageClass" used to provision the volume. Or the default
#! StorageClass will be used(the default).
#! Set it to "-" to disable dynamic provisioning
storageClass: ""
subPath: ""
accessMode: ReadWriteOnce
size: 10Gi
jobservice:
existingClaim: ""
storageClass: ""
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
database:
existingClaim: ""
storageClass: ""
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
redis:
existingClaim: ""
storageClass: ""
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
trivy:
existingClaim: ""
storageClass: ""
subPath: ""
accessMode: ReadWriteOnce
size: 5Gi
#! Define which storage backend is used for registry to store
#! images and charts. Refer to
#! https://github.com/docker/distribution/blob/master/docs/configuration.md#storage
#! for the detail.
imageChartStorage:
#! Specify whether to disable `redirect` for images and chart storage, for
#! backends which not supported it (such as using minio for `s3` storage type), please disable
#! it. To disable redirects, simply set `disableredirect` to `true` instead.
#! Refer to
#! https://github.com/docker/distribution/blob/master/docs/configuration.md#redirect
#! for the detail.
disableredirect: false
#! Specify the "caBundleSecretName" if the storage service uses a self-signed certificate.
#! The secret must contain keys named "ca.crt" which will be injected into the trust store
#! of registrys containers.
#! caBundleSecretName:
#! Specify the type of storage: "filesystem", "azure", "gcs", "s3", "swift",
#! "oss" and fill the information needed in the corresponding section. The type
#! must be "filesystem" if you want to use persistent volumes for registry
type: filesystem
filesystem:
rootdirectory: /storage
#maxthreads: 100
azure:
accountname: accountname #! required
accountkey: base64encodedaccountkey #! required
container: containername #! required
realm: core.windows.net #! optional
gcs:
bucket: bucketname #! required
#! The base64 encoded json file which contains the key
encodedkey: base64-encoded-json-key-file #! optional
rootdirectory: null #! optional
chunksize: 5242880 #! optional
s3:
region: us-west-1 #! required
bucket: bucketname #! required
accesskey: null #! eg, awsaccesskey
secretkey: null #! eg, awssecretkey
regionendpoint: null #! optional, eg, http://myobjects.local
encrypt: false #! optional
keyid: null #! eg, mykeyid
secure: true #! optional
skipverify: false #! optional
v4auth: true #! optional
chunksize: null #! optional
rootdirectory: null #! optional
storageclass: STANDARD #! optional
multipartcopychunksize: null #! optional
multipartcopymaxconcurrency: null #! optional
multipartcopythresholdsize: null #! optional
swift:
authurl: https://storage.myprovider.com/v3/auth
username: username
password: password
container: containername
region: null #! eg, fr
tenant: null #! eg, tenantname
tenantid: null #! eg, tenantid
domain: null #! eg, domainname
domainid: null #! eg, domainid
trustid: null #! eg, trustid
insecureskipverify: null #! bool eg, false
chunksize: null #! eg, 5M
prefix: null #! eg
secretkey: null #! eg, secretkey
accesskey: null #! eg, accesskey
authversion: null #! eg, 3
endpointtype: null #! eg, public
tempurlcontainerkey: null #! eg, false
tempurlmethods: null #! eg
oss:
accesskeyid: accesskeyid
accesskeysecret: accesskeysecret
region: regionname
bucket: bucketname
endpoint: null #! eg, endpoint
internal: null #! eg, false
encrypt: null #! eg, false
secure: null #! eg, true
chunksize: null #! eg, 10M
rootdirectory: null #! eg, rootdirectory
#! The http/https network proxy for core, exporter, jobservice, trivy
proxy:
httpProxy:
httpsProxy:
noProxy: 127.0.0.1,localhost,.local,.internal
#! The PSP names used by Harbor pods. The names are separated by ','. 'null' means all PSP can be used.
pspNames: null
#! The metrics used by core, registry and exporter
metrics:
enabled: false
core:
path: /metrics
port: 8001
registry:
path: /metrics
port: 8001
jobservice:
path: /metrics
port: 8001
exporter:
path: /metrics
port: 8001
network:
ipFamilies: ["IPv4","IPv6"]
EOF
Harborのパスワード等を自動生成・設定するスクリプトを実行する。
image_url=$(kubectl -n tanzu-package-repo-global get packages harbor.tanzu.vmware.com.$HARBOR_VER -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}')
imgpkg pull -b $image_url -o /tmp/harbor-package-$HARBOR_VER
bash /tmp/harbor-package-$HARBOR_VER/config/scripts/generate-passwords.sh ./harbor-default-values.yaml
ログインパスワードは任意のものにしたいので、そこだけ書き換える。
sed -ie 's/harborAdminPassword: .*/harborAdminPassword: xxxxx/g' harbor-default-values.yaml
HarborのFQDNも設定する。今回はharbor.tkg.tanzuとした。
sed -ie 's/hostname: .*/hostname: harbor.tkg.tanzu/g' harbor-default-values.yaml
公的な証明書を利用する場合はtls.crt、tls.key、ca.crtも設定するが、今回はオレオレ証明書で済ますため、特に設定しない。
またvSphere with TanzuだとstorageClassとpspNamesも設定する必要がある。
sed -ie 's/storageClass: .*/storageClass: xxxxx/g' harbor-default-values.yaml
sed -ie 's/pspNames: .*/pspNames: vmware-system-restricted/g' harbor-default-values.yaml
また、StorageClassがReadWriteManyをサポートしている場合、registry, jobservice, database, redisのaccessModeをReadWriteManyに変更する必要がある。
今回はまだReadWriteManyを有効化していなかったので、ここは設定しない。
最終的には以下の差分となった。
Harborの設定ファイルの差分
@@ -4,7 +4,7 @@
#! The namespace to install Harbor
namespace: tanzu-system-registry
#! The FQDN for accessing Harbor admin UI and Registry service.
-hostname: harbor.yourdomain.com
+hostname: harbor.tkg.tanzu
#! The network port of the Envoy service in Contour or other Ingress Controller.
port:
https: 443
@@ -29,12 +29,12 @@
#! Use contour http proxy instead of the ingress when it's true
enableContourHttpProxy: true
#! [Required] The initial password of Harbor admin.
-harborAdminPassword:
+harborAdminPassword: xxxxx
#! [Required] The secret key used for encryption. Must be a string of 16 chars.
-secretKey:
+secretKey: ndWrBUEHOMHNyot5
database:
#! [Required] The initial password of the postgres database.
- password:
+ password: wUe5bdoIRMot3een
shmSizeLimit:
maxIdleConns:
maxOpenConns:
@@ -43,19 +43,19 @@
core:
replicas: 1
#! [Required] Secret is used when core server communicates with other components.
- secret:
+ secret: ayaLYKLnj050aGfT
#! [Required] The XSRF key. Must be a string of 32 chars.
- xsrfKey:
+ xsrfKey: jXhEa3pn5xZGd4B5NNlHnIsW3KNggSuy
jobservice:
replicas: 1
#! [Required] Secret is used when job service communicates with other components.
- secret:
+ secret: rccYdGB03hgj1BhH
registry:
replicas: 1
#! [Required] Secret is used to secure the upload state from client
#! and registry storage backend.
#! See: https://github.com/docker/distribution/blob/master/docs/configuration.md#http
- secret:
+ secret: YstfT0CPcmiYU3xK
notary:
#! Whether to install Notary
enabled: true
@@ -87,31 +87,31 @@
#! Specify the "storageClass" used to provision the volume. Or the default
#! StorageClass will be used(the default).
#! Set it to "-" to disable dynamic provisioning
- storageClass: ""
+ storageClass: xxxxx
subPath: ""
accessMode: ReadWriteOnce
size: 10Gi
jobservice:
existingClaim: ""
- storageClass: ""
+ storageClass: xxxxx
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
database:
existingClaim: ""
- storageClass: ""
+ storageClass: xxxxx
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
redis:
existingClaim: ""
- storageClass: ""
+ storageClass: xxxxx
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
trivy:
existingClaim: ""
- storageClass: ""
+ storageClass: xxxxx
subPath: ""
accessMode: ReadWriteOnce
size: 5Gi
@@ -204,7 +204,7 @@
httpsProxy:
noProxy: 127.0.0.1,localhost,.local,.internal
#! The PSP names used by Harbor pods. The names are separated by ','. 'null' means all PSP can be used.
-pspNames: null
+pspNames: vmware-system-restricted
#! The metrics used by core, registry and exporter
metrics:
enabled: false
最後にコメントを削除する。
yq -i eval '... comments=""' harbor-default-values.yaml
コメント削除時、消しすぎることがあるので、先頭に---がなければ追加する。
sed -i '1i ---' harbor-default-values.yaml
作成した設定ファイルを利用してHarborをインストールする。
tanzu package install harbor \
--package-name harbor.tanzu.vmware.com \
--version $HARBOR_VER \
--values-file harbor-default-values.yaml \
--namespace $NAMESPACE
動作確認
今回はContourを利用しているため、HTTPProxyリソースが作成されている。
$ kubectl get httpproxy -n tanzu-system-registry
NAME FQDN TLS SECRET STATUS STATUS DESCRIPTION
harbor-httpproxy harbor.tkg.tanzu harbor-tls valid Valid HTTPProxy
harbor-httpproxy-notary notary.harbor.tkg.tanzu harbor-tls valid Valid HTTPProxy
Contourの場合のEndPointはEnvoyとなる。
$ kubectl get svc -n tanzu-system-ingress envoy
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
envoy LoadBalancer 10.102.115.203 10.220.46.101 80:30763/TCP,443:31909/TCP 15h
試しに通信してみると、通信できていることが分かる。
$ curl -H "Host: harbor.tkg.tanzu" 10.220.46.101 --head
HTTP/1.1 301 Moved Permanently
location: https://harbor.tkg.tanzu/
vary: Accept-Encoding
date: Tue, 01 Nov 2022 02:46:35 GMT
server: envoy
transfer-encoding: chunked
DNSサーバにIPとFQDNを登録してブラウザでアクセスすると、Harborの画面にもアクセス出来る。
