GitLab v11.0.0からCloud Native GitLab Helm ChartがGitLab Helm Chartに改名され、Beta版になりました。
AKSへのインストール公式手順はないので頑張って作ってみました。
事前準備
Mac OSで作業しています。事前に以下を揃えておく必要があります。
AKSクラスタの構築
CLIだけで成功した試しはないのでAzure Web Portalで作ります。
CLIで作るならこのようなコマンドがきっと役に立ちます：
az account list-locationsaz group create --name gitlab-chart-group --location eastusaz aks create -g gitlab-chart-group -n bakeneco -l eastus --enable-rbac --node-count 3 --generate-ssh-keys -k 1.9.6az aks delete -g gitlab-chart-group -n bakeneco
その後Azure CLIでログインし、クラスタに接続できるようにします。
> az login
Note, we have launched a browser for you to login. For old experience with device code, use "az login --use-device-code"
You have logged in. Now let us find all subscriptions you have access to...
[
{
"cloudName": "AzureCloud",
"id": "nyan",
"isDefault": true,
"name": "Microsoft Azure \u30a8\u30f3\u30bf\u30fc\u30d7\u30e9\u30a4\u30ba",
"state": "Enabled",
"tenantId": "nyan",
"user": {
"name": "nyan@nyan.com",
"type": "user"
}
}
]
> az aks get-credentials --resource-group gitlab-chart-group --name bakeneco
Merged "bakeneco" as current context in /Users/jb/.kube/config
> kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-agentpool-15572265-0 Ready agent 13m v1.9.6
aks-agentpool-15572265-1 Ready agent 13m v1.9.6
> kubectl get services --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 13m
kube-system addon-http-application-routing-default-http-backend ClusterIP 10.0.169.174 <none> 80/TCP 12m
kube-system addon-http-application-routing-nginx-ingress LoadBalancer 10.0.156.43 168.62.41.195 80:31997/TCP,443:31037/TCP 12m
kube-system heapster ClusterIP 10.0.76.188 <none> 80/TCP 12m
kube-system kube-dns ClusterIP 10.0.0.10 <none> 53/UDP,53/TCP 12m
kube-system kubernetes-dashboard ClusterIP 10.0.31.110 <none> 80/TCP 12m
> kubectl get deployments --all-namespaces
NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kube-system addon-http-application-routing-default-http-backend 1 1 1 1 13m
kube-system addon-http-application-routing-external-dns 1 1 1 1 13m
kube-system addon-http-application-routing-nginx-ingress-controller 1 1 1 1 13m
kube-system azureproxy 1 1 1 1 13m
kube-system heapster 1 1 1 1 13m
kube-system kube-dns-v20 2 2 2 2 13m
kube-system kubernetes-dashboard 1 1 1 1 13m
kube-system tunnelfront 1 1 1 1 13m
> kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system addon-http-application-routing-default-http-backend-66c97fvmg46 1/1 Running 0 13m
kube-system addon-http-application-routing-external-dns-6c65ddc45d-nlf4k 1/1 Running 0 13m
kube-system addon-http-application-routing-nginx-ingress-controller-64hzwq8 1/1 Running 0 13m
kube-system azureproxy-7c677567f6-hlgdl 1/1 Running 0 13m
kube-system heapster-55f855b47-pqf57 2/2 Running 0 5m
kube-system kube-dns-v20-7c556f89c5-mgm9w 3/3 Running 0 13m
kube-system kube-dns-v20-7c556f89c5-xkwrm 3/3 Running 0 13m
kube-system kube-proxy-6pjzz 1/1 Running 0 13m
kube-system kube-proxy-fzhc6 1/1 Running 0 13m
kube-system kube-svc-redirect-4r7jm 1/1 Running 2 13m
kube-system kube-svc-redirect-wdcbs 1/1 Running 1 13m
kube-system kubernetes-dashboard-b85c46fc-6sznr 1/1 Running 5 13m
kube-system tunnelfront-7b685bd4c5-fkgwh 1/1 Running 0 13m
クラスタ環境の設定
ネットワーク設定
最初からExternal IPがあるので、DNSレコードを登録します。
addon-http-application-routing-nginx-ingressが付与しているExternal IPとは別のものがインストール時に用意されますのでインストール実行後それを確認しDNS登録する必要があります。
ストレージ設定
AzureではデフォルトのStorageClassものが提供されています。今回それをそのまま使います。
Tiller
Some clusters require authentication to use kubectl to create the Tiller roles.
Tiller用のClusterRoleとServiceAccountを作成しcluster-adminロールを付与します。
cluster-admin-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
name: cluster-admin
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
> kubectl create -f cluster-admin-role.yaml
clusterrole.rbac.authorization.k8s.io "cluster-admin" created
rbac-config.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
> kubectl create -f rbac-config.yaml
serviceaccount "tiller" created
clusterrolebinding.rbac.authorization.k8s.io "tiller" created
そしてTillerをインストールし初期化します。
> helm init --service-account tiller
$HELM_HOME has been configured at /Users/jb/.helm.
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
GitLab Chartインストール
GitLab chart repositoryを追加します。
> helm repo add gitlab https://charts.gitlab.io/
"gitlab" has been added to your repositories
HelmでGitLab Chartをインストールします。
> helm upgrade --install gitlab gitlab/gitlab \
--timeout 600 \
--set global.hosts.domain=bakeneco.io \
--set gitlab.migrations.initialRootPassword="nyan" \
--set gitlab.gitaly.persistence.storageClass=default \
--set postgresql.persistence.storageClass=default \
--set gitlab.redis.persistence.storageClass=default \
--set gitlab.minio.persistence.storageClass=default \
--set certmanager-issuer.email=nyan@nyan.com
Release "gitlab" does not exist. Installing it now.
NAME: gitlab
LAST DEPLOYED: Sat Jul 21 18:38:43 2018
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1beta1/PodDisruptionBudget
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
gitlab-gitaly N/A 1 0 8s
gitlab-gitlab-shell N/A 1 0 8s
gitlab-sidekiq N/A 1 0 7s
gitlab-unicorn N/A 1 0 7s
gitlab-minio-v1 N/A 1 0 7s
gitlab-nginx-ingress-controller 2 N/A 0 7s
gitlab-nginx-ingress-default-backend 1 N/A 0 7s
gitlab-redis-v1 N/A 1 0 7s
gitlab-registry-v1 N/A 1 0 7s
==> v1/ServiceAccount
NAME SECRETS AGE
gitlab-certmanager-issuer 1 10s
certmanager-gitlab 1 10s
gitlab-gitlab-runner 1 10s
gitlab-nginx-ingress 1 10s
gitlab-prometheus-alertmanager 1 10s
gitlab-prometheus-kube-state-metrics 1 10s
gitlab-prometheus-node-exporter 1 10s
gitlab-prometheus-server 1 10s
==> v1/Role
NAME AGE
gitlab-certmanager-issuer 9s
gitlab-nginx-ingress 9s
==> v1/Job
NAME DESIRED SUCCESSFUL AGE
gitlab-issuer.1 1 0 8s
gitlab-migrations.1 1 0 8s
gitlab-minio-create-buckets.1 1 0 8s
==> v1/RoleBinding
NAME AGE
gitlab-certmanager-issuer 9s
gitlab-nginx-ingress 9s
==> v1beta2/StatefulSet
NAME DESIRED CURRENT AGE
gitlab-gitaly 1 1 8s
==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
gitlab-minio Pending default 10s
gitlab-postgresql Pending default 10s
gitlab-prometheus-server Pending default 10s
gitlab-redis Pending default 10s
==> v1beta1/CustomResourceDefinition
NAME AGE
certificates.certmanager.k8s.io 10s
clusterissuers.certmanager.k8s.io 10s
issuers.certmanager.k8s.io 9s
==> v1beta1/Role
gitlab-gitlab-runner 9s
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
certmanager-gitlab 1 1 1 0 9s
gitlab-gitlab-runner 1 1 1 0 9s
gitlab-postgresql 1 1 1 0 8s
gitlab-prometheus-server 1 1 1 0 8s
==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
gitlab-unicorn gitlab.bakeneco.io 80, 443 8s
gitlab-minio minio.bakeneco.io 80, 443 8s
gitlab-registry registry.bakeneco.io 80, 443 8s
==> v2beta1/HorizontalPodAutoscaler
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
gitlab-gitlab-shell Deployment/gitlab-gitlab-shell <unknown>/75% 2 10 0 8s
gitlab-sidekiq-all-in-1 Deployment/gitlab-sidekiq-all-in-1 <unknown>/75% 1 10 0 8s
gitlab-unicorn Deployment/gitlab-unicorn <unknown>/75% 2 10 0 8s
gitlab-registry Deployment/gitlab-registry <unknown>/75% 2 10 0 8s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
certmanager-gitlab-56f5486fdb-f6dpb 0/2 ContainerCreating 0 9s
gitlab-gitlab-runner-84dff6795b-ffjg6 0/1 Init:0/1 0 9s
gitlab-gitlab-shell-59c74669b-8fjwg 0/1 Init:0/1 0 9s
gitlab-sidekiq-all-in-1-f9c56885f-wct2v 0/1 Init:0/2 0 9s
gitlab-task-runner-69f766f6dc-l4tq4 0/1 Pending 0 9s
gitlab-unicorn-59ccf4d574-m56z4 0/1 Pending 0 8s
gitlab-minio-567c4bd69d-zrp8m 0/1 Pending 0 8s
gitlab-nginx-ingress-controller-754587444-7lxn7 0/1 Pending 0 8s
gitlab-nginx-ingress-controller-754587444-hgjwk 0/1 Error 0 8s
gitlab-nginx-ingress-controller-754587444-qx86s 0/1 Pending 0 8s
gitlab-nginx-ingress-default-backend-c955db9c6-plvvw 1/1 Running 0 8s
gitlab-nginx-ingress-default-backend-c955db9c6-qx5jh 0/1 Pending 0 8s
gitlab-postgresql-57c66b8d6b-2sxb6 0/2 Pending 0 8s
gitlab-prometheus-server-8cf4fdd8-2m4dk 0/2 Pending 0 8s
gitlab-redis-75764d5485-9bcpg 0/2 Pending 0 8s
gitlab-registry-5d797444d7-fnpb9 0/1 Pending 0 7s
gitlab-gitaly-0 0/1 Pending 0 8s
gitlab-issuer.1-tvnpc 0/1 ContainerCreating 0 8s
gitlab-migrations.1-25sv6 0/1 Pending 0 8s
gitlab-minio-create-buckets.1-5lz7p 0/1 Pending 0 8s
==> v1/ConfigMap
NAME DATA AGE
gitlab-certmanager-issuer-certmanager 2 10s
gitlab-gitlab-runner 3 10s
gitlab-gitaly 3 10s
gitlab-gitlab-shell 2 10s
gitlab-nginx-ingress-tcp 1 10s
gitlab-migrations 4 10s
gitlab-sidekiq-all-in-1 1 10s
gitlab-sidekiq 5 10s
gitlab-task-runner 4 10s
gitlab-unicorn 8 10s
gitlab-unicorn-tests 1 10s
gitlab-minio-config-cm 3 10s
gitlab-nginx-ingress-controller 7 10s
gitlab-postgresql 0 10s
gitlab-prometheus-server 3 10s
gitlab-redis 2 10s
gitlab-registry 2 10s
==> v1beta1/ClusterRole
NAME AGE
certmanager-gitlab 9s
gitlab-prometheus-kube-state-metrics 9s
gitlab-prometheus-server 9s
==> v1beta1/RoleBinding
NAME AGE
gitlab-gitlab-runner 9s
==> v1beta1/ClusterRoleBinding
NAME AGE
certmanager-gitlab 9s
gitlab-prometheus-alertmanager 9s
gitlab-prometheus-kube-state-metrics 9s
gitlab-prometheus-node-exporter 9s
gitlab-prometheus-server 9s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
gitlab-gitaly ClusterIP None <none> 8075/TCP,9236/TCP 9s
gitlab-gitlab-shell ClusterIP 10.0.216.145 <none> 22/TCP 9s
gitlab-unicorn ClusterIP 10.0.226.44 <none> 8080/TCP,8181/TCP 9s
gitlab-minio-svc ClusterIP 10.0.113.89 <none> 9000/TCP 9s
gitlab-nginx-ingress-controller LoadBalancer 10.0.65.171 <pending> 80:32670/TCP,443:31479/TCP,22:31190/TCP 9s
gitlab-nginx-ingress-default-backend ClusterIP 10.0.104.154 <none> 80/TCP 9s
gitlab-postgresql ClusterIP 10.0.238.193 <none> 5432/TCP 9s
gitlab-prometheus-server ClusterIP 10.0.222.23 <none> 80/TCP 9s
gitlab-redis ClusterIP 10.0.148.79 <none> 6379/TCP,9121/TCP 9s
gitlab-registry ClusterIP 10.0.49.168 <none> 5000/TCP 9s
==> v1beta2/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
gitlab-gitlab-shell 1 1 1 0 9s
gitlab-sidekiq-all-in-1 1 1 1 0 9s
gitlab-task-runner 1 1 1 0 9s
gitlab-unicorn 1 1 1 0 9s
gitlab-minio 1 1 1 0 8s
gitlab-nginx-ingress-controller 3 3 3 0 8s
gitlab-nginx-ingress-default-backend 2 2 2 1 8s
gitlab-redis 1 1 1 0 8s
gitlab-registry 1 1 1 0 8s
IngressのExternal IPを確認しDNSレコードに登録します。
> kubectl describe service gitlab-nginx-ingress-controller | grep Ingress
LoadBalancer Ingress: 23.101.134.115
しばらくするとすべてのPodsが立ち上がってログイン画面が見れます。
パスワードを取得してログインします。
> kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode
nyan
手順まとめ
az login
az aks get-credentials --resource-group gitlab-chart-group --name bakeneco
kubectl get nodes
kubectl get pods --all-namespaces
kubectl create -f cluster-admin-role.yaml
kubectl create -f rbac-config.yaml
helm init --service-account tiller
helm repo add gitlab https://charts.gitlab.io/
helm upgrade --install gitlab gitlab/gitlab \
--timeout 600 \
--set global.hosts.domain=bakeneco.io \
--set gitlab.migrations.initialRootPassword="nyan" \
--set gitlab.gitaly.persistence.storageClass=default \
--set postgresql.persistence.storageClass=default \
--set gitlab.redis.persistence.storageClass=default \
--set gitlab.minio.persistence.storageClass=default \
--set certmanager-issuer.email=nyan@nyan.com
kubectl describe service gitlab-nginx-ingress-controller | grep Ingress
kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode
おまけ：派手に失敗したazure-file StorageClassの採用
あ〜ストレージ、本当に楽しいです。
AzureではデフォルトのStorageClassものが提供されていますがAzure-Diskですのであまりうれしくないです。
AzureのStorageClassについて良い記事があったので参考にしました。
まずクラスタのノードと同じリソースグループ（クラスタサービスのグループではない！）にStorage Accountを作ります。
> az storage account create --resource-group MC_gitlab-chart-group_bakeneco_eastus --name gitlabchartstorage
{
"accessTier": null,
"creationTime": "2018-07-21T07:18:12.543726+00:00",
"customDomain": null,
"enableHttpsTrafficOnly": false,
"encryption": {
"keySource": "Microsoft.Storage",
"keyVaultProperties": null,
"services": {
"blob": {
"enabled": true,
"lastEnabledTime": "2018-07-21T07:18:12.762473+00:00"
},
"file": {
"enabled": true,
"lastEnabledTime": "2018-07-21T07:18:12.762473+00:00"
},
"queue": null,
"table": null
}
},
"id": "/subscriptions/cc07cdf5-785f-45be-9718-2283b78ab03a/resourceGroups/gitlab-chart-group/providers/Microsoft.Storage/storageAccounts/gitlabchartstorage",
"identity": null,
"kind": "Storage",
"lastGeoFailoverTime": null,
"location": "eastus",
"name": "gitlabchartstorage",
"networkRuleSet": {
"bypass": "AzureServices",
"defaultAction": "Allow",
"ipRules": [],
"virtualNetworkRules": []
},
"primaryEndpoints": {
"blob": "https://gitlabchartstorage.blob.core.windows.net/",
"file": "https://gitlabchartstorage.file.core.windows.net/",
"queue": "https://gitlabchartstorage.queue.core.windows.net/",
"table": "https://gitlabchartstorage.table.core.windows.net/"
},
"primaryLocation": "eastus",
"provisioningState": "Succeeded",
"resourceGroup": "gitlab-chart-group",
"secondaryEndpoints": {
"blob": "https://gitlabchartstorage-secondary.blob.core.windows.net/",
"file": null,
"queue": "https://gitlabchartstorage-secondary.queue.core.windows.net/",
"table": "https://gitlabchartstorage-secondary.table.core.windows.net/"
},
"secondaryLocation": "westus",
"sku": {
"capabilities": null,
"kind": null,
"locations": null,
"name": "Standard_RAGRS",
"resourceType": null,
"restrictions": null,
"tier": "Standard"
},
"statusOfPrimary": "available",
"statusOfSecondary": "available",
"tags": {},
"type": "Microsoft.Storage/storageAccounts"
}
それからStorageClassを作成してDefaultとします。
aks-storage-class.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: azurefilestorage
provisioner: kubernetes.io/azure-file
parameters:
storageAccount: gitlabchartstorage
reclaimPolicy: Retain
> kubectl create -f aks-storage-class.yaml
storageclass.storage.k8s.io "azurefilestorage" created
> kubectl patch storageclass azurefilestorage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io "azurefilestorage" patched
> kubectl delete sc default
storageclass.storage.k8s.io "default" deleted
> kubectl get storageclass
NAME PROVISIONER AGE
azurefilestorage (default) kubernetes.io/azure-file 43s
managed-premium kubernetes.io/azure-disk 40m
2つのStorageClassが存在すると厄介なことが起きるのでdefaultのStorageClassを削除してましたが、勝手に復活してしまいます。
HelmでGitLab Chartをインストールする直前に急いでdefaultのStorageClassを削除してしまえば復活しないうちにインストールが始まります。
> kubectl delete sc default
storageclass.storage.k8s.io "default" deleted
結果的にはうまくいかず断念しました。Persistent Volume ClaimがちゃんとできましたがminioとpostgresqlがVolumeに対して正しいパーミッションがなかったようです。
> kubectl logs gitlab-minio-567c4bd69d-6cf5s
time="2018-07-21T09:01:18Z" level=error msg="Initializing object layer failed" cause="Unable to initialize '.minio.sys' meta volume, mkdir /export/.minio.sys: permission denied" source="[server-main.go:214:serverMain()]"
> kubectl logs gitlab-postgresql-57c66b8d6b-b7tsh gitlab-postgresql
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
initdb: could not change permissions of directory "/var/lib/postgresql/data/pgdata": Operation not permitted
fixing permissions on existing directory /var/lib/postgresql/data/pgdata ...
どうにかできるはずですが、今回諦めました。。