GitLab v11.0.0からCloud Native GitLab Helm ChartがGitLab Helm Chartに改名され、Beta版になりました。
EKSへのインストール公式手順がイマイチだったのでこちらでまとめてみました。
事前準備
Mac OSで作業しています。事前に以下を揃えておく必要があります。
EKSクラスタの構築
以前ご紹介したEKSCTLツールでクラスタを作成します。
> eksctl create cluster \
--cluster-name $NAME \
--nodes 3 \
--nodes-min 3 \
--nodes-max 5 \
--node-type t2.medium \
--region us-west-2
2018-07-18T13:33:01+09:00 [ℹ] importing SSH public key "/Users/jb/.ssh/id_rsa.pub" as "EKS-bakeneco"
2018-07-18T13:33:02+09:00 [ℹ] creating EKS cluster "bakeneco" in "us-west-2" region
2018-07-18T13:33:02+09:00 [ℹ] creating VPC stack "EKS-bakeneco-VPC"
2018-07-18T13:33:02+09:00 [ℹ] creating ServiceRole stack "EKS-bakeneco-ServiceRole"
2018-07-18T13:33:22+09:00 [✔] created ServiceRole stack "EKS-bakeneco-ServiceRole"
2018-07-18T13:34:03+09:00 [✔] created VPC stack "EKS-bakeneco-VPC"
2018-07-18T13:34:03+09:00 [ℹ] creating control plane "bakeneco"
2018-07-18T13:46:05+09:00 [✔] created control plane "bakeneco"
2018-07-18T13:46:05+09:00 [ℹ] creating DefaultNodeGroup stack "EKS-bakeneco-DefaultNodeGroup"
2018-07-18T13:49:47+09:00 [✔] created DefaultNodeGroup stack "EKS-bakeneco-DefaultNodeGroup"
2018-07-18T13:49:47+09:00 [✔] all EKS cluster "bakeneco" resources has been created
2018-07-18T13:49:47+09:00 [ℹ] wrote "kubeconfig"
2018-07-18T13:49:48+09:00 [ℹ] the cluster has 0 nodes
2018-07-18T13:49:48+09:00 [ℹ] waiting for at least 3 nodes to become ready
2018-07-18T13:50:15+09:00 [ℹ] the cluster has 4 nodes
2018-07-18T13:50:15+09:00 [ℹ] node "ip-192-168-100-194.us-west-2.compute.internal" is ready
2018-07-18T13:50:15+09:00 [ℹ] node "ip-192-168-128-38.us-west-2.compute.internal" is ready
2018-07-18T13:50:15+09:00 [ℹ] node "ip-192-168-177-212.us-west-2.compute.internal" is ready
2018-07-18T13:50:15+09:00 [ℹ] node "ip-192-168-209-209.us-west-2.compute.internal" is ready
2018-07-18T13:50:17+09:00 [ℹ] all command should work, try '/usr/local/bin/kubectl --kubeconfig kubeconfig get nodes'
2018-07-18T13:50:17+09:00 [ℹ] EKS cluster "bakeneco" in "us-west-2" region is ready
kubectlで接続できるように$KUBECONFIG環境変数を設定のうえ、確認します。
> kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-100-194.us-west-2.compute.internal Ready <none> 14m v1.10.3
ip-192-168-106-191.us-west-2.compute.internal Ready <none> 13m v1.10.3
ip-192-168-128-38.us-west-2.compute.internal Ready <none> 14m v1.10.3
ip-192-168-177-212.us-west-2.compute.internal Ready <none> 14m v1.10.3
ip-192-168-209-209.us-west-2.compute.internal Ready <none> 14m v1.10.3
> kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 20m
> kubectl get deployments --all-namespaces
NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kube-system kube-dns 1 1 1 1 24m
> kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-node-d2v4m 1/1 Running 1 14m
kube-system aws-node-fbcr6 1/1 Running 0 13m
kube-system aws-node-ggnps 1/1 Running 1 14m
kube-system aws-node-s2mnr 1/1 Running 0 14m
kube-system aws-node-z274r 1/1 Running 0 14m
kube-system kube-dns-7cc87d595-kdkcv 3/3 Running 0 19m
kube-system kube-proxy-5m9gf 1/1 Running 0 13m
kube-system kube-proxy-8fh9v 1/1 Running 0 14m
kube-system kube-proxy-ggk7j 1/1 Running 0 14m
kube-system kube-proxy-qngg2 1/1 Running 0 14m
kube-system kube-proxy-rxhjn 1/1 Running 0 14m
クラスタ環境の設定
ネットワーク設定
Amazon EKS utilizes Elastic Load Balancers, which are addressed by DNS name and cannot be known ahead of time. Skip this section.
はい、スキップします。
ストレージ設定
Persistent volumeはdynamic provisioningを採用します。特定のゾーンに縛られてしまいますが現時点で仕方がないと思います。
各ストレージが分けられますが今回シンプルに1ストレージクラスにします。
gp2-storage-class.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: gp2
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
zone: us-west-2a
reclaimPolicy: Retain
mountOptions:
- debug
> kubectl create -f gp2-storage-class.yaml
storageclass.storage.k8s.io "gp2" created
> kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io "gp2" patched
> kubectl get storageclass
NAME PROVISIONER AGE
gp2 (default) kubernetes.io/aws-ebs 37s
Tiller
Some clusters require authentication to use kubectl to create the Tiller roles.
For clusters like Amazon EKS, you can directly upload the RBAC configuration.
ということで、Tiller用のServiceAccountを作成しcluster-adminロールを付与します。
rbac-config.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
> kubectl create -f rbac-config.yaml
serviceaccount "tiller" created
clusterrolebinding.rbac.authorization.k8s.io "tiller" created
そしてTillerをインストールし初期化します。
> helm init --service-account tiller
$HELM_HOME has been configured at /Users/jb/.helm.
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
GitLab Chartインストール
GitLab chart repositoryを追加します。
> helm repo add gitlab https://charts.gitlab.io/
"gitlab" has been added to your repositories
HelmでGitLab Chartをインストールします。
> helm upgrade --install gitlab gitlab/gitlab \
--timeout 600 \
--set global.hosts.domain=bakeneco.io \
--set gitlab.migrations.initialRootPassword="XXX" \
--set certmanager-issuer.email=XX@YYY.com
Release "gitlab" does not exist. Installing it now.
NAME: gitlab
LAST DEPLOYED: Wed Jul 18 17:50:08 2018
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/ServiceAccount
NAME SECRETS AGE
gitlab-certmanager-issuer 1 11s
certmanager-gitlab 1 11s
gitlab-gitlab-runner 1 11s
gitlab-nginx-ingress 1 11s
gitlab-prometheus-alertmanager 1 11s
gitlab-prometheus-kube-state-metrics 1 11s
gitlab-prometheus-node-exporter 1 11s
gitlab-prometheus-server 1 11s
==> v1/RoleBinding
NAME AGE
gitlab-certmanager-issuer 3s
gitlab-nginx-ingress 3s
==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
gitlab-unicorn gitlab.bakeneco.io 80, 443 2s
gitlab-minio minio.bakeneco.io 80, 443 2s
gitlab-registry registry.bakeneco.io 80, 443 2s
==> v2beta1/HorizontalPodAutoscaler
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
gitlab-gitlab-shell Deployment/gitlab-gitlab-shell <unknown>/75% 2 10 0 2s
gitlab-sidekiq-all-in-1 Deployment/gitlab-sidekiq-all-in-1 <unknown>/75% 1 10 0 2s
gitlab-unicorn Deployment/gitlab-unicorn <unknown>/75% 2 10 0 2s
gitlab-registry Deployment/gitlab-registry <unknown>/75% 2 10 0 2s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
certmanager-gitlab-54467869c4-c7q6t 0/2 ContainerCreating 0 2s
gitlab-gitlab-runner-845c5b46d8-9vwjm 0/1 Init:0/1 0 2s
gitlab-gitlab-shell-7d8cd44948-lccrx 0/1 Init:0/1 0 2s
gitlab-sidekiq-all-in-1-58c996c9fb-mrcd5 0/1 Init:0/2 0 2s
gitlab-task-runner-866bc87864-tgg4p 0/1 Init:0/1 0 2s
gitlab-unicorn-5c69b9487c-v4pnq 0/1 Init:0/2 0 2s
gitlab-minio-99bff897b-6cplp 0/1 Pending 0 2s
gitlab-nginx-ingress-controller-65d58cbf4d-5zxlm 0/1 ContainerCreating 0 2s
gitlab-nginx-ingress-controller-65d58cbf4d-kp9fg 0/1 ContainerCreating 0 2s
gitlab-nginx-ingress-controller-65d58cbf4d-xttvr 0/1 ContainerCreating 0 2s
gitlab-nginx-ingress-default-backend-699b9476dd-5kjjt 0/1 ContainerCreating 0 1s
gitlab-nginx-ingress-default-backend-699b9476dd-jx86w 0/1 Pending 0 1s
gitlab-postgresql-5578b89f58-4jp9k 0/2 ContainerCreating 0 1s
gitlab-prometheus-server-847c8bb76-9jqm6 0/2 Pending 0 1s
gitlab-redis-6b8b6dbfd9-bs24z 0/2 Init:0/1 0 1s
gitlab-registry-7f4b9ccfc8-k4zsj 0/1 Pending 0 1s
gitlab-gitaly-0 0/1 Pending 0 2s
gitlab-issuer.1-2xqhq 0/1 ContainerCreating 0 2s
gitlab-migrations.1-zs7tj 0/1 Init:0/1 0 2s
gitlab-minio-create-buckets.1-97wvg 0/1 ContainerCreating 0 2s
==> v1beta1/CustomResourceDefinition
NAME AGE
certificates.certmanager.k8s.io 7s
clusterissuers.certmanager.k8s.io 3s
issuers.certmanager.k8s.io 3s
==> v1beta1/ClusterRole
certmanager-gitlab 3s
gitlab-prometheus-kube-state-metrics 3s
gitlab-prometheus-server 3s
==> v1beta1/ClusterRoleBinding
NAME AGE
certmanager-gitlab 3s
gitlab-prometheus-alertmanager 3s
gitlab-prometheus-kube-state-metrics 3s
gitlab-prometheus-node-exporter 3s
gitlab-prometheus-server 3s
==> v1/Role
NAME AGE
gitlab-certmanager-issuer 3s
gitlab-nginx-ingress 3s
==> v1beta2/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
gitlab-gitlab-shell 1 1 1 0 2s
gitlab-sidekiq-all-in-1 1 1 1 0 2s
gitlab-task-runner 1 1 1 0 2s
gitlab-unicorn 1 1 1 0 2s
gitlab-minio 1 1 1 0 2s
gitlab-nginx-ingress-controller 3 0 0 0 2s
gitlab-nginx-ingress-default-backend 2 0 0 0 2s
gitlab-redis 1 0 0 0 2s
gitlab-registry 1 0 0 0 2s
==> v1beta2/StatefulSet
NAME DESIRED CURRENT AGE
gitlab-gitaly 1 1 2s
==> v1beta1/PodDisruptionBudget
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
gitlab-gitaly N/A 1 0 2s
gitlab-gitlab-shell N/A 1 0 2s
gitlab-sidekiq N/A 1 0 2s
gitlab-unicorn N/A 1 0 2s
gitlab-minio-v1 N/A 1 0 2s
gitlab-nginx-ingress-controller 2 N/A 0 2s
gitlab-nginx-ingress-default-backend 1 N/A 0 2s
gitlab-redis-v1 N/A 1 0 2s
gitlab-registry-v1 N/A 1 0 2s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
gitlab-gitaly ClusterIP None <none> 8075/TCP,9236/TCP 3s
gitlab-gitlab-shell ClusterIP 10.100.96.36 <none> 22/TCP 3s
gitlab-unicorn ClusterIP 10.100.170.239 <none> 8080/TCP,8181/TCP 3s
gitlab-minio-svc ClusterIP 10.100.14.232 <none> 9000/TCP 3s
gitlab-nginx-ingress-controller LoadBalancer 10.100.234.221 <pending> 80:30364/TCP,443:30849/TCP,22:31508/TCP 3s
gitlab-nginx-ingress-default-backend ClusterIP 10.100.243.113 <none> 80/TCP 3s
gitlab-postgresql ClusterIP 10.100.10.190 <none> 5432/TCP 3s
gitlab-prometheus-server ClusterIP 10.100.191.234 <none> 80/TCP 3s
gitlab-redis ClusterIP 10.100.110.213 <none> 6379/TCP,9121/TCP 3s
gitlab-registry ClusterIP 10.100.51.212 <none> 5000/TCP 2s
==> v1/Job
NAME DESIRED SUCCESSFUL AGE
gitlab-issuer.1 1 0 2s
gitlab-migrations.1 1 0 2s
gitlab-minio-create-buckets.1 1 0 2s
==> v1/ConfigMap
NAME DATA AGE
gitlab-certmanager-issuer-certmanager 2 11s
gitlab-gitlab-runner 3 11s
gitlab-gitaly 3 11s
gitlab-gitlab-shell 2 11s
gitlab-nginx-ingress-tcp 1 11s
gitlab-migrations 4 11s
gitlab-sidekiq-all-in-1 1 11s
gitlab-sidekiq 6 11s
gitlab-task-runner 4 11s
gitlab-unicorn 8 11s
gitlab-unicorn-tests 1 11s
gitlab-minio-config-cm 3 11s
gitlab-nginx-ingress-controller 7 11s
gitlab-postgresql 0 11s
gitlab-prometheus-server 3 11s
gitlab-redis 2 11s
gitlab-registry 2 11s
==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
gitlab-minio Pending gp2 11s
gitlab-postgresql Bound pvc-9b3e012b-8a67-11e8-8b92-02b5389a29ae 8Gi RWO gp2 11s
gitlab-prometheus-server Bound pvc-9b3f2af5-8a67-11e8-8b92-02b5389a29ae 8Gi RWO gp2 11s
gitlab-redis Bound pvc-9b400dee-8a67-11e8-8b92-02b5389a29ae 5Gi RWO gp2 11s
==> v1beta1/Role
NAME AGE
gitlab-gitlab-runner 3s
==> v1beta1/RoleBinding
NAME AGE
gitlab-gitlab-runner 3s
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
certmanager-gitlab 1 1 1 0 2s
gitlab-gitlab-runner 1 1 1 0 2s
gitlab-postgresql 1 1 1 0 2s
gitlab-prometheus-server 1 1 1 0 2s
暫く見てみたら、GitLab Runnerがうまく立ち上がらないことに気づきました。
> kubectl logs gitlab-gitlab-runner-845c5b46d8-9vwjm
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...
ERROR: Registering runner... failed runner=3clGLmm7 status=couldn't execute POST against https://gitlab.bakeneco.io/api/v4/runners: Post https://gitlab.bakeneco.io/api/v4/runners: dial tcp: lookup gitlab.bakeneco.io on 10.100.0.10:53: no such host
PANIC: Failed to register this runner. Perhaps you are having network problems
どうやらドメインのDNSレコードにIngress LoadBalancerのアドレスを登録しておく必要があるようです。
これはHelm installからでないと分からないのでこのタイミングで確認して登録します。
> kubectl describe service gitlab-nginx-ingress-controller | grep Ingress
LoadBalancer Ingress: aa03b57e68a6711e88b9202b5389a29a-974622501.us-west-2.elb.amazonaws.com
DNSレコード登録後、暫く待つと立ち上がります。
> kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default certmanager-gitlab-54467869c4-c7q6t 2/2 Running 0 44m
default gitlab-gitaly-0 1/1 Running 0 44m
default gitlab-gitlab-runner-845c5b46d8-9vwjm 1/1 Running 13 44m
default gitlab-gitlab-shell-7d8cd44948-fxxls 1/1 Running 0 44m
default gitlab-gitlab-shell-7d8cd44948-lccrx 1/1 Running 0 44m
default gitlab-issuer.1-2xqhq 0/1 Completed 0 44m
default gitlab-migrations.1-zs7tj 0/1 Completed 0 44m
default gitlab-minio-99bff897b-6cplp 1/1 Running 0 44m
default gitlab-minio-create-buckets.1-97wvg 0/1 Completed 0 44m
default gitlab-nginx-ingress-controller-65d58cbf4d-5zxlm 1/1 Running 0 44m
default gitlab-nginx-ingress-controller-65d58cbf4d-kp9fg 1/1 Running 0 44m
default gitlab-nginx-ingress-controller-65d58cbf4d-xttvr 1/1 Running 0 44m
default gitlab-nginx-ingress-default-backend-699b9476dd-5kjjt 1/1 Running 0 44m
default gitlab-nginx-ingress-default-backend-699b9476dd-jx86w 1/1 Running 0 44m
default gitlab-postgresql-5578b89f58-4jp9k 2/2 Running 0 44m
default gitlab-prometheus-server-847c8bb76-9jqm6 2/2 Running 0 44m
default gitlab-redis-6b8b6dbfd9-bs24z 2/2 Running 0 44m
default gitlab-registry-7f4b9ccfc8-k4zsj 1/1 Running 0 44m
default gitlab-registry-7f4b9ccfc8-kf6sm 1/1 Running 0 44m
default gitlab-sidekiq-all-in-1-58c996c9fb-mrcd5 1/1 Running 0 44m
default gitlab-task-runner-866bc87864-tgg4p 1/1 Running 0 44m
default gitlab-unicorn-5c69b9487c-j5klw 1/1 Running 0 44m
default gitlab-unicorn-5c69b9487c-v4pnq 1/1 Running 0 44m
kube-system aws-node-d2v4m 1/1 Running 1 4h
kube-system aws-node-fbcr6 1/1 Running 0 4h
kube-system aws-node-ggnps 1/1 Running 1 4h
kube-system aws-node-s2mnr 1/1 Running 0 4h
kube-system aws-node-z274r 1/1 Running 0 4h
kube-system kube-dns-7cc87d595-kdkcv 3/3 Running 0 4h
kube-system kube-proxy-5m9gf 1/1 Running 0 4h
kube-system kube-proxy-8fh9v 1/1 Running 0 4h
kube-system kube-proxy-ggk7j 1/1 Running 0 4h
kube-system kube-proxy-qngg2 1/1 Running 0 4h
kube-system kube-proxy-rxhjn 1/1 Running 0 4h
kube-system tiller-deploy-f5597467b-b2c5c 1/1 Running 0 4h
GitLabインスタンスにログインしてみたところパラメータで渡したパスワードが効かないようです。
以下で設定されたパスワードを確認しログインします。
> kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode
nyanyanya
デフォルトでHTTPS化されていて, GitLab Shared Runners, Container Registryが入っています。