はじめに
公式のGitLab Helm Chartでは相対URLが使用できない制限があります。
今回はHelm Chartの中身を一部修正して、相対URLを可能にする手順を説明します。
AWS環境で確認します。
環境情報
EKS 1.16
Helm 2.14.3
Helmfile 0.122.0
GitLab Helm Chart 2.3.7
kube2iam Helm Chart 2.5.0
ALB Ingress Controller Helm Chart 0.1.11
NGINX Ingress Controller Helm Chart 1.39.0
手順
EKS, S3, RDS, ALB Ingress ControllerとS3アクセスのためのIAM Roleは既に作成されているものとします。
GitLabとS3の連携に関してはこちらを参考にしてください。
ALB Ingress ControllerとNGINX Ingress Controllerの組み合わせに関してはこちらを参考にしてください。
必要なKubernetesリソースのデプロイ
kube2iam, ALB Ingress Controller, NGINX Ingress ControllerをHelmfileでデプロイします。
NGINX Ingress Controllerを使用する理由は、rewrite targetによるリダイレクト処理を行いたいからです。
helmfile.yaml
repositories:
- name: stable
url: https://kubernetes-charts.storage.googleapis.com
- name: incubator
url: https://kubernetes-charts-incubator.storage.googleapis.com
releases:
- name: kube2iam
namespace: kube-system
chart: stable/kube2iam
version: 2.5.0
values:
- host:
iptables: true
interface: eni+
extraArgs:
auto-discover-base-arn: ""
rbac:
create: true
- name: aws-alb-ingress-controller
namespace: kube-system
chart: incubator/aws-alb-ingress-controller
version: 0.1.11
values:
- clusterName: test-cluster
autoDiscoverAwsRegion: true
autoDiscoverAwsVpcID: true
podAnnotations:
iam.amazonaws.com/role: aws-alb-ingress-controller
- name: nginx-ingress
namespace: kube-system
chart: stable/nginx-ingress
version: 1.39.0
values:
- controller:
service:
type: NodePort
$ helmfile apply -f ./helmfile.yaml
次に、ALBからNGINX Ingress ControllerへルーティングさせるためのIngressを作成します。
全リクエストを通過するようにします。
alb-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
kubernetes.io/ingress.class: alb
name: alb-ingress
namespace: kube-system
spec:
rules:
- http:
paths:
- backend:
serviceName: nginx-ingress-controller
servicePort: 80
$ kubectl apply -f ./alb-ingress.yaml
GitLab Helm Chartの編集
templatesの中身を修正する必要があるので、次のコマンドで手元にHelm Chartを用意します。
$ helm fetch gitlab/gitlab --version 2.3.7
templates
まずはIngressを編集します。
assetsの対応をしないとcss等を読み込むことができないのでIngressのrewrite targetで/gitlab/assetsを/assetsにリダイレクトさせます。
gitlab/charts/gitlab/charts/unicorn/templates/ingress.yaml
{{- if .Values.enabled -}}
{{- if eq (include "gitlab.ingress.enabled" $) "true" -}}
{{- $gitlabHostname := include "gitlab.gitlab.hostname" . -}}
{{- $tlsSecret := include "unicorn.tlsSecret" . -}}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: {{ template "fullname" . }}
namespace: {{ $.Release.Namespace }}
labels:
{{ include "gitlab.standardLabels" . | indent 4 }}
annotations:
kubernetes.io/ingress.class: "{{ template "gitlab.ingressclass" . }}"
kubernetes.io/ingress.provider: nginx
nginx.ingress.kubernetes.io/proxy-body-size: {{ .Values.ingress.proxyBodySize | quote }}
nginx.ingress.kubernetes.io/proxy-read-timeout: {{ .Values.ingress.proxyReadTimeout | quote }}
nginx.ingress.kubernetes.io/proxy-connect-timeout: {{ .Values.ingress.proxyConnectTimeout | quote }}
{{ include "gitlab.certmanager_annotations" . }}
{{- range $key, $value := merge .Values.ingress.annotations .Values.global.ingress.annotations }}
{{ $key }}: {{ $value | quote }}
{{- end }}
spec:
rules:
- host: {{ $gitlabHostname }}
http:
paths:
- - path: /
+ - path: /gitlab
backend:
serviceName: {{ template "fullname" . }}
servicePort: {{ .Values.service.workhorseExternalPort }}
- - path: /admin/sidekiq
+ - path: /gitlab/admin/sidekiq
backend:
serviceName: {{ template "fullname" . }}
servicePort: {{ .Values.service.externalPort }}
+ - path: /assets
+ backend:
+ serviceName: {{ template "fullname" . }}
+ servicePort: {{ .Values.service.workhorseExternalPort }}
{{- if (and $tlsSecret (eq (include "gitlab.ingress.tls.enabled" $) "true" )) }}
tls:
- hosts:
- {{ $gitlabHostname }}
secretName: {{ $tlsSecret }}
{{- else }}
tls: []
{{- end }}
+ ---
+ apiVersion: extensions/v1beta1
+ kind: Ingress
+ metadata:
+ name: {{ template "fullname" . }}-assets
+ namespace: {{ $.Release.Namespace }}
+ annotations:
+ nginx.ingress.kubernetes.io/rewrite-target: /assets/$1
+ {{ include "gitlab.certmanager_annotations" . }}
+ {{- range $key, $value := merge .Values.ingress.annotations .Values.global.ingress.annotations }}
+ {{ $key }}: {{ $value | quote }}
+ {{- end }}
+ spec:
+ rules:
+ - host: {{ $gitlabHostname }}
+ http:
+ paths:
+ - path: /gitlab/assets/(.*)
+ backend:
+ serviceName: {{ template "fullname" . }}
+ servicePort: {{ .Values.service.workhorseExternalPort }}
{{- end -}}
{{- end -}}
unicornのConfigMapで/gitlabでアクセスするための設定を入れます。
gitlab/charts/gitlab/charts/unicorn/templates/configmap.yaml
{{- if .Values.enabled -}}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "fullname" . }}
namespace: {{ $.Release.Namespace }}
labels:
{{ include "gitlab.standardLabels" . | indent 4 }}
data:
installation_type: |
gitlab-helm-chart
database.yml.erb: |
production:
adapter: postgresql
encoding: unicode
database: {{ template "gitlab.psql.database" . }}
pool: 10
username: {{ template "gitlab.psql.username" . }}
password: "<%= File.read("/etc/gitlab/postgres/psql-password").strip.dump[1..-2] %>"
host: {{ template "gitlab.psql.host" . }}
port: {{ template "gitlab.psql.port" . }}
prepared_statements: {{ template "gitlab.psql.preparedStatements" . }}
# load_balancing:
# hosts:
# - host1.example.com
# - host2.example.com
{{- include "gitlab.psql.ssl.config" . | indent 6 }}
smtp_settings.rb: |
{{ include "gitlab.smtp_settings" . | indent 4 }}
resque.yml.erb: |
production:
# Redis (single instance)
url: {{ template "gitlab.redis.url" . }}
id:
unicorn.rb: |
# This file should be equivalent to `unicorn.rb` from:
# * gitlab-foss: https://gitlab.com/gitlab-org/gitlab-foss/blob/master/config/unicorn.rb.example
# * omnibus: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/unicorn.rb.erb
worker_processes {{ .Values.workerProcesses }}
working_directory "/srv/gitlab"
listen "0.0.0.0:{{ .Values.service.internalPort }}", :tcp_nopush => true
timeout {{ .Values.workerTimeout }}
pid "/home/git/unicorn.pid"
preload_app true
require_relative "/srv/gitlab/lib/gitlab/cluster/lifecycle_events"
before_exec do |server|
# Signal application hooks that we're about to restart
Gitlab::Cluster::LifecycleEvents.do_master_restart
end
run_once = true
before_fork do |server, worker|
if run_once
# There is a difference between Puma and Unicorn:
# - Puma calls before_fork once when booting up master process
# - Unicorn runs before_fork whenever new work is spawned
# To unify this behavior we call before_fork only once (we use
# this callback for deleting Prometheus files so for our purposes
# it makes sense to align behavior with Puma)
run_once = false
# Signal application hooks that we're about to fork
Gitlab::Cluster::LifecycleEvents.do_before_fork
end
# The following is only recommended for memory/DB-constrained
# installations. It is not needed if your system can house
# twice as many worker_processes as you have configured.
#
# This allows a new master process to incrementally
# phase out the old master process with SIGTTOU to avoid a
# thundering herd (especially in the "preload_app false" case)
# when doing a transparent upgrade. The last worker spawned
# will then kill off the old master process with a SIGQUIT.
old_pid = "#{server.config[:pid]}.oldbin"
if old_pid != server.pid
begin
sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
Process.kill(sig, File.read(old_pid).to_i)
rescue Errno::ENOENT, Errno::ESRCH
end
end
#
# Throttle the master from forking too quickly by sleeping. Due
# to the implementation of standard Unix signal handlers, this
# helps (but does not completely) prevent identical, repeated signals
# from being lost when the receiving process is busy.
# sleep 1
end
after_fork do |server, worker|
# Signal application hooks of worker start
Gitlab::Cluster::LifecycleEvents.do_worker_start
# per-process listener ports for debugging/admin/migrations
# addr = "127.0.0.1:#{9293 + worker.nr}"
# server.listen(addr, :tries => -1, :delay => 5, :tcp_nopush => true)
end
ENV['GITLAB_UNICORN_MEMORY_MIN'] = ({{ int .Values.memory.min }} * 1 << 20).to_s
ENV['GITLAB_UNICORN_MEMORY_MAX'] = ({{ int .Values.memory.max }} * 1 << 20).to_s
+ ENV['RAILS_RELATIVE_URL_ROOT'] = "/gitlab"
gitlab.yml.erb: |
production: &base
gitlab:
host: {{ template "gitlab.gitlab.hostname" . }}
https: {{ hasPrefix "https://" (include "gitlab.gitlab.url" .) }}
+ relative_url_root: /gitlab
{{- with .Values.global.hosts.ssh }}
ssh_host: {{ . | quote }}
{{- end }}
{{- with .Values.global.appConfig }}
impersonation_enabled: {{ .enableImpersonation }}
usage_ping_enabled: {{ eq .enableUsagePing true }}
default_can_create_group: {{ eq .defaultCanCreateGroup true }}
username_changing_enabled: {{ eq .usernameChangingEnabled true }}
issue_closing_pattern: {{ .issueClosingPattern | quote }}
default_theme: {{ .defaultTheme }}
{{- include "gitlab.appConfig.defaultProjectsFeatures.configuration" $ | nindent 8 }}
webhook_timeout: {{ .webhookTimeout }}
{{- end }}
trusted_proxies:
{{- if .Values.trusted_proxies }}
{{ toYaml .Values.trusted_proxies | indent 10 }}
{{- end }}
time_zone: {{ .Values.global.time_zone | quote }}
email_from: {{ template "gitlab.email.from" . }}
email_display_name: {{ .Values.global.email.display_name | quote }}
email_reply_to: {{ template "gitlab.email.reply_to" . }}
email_subject_suffix: {{ .Values.global.email.subject_suffix | quote }}
{{- with .Values.global.appConfig }}
{{- if eq .incomingEmail.enabled true }}
{{ include "gitlab.appConfig.incoming_email" . | indent 6 }}
{{- end }}
{{- include "gitlab.appConfig.cronJobs" . | nindent 6 }}
gravatar:
plain_url: {{ .gravatar.plainUrl }}
ssl_url: {{ .gravatar.sslUrl }}
{{ include "gitlab.appConfig.extra" . | indent 6 }}
{{- end }}
{{- include "gitlab.appConfig.artifacts.configuration" (dict "config" $.Values.global.appConfig.artifacts "context" $) | nindent 6 }}
{{- include "gitlab.appConfig.lfs.configuration" (dict "config" $.Values.global.appConfig.lfs "context" $) | nindent 6 }}
{{- include "gitlab.appConfig.uploads.configuration" (dict "config" $.Values.global.appConfig.uploads "context" $) | nindent 6 }}
{{- include "gitlab.appConfig.packages.configuration" (dict "config" $.Values.global.appConfig.packages "context" $) | nindent 6 }}
{{- include "gitlab.appConfig.external_diffs.configuration" (dict "config" $.Values.global.appConfig.externalDiffs "context" $) | nindent 6 }}
pages:
enabled: false
mattermost:
enabled: false
gitlab_ci:
{{- include "gitlab.appConfig.ldap.configuration" $ | nindent 6 }}
{{- include "gitlab.appConfig.omniauth.configuration" $ | nindent 6 }}
kerberos:
enabled: false
shared:
{{ include "gitlab.appConfig.gitaly" . | indent 6 }}
{{ include "gitlab.appConfig.repositories" . | indent 6 }}
backup:
path: "tmp/backups" # Relative paths are relative to Rails.root (default: tmp/backups/)
{{ include "gitlab.appConfig.shell" . | indent 6 }}
{{ include "gitlab.appConfig.shell.ssh_port" . | indent 8 }}
{{ include "gitlab.appConfig.shell.secret_file" . | indent 8 }}
workhorse:
secret_file: /etc/gitlab/gitlab-workhorse/secret
git:
bin_path: /usr/bin/git
webpack:
monitoring:
ip_whitelist:
{{- if kindIs "array" .Values.monitoring.ipWhitelist }}
{{ toYaml .Values.monitoring.ipWhitelist | nindent 10 | trim }}
{{- end }}
sidekiq_exporter:
{{ include "gitlab.appConfig.rackAttack" . | indent 6 }}
## Registry Integration
{{- include "gitlab.appConfig.registry.configuration" $ | nindent 6 }}
configure: |
{{- include "gitlab.scripts.configure.secrets" (dict) | nindent 4 -}}
{{- include "gitlab.psql.ssl.initScript" . | nindent 4 }}
+ relative_url.rb: |
+ Rails.application.configure do
+ config.relative_url_root = "/gitlab"
+ end
---
apiVersion: v1
kind: ConfigMap
metadata:
name: {{.Release.Name }}-workhorse-config
namespace: {{ $.Release.Namespace }}
labels:
{{ include "gitlab.standardLabels" . | indent 4 }}
data:
installation_type: |
gitlab-helm-chart
workhorse-config.toml.erb: |
[redis]
URL = "{{ template "gitlab.redis.scheme" . }}://{{ template "gitlab.redis.host" . }}:{{ template "gitlab.redis.port" . }}"
{{- if .Values.global.redis.password.enabled }}
Password = "<%= File.read("/etc/gitlab/redis/password").strip.dump[1..-2] %>"
{{- end }}
configure: |
set -e
mkdir -p /init-secrets-workhorse/gitlab-workhorse
cp -v -r -L /init-config/gitlab-workhorse/secret /init-secrets-workhorse/gitlab-workhorse/secret
{{- if .Values.global.redis.password.enabled }}
mkdir -p /init-secrets-workhorse/redis
cp -v -r -L /init-config/redis/password /init-secrets-workhorse/redis/
{{- end }}
# Leave this here - This line denotes end of block to the parser.
{{- end }}
unicornのDeploymentで新規追加するファイルをマウントさせるための設定を入れます。
また、相対パスにすることによりliveness, readinessが失敗するようになるので該当部分を削除します。
/tmpを追加するのはGitLab CI実行時にS3へartifactsアップロードに失敗しないようにするためです。
gitlab/charts/gitlab/charts/unicorn/templates/deployment.yaml
{{- if .Values.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "fullname" . }}
namespace: {{ $.Release.Namespace }}
labels:
{{ include "gitlab.standardLabels" . | indent 4 }}
spec:
{{- if .Values.global.operator.enabled }}
paused: true
{{- end }}
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: {{ template "name" . }}
release: {{ .Release.Name }}
template:
metadata:
labels:
app: {{ template "name" . }}
release: {{ .Release.Name }}
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yml") . | sha256sum }}
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
{{- range $key, $value := .Values.annotations }}
{{ $key }}: {{ $value | quote }}
{{- end }}
{{- if .Values.metrics.enabled }}
{{ toYaml .Values.metrics.annotations | indent 8 }}
{{- end }}
spec:
{{- if .Values.tolerations }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
{{- end }}
securityContext:
runAsUser: 1000
fsGroup: 1000
{{- if eq (default .Values.global.antiAffinity .Values.antiAffinity) "hard" }}
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app: {{ template "name" . }}
release: {{ .Release.Name }}
{{- else if eq (default .Values.global.antiAffinity .Values.antiAffinity) "soft" }}
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app: {{ template "name" . }}
release: {{ .Release.Name }}
{{- end }}
initContainers:
{{ include "gitlab.extraInitContainers" . | indent 8 }}
{{ include "gitlab.certificates.initContainer" . | indent 8 }}
- name: configure
command: ['sh']
args: [ '-c', 'sh -x /config-unicorn/configure ; sh -x /config-workhorse/configure']
image: {{ .Values.init.image }}:{{ .Values.init.tag }}
volumeMounts:
{{ include "gitlab.extraVolumeMounts" . | indent 10 }}
{{ include "gitlab.psql.ssl.volumeMount" . | indent 10 }}
- name: unicorn-config
mountPath: /config-unicorn
readOnly: true
- name: workhorse-config
mountPath: /config-workhorse
readOnly: true
- name: init-unicorn-secrets
mountPath: /init-config
readOnly: true
- name: unicorn-secrets
mountPath: /init-secrets
readOnly: false
- name: workhorse-secrets
mountPath: /init-secrets-workhorse
readOnly: false
resources:
{{ toYaml .Values.init.resources | indent 12 }}
- name: dependencies
image: "{{ coalesce .Values.image.repository (include "image.repository" .) }}:{{ coalesce .Values.image.tag (include "gitlab.versionTag" . ) }}"
{{ template "gitlab.imagePullPolicy" . }}
args:
- /scripts/wait-for-deps
env:
{{- if .Values.global.operator.enabled }}
- name: BYPASS_SCHEMA_VERSION
value: 'true'
{{- end }}
- name: GITALY_FEATURE_DEFAULT_ON
value: "1"
- name: CONFIG_TEMPLATE_DIRECTORY
value: '/var/opt/gitlab/templates'
- name: CONFIG_DIRECTORY
value: '/srv/gitlab/config'
- name: WORKHORSE_ARCHIVE_CACHE_DISABLED
value: "1"
volumeMounts:
{{ include "gitlab.extraVolumeMounts" . | indent 12 }}
- name: unicorn-config
mountPath: '/var/opt/gitlab/templates'
- name: unicorn-secrets
mountPath: '/etc/gitlab'
readOnly: true
resources:
{{ toYaml .Values.init.resources | indent 12 }}
{{- include "pullsecrets" .Values.image | indent 6}}
containers:
{{ include "gitlab.extraContainers" . | indent 8 }}
- name: {{ .Chart.Name }}
image: "{{ coalesce .Values.image.repository (include "image.repository" .) }}:{{ coalesce .Values.image.tag (include "gitlab.versionTag" . ) }}"
{{ template "gitlab.imagePullPolicy" . }}
ports:
- containerPort: {{ .Values.service.internalPort }}
name: unicorn
env:
- name: GITALY_FEATURE_DEFAULT_ON
value: "1"
- name: CONFIG_TEMPLATE_DIRECTORY
value: '/var/opt/gitlab/templates'
- name: CONFIG_DIRECTORY
value: '/srv/gitlab/config'
{{- if .Values.metrics.enabled }}
- name: prometheus_multiproc_dir
value: /metrics
{{- end }}
{{- if .Values.workhorse.sentryDSN }}
- name: GITLAB_WORKHORSE_SENTRY_DSN
value: {{ .Values.workhorse.sentryDSN }}
{{- end }}
volumeMounts:
{{- if .Values.metrics.enabled }}
- name: unicorn-metrics
mountPath: '/metrics'
{{- end }}
- name: unicorn-config
mountPath: '/var/opt/gitlab/templates'
- name: unicorn-secrets
mountPath: '/etc/gitlab'
readOnly: true
- name: unicorn-secrets
mountPath: /srv/gitlab/config/secrets.yml
subPath: rails-secrets/secrets.yml
- name: unicorn-config
mountPath: '/srv/gitlab/config/initializers/smtp_settings.rb'
subPath: smtp_settings.rb
+ - name: unicorn-config
+ mountPath: '/srv/gitlab/config/initializers/relative_url.rb'
+ subPath: relative_url.rb
- name: unicorn-config
mountPath: '/srv/gitlab/INSTALLATION_TYPE'
subPath: installation_type
- name: shared-upload-directory
mountPath: /srv/gitlab/public/uploads/tmp
readOnly: false
- name: shared-artifact-directory
mountPath: /srv/gitlab/shared
readOnly: false
+ - name: shared-tmp
+ mountPath: '/tmp'
+ readOnly: false
{{ include "gitlab.certificates.volumeMount" . | indent 12 }}
{{ include "gitlab.extraVolumeMounts" . | indent 12 }}
- livenessProbe:
- exec:
- command:
- - /scripts/healthcheck
- initialDelaySeconds: 20
- timeoutSeconds: 30
- periodSeconds: 60
- readinessProbe:
- exec:
- command:
- - /scripts/healthcheck
- timeoutSeconds: 2
lifecycle:
preStop:
exec:
command: ["/bin/bash", "-c", "pkill -SIGQUIT -f 'unicorn master'"]
resources:
{{ toYaml .Values.resources | indent 12 }}
- name: gitlab-workhorse
image: "{{ coalesce .Values.workhorse.image (include "workhorse.repository" .) }}:{{ coalesce .Values.workhorse.tag (include "gitlab.versionTag" . ) }}"
{{ template "gitlab.imagePullPolicy" . }}
ports:
- containerPort: {{ .Values.service.workhorseInternalPort }}
name: workhorse
env:
- name: GITLAB_WORKHORSE_EXTRA_ARGS
value: {{ .Values.workhorse.extraArgs | quote }}
- name: GITLAB_WORKHORSE_LISTEN_PORT
value: {{ default 8181 .Values.service.workhorseInternalPort | int | quote }}
- name: CONFIG_TEMPLATE_DIRECTORY
value: '/var/opt/gitlab/templates'
- name: CONFIG_DIRECTORY
value: '/srv/gitlab/config'
volumeMounts:
- name: workhorse-config
mountPath: '/var/opt/gitlab/templates'
- name: workhorse-secrets
mountPath: '/etc/gitlab'
readOnly: true
- name: shared-upload-directory
mountPath: /srv/gitlab/public/uploads/tmp
readOnly: false
- name: shared-artifact-directory
mountPath: /srv/gitlab/shared
readOnly: false
+ - name: shared-tmp
+ mountPath: '/tmp'
+ readOnly: false
{{ include "gitlab.certificates.volumeMount" . | indent 12 }}
{{ include "gitlab.extraVolumeMounts" . | indent 12 }}
- livenessProbe:
- exec:
- command:
- - /scripts/healthcheck
- initialDelaySeconds: 20
- timeoutSeconds: 30
- periodSeconds: 60
- readinessProbe:
- exec:
- command:
- - /scripts/healthcheck
- timeoutSeconds: 2
resources:
{{ toYaml .Values.workhorse.resources | indent 12 }}
volumes:
{{ include "gitlab.extraVolumes" . | indent 6 }}
{{ include "gitlab.psql.ssl.volume" . | indent 6 }}
{{- if .Values.metrics.enabled }}
- name: unicorn-metrics
emptyDir:
medium: "Memory"
{{- end }}
- name: unicorn-config
configMap:
name: {{ template "fullname" . }}
- name: workhorse-config
configMap:
name: {{ .Release.Name }}-workhorse-config
- name: init-unicorn-secrets
projected:
defaultMode: 0400
sources:
- secret:
name: {{ template "gitlab.rails-secrets.secret" . }}
items:
- key: secrets.yml
path: rails-secrets/secrets.yml
- secret:
name: {{ template "gitlab.gitlab-shell.authToken.secret" . }}
items:
- key: {{ template "gitlab.gitlab-shell.authToken.key" . }}
path: shell/.gitlab_shell_secret
- secret:
name: {{ template "gitlab.gitaly.authToken.secret" . }}
items:
- key: {{ template "gitlab.gitaly.authToken.key" . }}
path: gitaly/gitaly_token
{{- if .Values.global.redis.password.enabled }}
- secret:
name: {{ template "gitlab.redis.password.secret" . }}
items:
- key: {{ template "gitlab.redis.password.key" . }}
path: redis/password
{{- end }}
- secret:
name: {{ template "gitlab.psql.password.secret" . }}
items:
- key: {{ template "gitlab.psql.password.key" . }}
path: postgres/psql-password
- secret:
name: {{ template "gitlab.registry.certificate.secret" . }}
items:
- key: registry-auth.key
path: registry/gitlab-registry.key
- secret:
name: {{ template "gitlab.workhorse.secret" . }}
items:
- key: {{ template "gitlab.workhorse.key" . }}
path: gitlab-workhorse/secret
{{- include "gitlab.minio.mountSecrets" $ | nindent 10 }}
{{- include "gitlab.appConfig.objectStorage.mountSecrets" (dict "name" "artifacts" "config" $.Values.global.appConfig.artifacts) | nindent 10 }}
{{- include "gitlab.appConfig.objectStorage.mountSecrets" (dict "name" "lfs" "config" $.Values.global.appConfig.lfs) | nindent 10 }}
{{- include "gitlab.appConfig.objectStorage.mountSecrets" (dict "name" "uploads" "config" $.Values.global.appConfig.uploads) | nindent 10 }}
{{- include "gitlab.appConfig.objectStorage.mountSecrets" (dict "name" "packages" "config" $.Values.global.appConfig.packages) | nindent 10 }}
{{- include "gitlab.appConfig.objectStorage.mountSecrets" (dict "name" "external_diffs" "config" $.Values.global.appConfig.externalDiffs) | nindent 10 }}
{{- include "gitlab.appConfig.ldap.servers.mountSecrets" $ | nindent 10 }}
{{- include "gitlab.appConfig.omniauth.mountSecrets" $ | nindent 10 }}
{{- if and $.Values.global.smtp.enabled $.Values.global.smtp.authentication }}
- secret:
name: {{ .Values.global.smtp.password.secret | required "Missing required secret containing the SMTP password. Make sure to set `global.smtp.password.secret`" }}
items:
- key: {{ .Values.global.smtp.password.key }}
path: smtp/smtp-password
{{- end }}
- name: unicorn-secrets
emptyDir:
medium: "Memory"
- name: workhorse-secrets
emptyDir:
medium: "Memory"
- name: shared-upload-directory
emptyDir: {}
- name: shared-artifact-directory
emptyDir: {}
+ - name: shared-tmp
+ emptyDir: {}
{{ include "gitlab.certificates.volumes" . | indent 6 }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
{{- end }}
sidekiqのConfigMapにも同様に相対パスの設定を入れます。
gitlab/charts/gitlab/charts/sidekiq/templates/configmap.yaml
{{- if .Values.enabled -}}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "fullname" . }}
namespace: {{ $.Release.Namespace }}
labels:
{{ include "gitlab.standardLabels" . | indent 4 }}
data:
installation_type: |
gitlab-helm-chart
database.yml.erb: |
production:
adapter: postgresql
encoding: unicode
database: {{ template "gitlab.psql.database" . }}
pool: 10
username: {{ template "gitlab.psql.username" . }}
password: "<%= File.read("/etc/gitlab/postgres/psql-password").strip.dump[1..-2] %>"
host: {{ template "gitlab.psql.host" . }}
port: {{ template "gitlab.psql.port" . }}
prepared_statements: {{ template "gitlab.psql.preparedStatements" . }}
# load_balancing:
# hosts:
# - host1.example.com
# - host2.example.com
{{- include "gitlab.psql.ssl.config" . | indent 6 }}
smtp_settings.rb: |
{{ include "gitlab.smtp_settings" . | indent 4 }}
resque.yml.erb: |
production:
# Redis (single instance)
url: {{ template "gitlab.redis.url" . }}
id:
gitlab.yml.erb: |
production: &base
gitlab:
host: {{ template "gitlab.gitlab.hostname" . }}
https: {{ hasPrefix "https://" (include "gitlab.gitlab.url" .) }}
+ relative_url_root: /gitlab
{{- with .Values.global.hosts.ssh }}
ssh_host: {{ . | quote }}
{{- end }}
{{- with .Values.global.appConfig }}
impersonation_enabled: {{ .enableImpersonation }}
usage_ping_enabled: {{ eq .enableUsagePing true }}
default_can_create_group: {{ eq .defaultCanCreateGroup true }}
username_changing_enabled: {{ eq .usernameChangingEnabled true }}
issue_closing_pattern: {{ .issueClosingPattern | quote }}
default_theme: {{ .defaultTheme }}
{{- include "gitlab.appConfig.defaultProjectsFeatures.configuration" $ | nindent 8 }}
webhook_timeout: {{ .webhookTimeout }}
{{- end }}
trusted_proxies:
{{- if .Values.trusted_proxies }}
{{ toYaml .Values.trusted_proxies | indent 10 }}
{{- end }}
time_zone: {{ .Values.global.time_zone | quote }}
email_from: {{ template "gitlab.email.from" . }}
email_display_name: {{ .Values.global.email.display_name | quote }}
email_reply_to: {{ template "gitlab.email.reply_to" . }}
email_subject_suffix: {{ .Values.global.email.subject_suffix | quote }}
{{- with .Values.global.appConfig }}
{{- if eq .incomingEmail.enabled true }}
{{ include "gitlab.appConfig.incoming_email" . | indent 6 }}
{{- end }}
gravatar:
plain_url: {{ .gravatar.plainUrl }}
ssl_url: {{ .gravatar.sslUrl }}
{{- include "gitlab.appConfig.cronJobs" . | nindent 6 }}
{{ include "gitlab.appConfig.extra" . | indent 6 }}
{{- end }}
{{- include "gitlab.appConfig.artifacts.configuration" (dict "config" $.Values.global.appConfig.artifacts "context" $) | nindent 6 }}
{{- include "gitlab.appConfig.lfs.configuration" (dict "config" $.Values.global.appConfig.lfs "context" $) | nindent 6 }}
{{- include "gitlab.appConfig.uploads.configuration" (dict "config" $.Values.global.appConfig.uploads "context" $) | nindent 6 }}
{{- include "gitlab.appConfig.packages.configuration" (dict "config" $.Values.global.appConfig.packages "context" $) | nindent 6 }}
{{- include "gitlab.appConfig.external_diffs.configuration" (dict "config" $.Values.global.appConfig.externalDiffs "context" $) | nindent 6 }}
{{- include "gitlab.appConfig.pseudonymizer.configuration" $ | nindent 6 }}
pages:
enabled: false
mattermost:
enabled: false
## Registry Integration
{{- include "gitlab.appConfig.registry.configuration" $ | nindent 6 }}
gitlab_ci:
{{- include "gitlab.appConfig.ldap.configuration" $ | nindent 6 }}
{{- include "gitlab.appConfig.omniauth.configuration" $ | nindent 6 }}
kerberos:
enabled: false
shared:
{{ include "gitlab.appConfig.gitaly" . | indent 6 }}
{{ include "gitlab.appConfig.repositories" . | indent 6 }}
backup:
path: "tmp/backups" # Relative paths are relative to Rails.root (default: tmp/backups/)
{{ include "gitlab.appConfig.shell" . | indent 6 }}
workhorse:
git:
bin_path: /usr/bin/git
webpack:
monitoring:
ip_whitelist:
- 127.0.0.0/8
sidekiq_exporter:
{{- if .Values.metrics.enabled }}
enabled: true
address: 0.0.0.0
port: {{ .Values.metrics.port }}
{{- end }}
configure: |
{{- include "gitlab.scripts.configure.secrets" (dict "required" "gitaly registry postgres rails-secrets") | nindent 4 -}}
{{- include "gitlab.psql.ssl.initScript" . | nindent 4 }}
# Leave this here - This line denotes end of block to the parser.
{{- end }}
gitalyのConfigMapにも相対パスの設定を入れます。
gitlab/charts/gitlab/charts/gitaly/templates/configmap.yaml
{{- if .Values.enabled -}}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "fullname" . }}
namespace: {{ $.Release.Namespace }}
labels:
{{ include "gitlab.standardLabels" . | indent 4 }}
data:
configure: |
set -e
mkdir -p /init-secrets/gitaly /init-secrets/shell
cp -v -r -L /init-config/.gitlab_shell_secret /init-secrets/shell/.gitlab_shell_secret
cp -v -r -L /init-config/gitaly_token /init-secrets/gitaly/gitaly_token
{{- if .Values.global.redis.password.enabled }}
mkdir -p /init-secrets/redis
cp -v -r -L /init-config/redis_password /init-secrets/redis/redis_password
{{- end }}
config.toml.erb: |
# The directory where Gitaly's executables are stored
bin_dir = "/usr/local/bin"
# listen on a TCP socket. This is insecure (no authentication)
listen_addr = "0.0.0.0:8075"
# If metrics collection is enabled, inform gitaly about that
{{- if .Values.metrics.enabled }}
prometheus_listen_addr = "localhost:{{ .Values.metrics.metricsPort }}"
{{- end }}
<% @storages = [ {{- range (coalesce .Values.internal.names .Values.global.gitaly.internal.names) }} {{ . | quote }}, {{- end }} ] %>
<% @index=`echo ${HOSTNAME##*-}`.to_i %>
<% if @storages.length > @index %>
[[storage]]
name = "<%= @storages[@index] %>"
path = "/home/git/repositories"
<% else %>
<% raise Exception, "Storage for node #{@index} is not present in the storageNames array. Did you use kubectl to scale up ? You need to solely use helm for this purpose" %>
<% end %>
[logging]
{{- with .Values.logging }}
{{- if .level }}
level = "{{ .level }}"
{{- end }}
{{- if .format }}
format = "{{ .format }}"
{{- end }}
{{- if .sentryDsn }}
sentry_dsn = "{{ .sentryDsn }}"
{{- end }}
{{- if .rubySentryDsn }}
ruby_sentry_dsn = "{{ .rubySentryDsn }}"
{{- end }}
{{- if .sentryEnvironment }}
sentry_environment = "{{ .sentryEnvironment }}"
{{- end }}
{{- end }}
{{- if .Values.prometheus.grpcLatencyBuckets }}
[prometheus]
grpc_latency_buckets = {{ .Values.prometheus.grpcLatencyBuckets }}
{{- end }}
[auth]
token = "<%= File.read('/etc/gitlab-secrets/gitaly/gitaly_token').strip.dump[1..-2] %>"
[git]
{{- with .Values.git }}
{{- if .catFileCacheSize }}
catfile_cache_size = {{ .catFileCacheSize }}
{{- end }}
{{- end }}
[gitaly-ruby]
# The directory where gitaly-ruby is installed
dir = "/srv/gitaly-ruby"
{{- with .Values.ruby }}
{{- if .maxRss }}
max_rss = {{ .maxRss }}
{{- end }}
{{- if .gracefulRestartTimeout }}
graceful_restart_timeout = "{{ .gracefulRestartTimeout }}"
{{- end }}
{{- if .restartDelay }}
restart_delay = "{{ .restartDelay }}"
{{- end }}
{{- if .numWorkers }}
num_workers = {{ .numWorkers }}
{{- end }}
{{- end }}
[gitlab-shell]
# The directory where gitlab-shell is installed
dir = "/srv/gitlab-shell"
{{- if .Values.shell.concurrency }}
{{- range .Values.shell.concurrency }}
{{- if and .rpc .maxPerRepo }}
[[concurrency]]
rpc = "{{ .rpc }}"
max_per_repo = {{ .maxPerRepo }}
{{- end }}
{{- end }}
{{- end }}
shell-config.yml.erb: |
# GitLab user. git by default
user: git
# Url to gitlab instance. Used for api calls. Should end with a slash.
- gitlab_url: "http://{{ template "gitlab.unicorn.host" . }}:{{ default 8080 .Values.unicorn.port }}/"
+ gitlab_url: "http://{{ template "gitlab.unicorn.host" . }}:{{ default 8080 .Values.unicorn.port }}/gitlab/"
secret_file: /etc/gitlab-secrets/shell/.gitlab_shell_secret
http_settings:
self_signed_cert: false
# File used as authorized_keys for gitlab user
auth_file: "/home/git/.ssh/authorized_keys"
# Redis settings used for pushing commit notices to gitlab
redis:
host: {{ template "gitlab.redis.host" . }}
port: {{ template "gitlab.redis.port" . }}
{{- if .Values.global.redis.password.enabled }}
pass: "<%= File.read("/etc/gitlab-secrets/redis/redis_password").strip.dump[1..-2] %>"
{{- end }}
database: nil
namespace: resque:gitlab
# Log file.
# Default is gitlab-shell.log in the root directory.
log_file: "/var/log/gitaly/gitlab-shell.log"
# Log level. INFO by default
log_level: INFO
# Audit usernames.
# Set to true to see real usernames in the logs instead of key ids, which is easier to follow, but
# incurs an extra API call on every gitlab-shell command.
audit_usernames: false
# Leave this here - This line denotes end of block to the parser.
{{- end }}
values
XXXXXXXXXX.ap-northeast-1.elb.amazonaws.comはALBのエンドポイント、YYYYYYYYYY.ap-northeast-1.rds.amazonaws.comはRDSのエンドポイントとなります。
IngressはNGINX Ingress Controllerに読み込ませます。
IAM Roleは事前に作成したものです。
gitlab/my-values.yaml
global:
edition: ce
hosts:
https: false
gitlab:
name: XXXXXXXXXX.ap-northeast-1.elb.amazonaws.com
https: false
ingress:
configureCertmanager: false
enabled: false
tls:
enabled: false
psql:
password:
secret: gitlab-postgresql
key: password
host: YYYYYYYYYY.ap-northeast-1.rds.amazonaws.com
port: 5432
username: gitlab
database: gitlabhq_production
minio:
enabled: false
appConfig:
lfs:
bucket: test-gitlab-lfs
connection:
secret: gitlab-rails-storage
key: connection
artifacts:
bucket: test-gitlab-artifacts
connection:
secret: gitlab-rails-storage
key: connection
uploads:
bucket: test-gitlab-uploads
connection:
secret: gitlab-rails-storage
key: connection
packages:
bucket: test-gitlab-packages
connection:
secret: gitlab-rails-storage
key: connection
externalDiffs:
enabled: true
bucket: test-gitlab-mr-diffs
connection:
secret: gitlab-rails-storage
key: connection
pseudonymizer:
configMap:
bucket: test-gitlab-pseudo
connection:
secret: gitlab-rails-storage
key: connection
backups:
bucket: test-gitlab-backups
tmpBucket: test-gitlab-tmp
time_zone: Tokyo
upgradeCheck:
enabled: false
certmanager:
install: false
nginx-ingress:
enabled: false
prometheus:
install: false
postgresql:
install: false
gitlab-runner:
install: false
registry:
minReplicas: 1
storage:
secret: registry-storage
key: config
annotations:
iam.amazonaws.com/role: s3-full-access-role
ingress:
enabled: false
gitlab:
gitlab-shell:
enabled: false
gitlab-exporter:
enabled: false
unicorn:
annotations:
iam.amazonaws.com/role: s3-full-access-role
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
sidekiq:
annotations:
iam.amazonaws.com/role: s3-full-access-role
task-runner:
enabled: false
GitLabのデプロイ
次のコマンドでGitLabをデプロイします。
$ helm install ¥
--name test-gitlab ¥
--namespace test ¥
--values ./gitlab/my-values.yaml ¥
./gitlab
確認
/gitlabにアクセスして問題ないことが確認できます。
GitLab RunnerをデプロイしておけばGitLab CIを実行することも可能です。
まとめ
EKSにHelmでGitLabをデプロイするときでも相対URLを使用できることを確認しました。
assetsに関してはNGINX Ingress Controllerのrewrite targetで回避せずとも解決できる方法を模索しましたが、すぐに対処することが厳しそうでした。
今回の対応ではtemplatesの中身を変更することになるので、バージョンアップの際には都度対応が必要ということに注意してください。
参考
https://gitlab.com/gitlab-org/charts/gitlab/-/issues/406
https://docs.gitlab.com/ee/install/relative_url.html
https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1647