ArgoCDのYamlをうまく管理する方法

課題

• ダウンロードしてきたYamlを変更する必要がある
  • Deploymentの設定
  • Endpoint
  • Auth
  • RBAC
• 変更するとVersion Upgrade時に同じ変更をまた手動でやるのはキツイ

基本方針: Kustomizeで管理

• BaseでVersion指定
• overlaysで必要に応じていじる
  • Deploymentをいじるけい
  • ELBにする
  • Ingressにする
  • Google OAuthを使う
  • RBACの設定
  • など

参照: https://argoproj.github.io/argo-cd/operator-manual/declarative-setup/#manage-argo-cd-using-argo-cd

例:

kustomization.yaml では resourcesとpathcesStrategicMergeを使い分ける (SecretをRepoにいれたくなかったので今回は、SealedSecretを利用)

kustomization.yaml

namespace: argocd
bases:
  - github.com/argoproj/argo-cd//manifests/cluster-install?ref=v1.6.0
patchesStrategicMerge:
  - argocd-notifications-cm.yaml
  - argocd-rbac-cm.yaml
  - argocd-cm.yaml
  - argocd-repo-server-deployment.yaml
  - argocd-dex-server-deployment.yaml
resources:
  - argocd-notifications-sealedsecret.yaml
  - argocd-google-oauth-sealedsecret.yaml
  - ingress.yaml

こうしておけば、Applyは以下のコマンドでいい

kubectl apply -k .

カスタム設定

1. nodeSelector, priorityClassNameなど自分たちのKubernetesクラスタ上での管理設定をDeploymentに追加

ArgoCDに限らず普通のKustomizeの設定なので、Deploymentに対してOverlaysを準備してあげる

例: nodeSelectorを追加して挙げる例

argocd-server-deploymemt.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-server
spec:
  template:
    spec:
      containers:
        - name: argocd-server
          nodeSelector:
            disktype: ssd

kustomization.yaml

 namespace: argocd
 bases:
   - github.com/argoproj/argo-cd//manifests/cluster-install?ref=v1.6.0
 patchesStrategicMerge:
   - ...
+  - argocd-server-deploymemt.yaml

2. ELBを使う (AWSの話)

ELBかIngressどっちかでいい

serviceを追加して、kustomization.yamlに入れるだけでいい

argocd-server-svc.yaml

apiVersion: v1
kind: Service
metadata:
    labels:
      app.kubernetes.io/component: server
      app.kubernetes.io/name: argocd-server
      app.kubernetes.io/part-of: argocd
    name: argocd-server
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:ap-northeast-1:xxxxxxxx:certificate/xxxxxxxxxxxxxxxxxxxxxxxxxx # httpsにする場合
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "5"
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "2"
spec:
    type: LoadBalancer
    ports:
    - name: http
        port: 80
        protocol: TCP
        targetPort: 8080
    - name: https
        port: 443
        protocol: TCP
        targetPort: 8080
    selector:
      app.kubernetes.io/name: argocd-server
    loadBalancerSourceRanges:
      - xxx.xxx.xxx.xxx/32 # IP制限したい場合

kustomization.yaml

 namespace: argocd
 bases:
   - github.com/argoproj/argo-cd//manifests/cluster-install?ref=v1.6.0
 patchesStrategicMerge:
   - ...
+  - argocd-server-svc.yaml

3. Ingressにする

参照: https://argoproj.github.io/argo-cd/operator-manual/ingress/
ELBかIngressどっちかでいい

• prerequisite: すでにingress controllerとexternal-dnsが存在
• 以下の2つのファイルをkustomization.yamlに追加する
• ハマリポイント: SSL終端をELBでやると TOO_MANY_REDIRECTS 問題に落ちいいるので、 insecure と force-ssl-redirect を false にした

ingress.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  name: argocd
spec:
  rules:
  - host: argocd.example.com
    http:
      paths:
      - backend:
          serviceName: argocd-server
          servicePort: http

argocd-server-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-server
spec:
  template:
    spec:
      containers:
        - name: argocd-server
          command:
            - argocd-server
            - --staticassets
            - /shared/app
            - --insecure # これが必要になる

4. Google Oauthを使う

参照: https://argoproj.github.io/argo-cd/operator-manual/user-management/google/

GOOGLE_CLIENT_ID と GOOGLE_CLIENT_SECRET は取得しておく

argocd-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
  namespace: argocd
data:
  url: https://argocd.example.com

  dex.config: |
    connectors:
    - type: oidc
      id: google
      name: Google
      config:
        # Canonical URL of the provider, also used for configuration discovery.
        # This value MUST match the value returned in the provider config discovery.
        # See: https://openid.net/specs/openid-connect-discovery-1_0.html#ProviderConfig
        issuer: https://accounts.google.com

        # Connector config values starting with a "$" will read from the environment.
        clientID: $GOOGLE_CLIENT_ID
        clientSecret: $GOOGLE_CLIENT_SECRET

        # Dex's issuer URL + "/callback"
        redirectURI: https://argocd.example.com/api/dex/callback

        # Google supports whitelisting allowed domains when using G Suite
        # (Google Apps). The following field can be set to a list of domains
        # that can log in:
        hostedDomains:
          - example.com

以下のsealedsecret.yamlはargocd-google-oauth-secret.yaml作成後に以下のコマンドで作成

kubeseal -o yaml < argocd-google-oauth-secret.yaml > argocd-google-oauth-sealedsecret.yaml

argocd-google-oauth-sealedsecret.yaml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: argocd-google-oauth-secret
  namespace: argocd
spec:
  encryptedData:
    GOOGLE_CLIENT_ID: xxxxxxxxxxxxxxx
    GOOGLE_CLIENT_SECRET: xxxxxxxxxxxxxx
  template:
    metadata:
      creationTimestamp: null
      name: argocd-google-oauth-secret
      namespace: argocd
    type: Opaque

argocd-dex-server-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-dex-server
spec:
  template:
    spec:
      containers:
      - name: dex
        envFrom:
          - secretRef:
              name: argocd-google-oauth-secret

kustomization.yaml

 namespace: argocd
 bases:
   - github.com/argoproj/argo-cd//manifests/cluster-install?ref=v1.6.0
 patchesStrategicMerge:
+  - argocd-cm.yaml
+  - argocd-dex-server-deployment.yaml
 resources:
+  - argocd-google-oauth-sealedsecret.yaml

5. RBACを使う

参照: https://argoproj.github.io/argo-cd/operator-manual/rbac/

argocd-rbac-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-rbac-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  scopes: '[email]' # google oauth使ってる場合
  policy.csv: |
    p, role:org-admin, applications, *, */*, allow
    p, role:org-admin, clusters, get, *, allow
    p, role:org-admin, repositories, get, *, allow
    p, role:org-admin, repositories, create, *, allow
    p, role:org-admin, repositories, update, *, allow
    p, role:org-admin, repositories, delete, *, allow

    g, [e-mail], role:org-admin

kustomization.yaml

 namespace: argocd
 bases:
   - github.com/argoproj/argo-cd//manifests/cluster-install?ref=v1.6.0
 patchesStrategicMerge:
+  - argocd-rbac-cm.yaml

Upgrade version

1. kustomization.yamlで参照してるversionを変更
2. 必要に応じてdiffチェック

   kubectl diff -k .
   

3. apply

   kubectl apply -k .