LoginSignup
1
2

More than 1 year has passed since last update.

KubernetesAdvent Calendar 2021Day 16
@riita10069(Ryota Yamada)

NATS JetStreamをKubernetes上で動かしてみた

Last updated at Posted at 2021-12-08

NATS JetStreamについて

概要

NATS JetStreamのドキュメントには、Kubernetes上で動かす方法に関して特に記載がなく、
あるとしたら、Helm Chartを使ってね。という感じのことが書いてあるだけ。
なので、Helm Chartを解読して、シンプルなyamlで記述できるようにしました。

なぜ、Helm Chartを使わないのか

  • ブラックボックスのままでは、拡張性に乏しい
  • Helmが黒魔術化する
  • Helmが宣言的ではない
    • helm install --set や helm install --valuesといった手続き的なCLI操作を求められる

yaml

apiVersion: v1
kind: Namespace
metadata:
  name: nats-ns
  labels:
    app.kubernetes.io/name: nats
    app.kubernetes.io/instance: nats-js
    app.kubernetes.io/version: 2.6.2
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nats-js-config
  namespace: nats-ns
  labels:
    app.kubernetes.io/name: nats
    app.kubernetes.io/instance: nats-js
    app.kubernetes.io/version: 2.6.2
data:
  nats.conf: |
    pid_file: "/var/run/nats/nats.pid"
    http: 8222
    server_name: $POD_NAME

    jetstream {
      store_dir: "/data/jetstream/store"
      max_file_store: 1G
      max_memory_store: 1G
    }

    cluster {
      name: example
      port: 6222

      # Based on 5 replicas
      routes = [
        nats-route://$POD_NAME-0.nats-js.$POD_NAMESPACE.svc.cluster.local:6222,
        nats-route://$POD_NAME-1.nats-js.$POD_NAMESPACE.svc.cluster.local:6222,
        nats-route://$POD_NAME-2.nats-js.$POD_NAMESPACE.svc.cluster.local:6222,
        nats-route://$POD_NAME-3.nats-js.$POD_NAMESPACE.svc.cluster.local:6222,
        nats-route://$POD_NAME-4.nats-js.$POD_NAMESPACE.svc.cluster.local:6222,
      ]
      cluster_advertise: $CLUSTER_ADVERTISE
      connect_retries: 30
    }

    lame_duck_duration: 60s # it should be equal to terminationGracePeriodSeconds of Pod and default is 30s

    }
---
apiVersion: v1
kind: Service
metadata:
  name: nats-js
  namespace: nats-ns
  labels:
    app.kubernetes.io/name: nats
    app.kubernetes.io/instance: nats-js
    app.kubernetes.io/version: 2.6.2
spec:
  selector:
    app.kubernetes.io/name: nats
    app.kubernetes.io/instance: nats-js
  clusterIP: None
  ports:
  - name: client
    port: 4222
  - name: cluster
    port: 6222
  - name: monitor
    port: 8222
  - name: metrics
    port: 7777
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nats-js
  namespace: nats-ns
  labels:
    app.kubernetes.io/name: nats
    app.kubernetes.io/instance: nats-js
    app.kubernetes.io/version: 2.6.2
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: nats
      app.kubernetes.io/instance: nats-js
  replicas: 5
  serviceName: "nats-js"
  volumeClaimTemplates:
  - metadata:
      name: nats-js-sts-vol
    spec:
      accessModes:
      - ReadWriteOnce
      volumeMode: "Filesystem"
      resources:
        requests:
          storage: 10Gi
  template:
    metadata:
      labels:
        app.kubernetes.io/name: nats
        app.kubernetes.io/instance: nats-js
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "7777"
        prometheus.io/scrape: "true"

        sidecar.istio.io/inject: "false" # should considerate
    spec:
      # Common volumes for the containers
      volumes:
      - name: config-volume
        configMap:
          name: nats-js-config
      - name: pid
        emptyDir: {}

      # Required to be able to HUP signal and apply config reload
      # to the server without restarting the pod.
      shareProcessNamespace: true

      #################
      #               #
      #  NATS Server  #
      #               #
      #################
      terminationGracePeriodSeconds: 60
      containers:
      - name: nats
        image: synadia/nats-server:nightly
        ports:
        - containerPort: 4222
          name: client
        - containerPort: 6222
          name: cluster
        - containerPort: 8222
          name: monitor
        - containerPort: 7777
          name: metrics
        command:
         - "nats-server"
         - "--config"
         - "/etc/nats-config/nats.conf"

        # Required to be able to define an environment variable
        # that refers to other environment variables.  This env var
        # is later used as part of the configuration file.
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: CLUSTER_ADVERTISE
          value: $(POD_NAME).nats-js.$(POD_NAMESPACE).svc
        volumeMounts:
          - name: config-volume
            mountPath: /etc/nats-config
          - name: pid
            mountPath: /var/run/nats
          - name: nats-js-sts-vol
            mountPath: /data/jetstream

        # Liveness/Readiness probes against the monitoring
        #
        livenessProbe:
          httpGet:
            path: /
            port: 8222
          initialDelaySeconds: 10
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /
            port: 8222
          initialDelaySeconds: 10
          timeoutSeconds: 5

        resources:
          requests:
            cpu: 300m
            memory: 600Mi
          limits:
            cpu: 300m
            memory: 600Mi

        # Gracefully stop NATS Server on pod deletion or image upgrade.
        #
        lifecycle:
          preStop:
            exec:
              # Using the alpine based NATS image, we add an extra sleep that is
              # the same amount as the terminationGracePeriodSeconds to allow
              # the NATS Server to gracefully terminate the client connections.
              #
              command: ["/bin/sh", "-c", "/nats-server -sl=ldm=/var/run/nats/nats.pid && /bin/sleep 60"]

これを
kubectl apply -f nats.yamlでApplyすると。

log
└─(17:03:54)──> kubectl logs -f nats-js-0  -c nats  --namespace nats-ns                                                                          ──(火,1207)─┘
[8] 2021/12/07 08:03:49.354395 [INF] Starting nats-server
[8] 2021/12/07 08:03:49.355813 [INF]   Version:  2.6.6
[8] 2021/12/07 08:03:49.355852 [INF]   Git:      [893b4154]
[8] 2021/12/07 08:03:49.355907 [INF]   Name:     nats-js-0
[8] 2021/12/07 08:03:49.356005 [INF]   Node:     yLCaJyhQ
[8] 2021/12/07 08:03:49.356124 [INF]   ID:       NAZZFGATIGMG57CFFJXGXNJPE3QBIM5ZARZWFZKPODA7BYGFCAGJBJKV
[8] 2021/12/07 08:03:49.356239 [INF] Using configuration file: /etc/nats-config/nats.conf
[8] 2021/12/07 08:03:49.446026 [INF] Starting JetStream
[8] 2021/12/07 08:03:49.451437 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[8] 2021/12/07 08:03:49.451678 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[8] 2021/12/07 08:03:49.451711 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[8] 2021/12/07 08:03:49.451726 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[8] 2021/12/07 08:03:49.451896 [INF] 
[8] 2021/12/07 08:03:49.452182 [INF]          https://docs.nats.io/jetstream
[8] 2021/12/07 08:03:49.452206 [INF] 
[8] 2021/12/07 08:03:49.452218 [INF] ---------------- JETSTREAM ----------------
[8] 2021/12/07 08:03:49.455665 [INF]   Max Memory:      953.67 MB
[8] 2021/12/07 08:03:49.456175 [INF]   Max Storage:     953.67 MB
[8] 2021/12/07 08:03:49.456358 [INF]   Store Directory: "/data/jetstream/store/jetstream"
[8] 2021/12/07 08:03:49.456399 [INF] -------------------------------------------
[8] 2021/12/07 08:03:49.459252 [INF] Starting JetStream cluster
[8] 2021/12/07 08:03:49.459362 [INF] Creating JetStream metadata controller
[8] 2021/12/07 08:03:49.545305 [INF] JetStream cluster recovering state
[8] 2021/12/07 08:03:49.656808 [INF] Starting http monitor on 0.0.0.0:8222
[8] 2021/12/07 08:03:49.657967 [INF] Listening for client connections on 0.0.0.0:4222
[8] 2021/12/07 08:03:49.744362 [INF] Server is ready
[8] 2021/12/07 08:03:49.745196 [INF] Cluster name is example
[8] 2021/12/07 08:03:49.745468 [INF] Listening for route connections on 0.0.0.0:6222
[8] 2021/12/07 08:03:49.754961 [ERR] Error trying to connect to route (attempt 1): lookup for host "$POD_NAME-1.nats-js.$POD_NAMESPACE.svc.cluster.local": lookup $POD_NAME-1.nats-js.$POD_NAMESPACE.svc.cluster.local: no such host
[8] 2021/12/07 08:03:49.754984 [ERR] Error trying to connect to route (attempt 1): lookup for host "$POD_NAME-4.nats-js.$POD_NAMESPACE.svc.cluster.local": lookup $POD_NAME-4.nats-js.$POD_NAMESPACE.svc.cluster.local: no such host
[8] 2021/12/07 08:03:49.755098 [ERR] Error trying to connect to route (attempt 1): lookup for host "$POD_NAME-0.nats-js.$POD_NAMESPACE.svc.cluster.local": lookup $POD_NAME-0.nats-js.$POD_NAMESPACE.svc.cluster.local: no such host
[8] 2021/12/07 08:03:49.755142 [ERR] Error trying to connect to route (attempt 1): lookup for host "$POD_NAME-2.nats-js.$POD_NAMESPACE.svc.cluster.local": lookup $POD_NAME-2.nats-js.$POD_NAMESPACE.svc.cluster.local: no such host
[8] 2021/12/07 08:03:49.755243 [ERR] Error trying to connect to route (attempt 1): lookup for host "$POD_NAME-3.nats-js.$POD_NAMESPACE.svc.cluster.local": lookup $POD_NAME-3.nats-js.$POD_NAMESPACE.svc.cluster.local: no such host

起動時にクラスタが見つからないエラーを返している。
当然で、他のクラスタは後から立ち上がってくるから

└─(17:14:44)──> kubectl get pods --namespace nats-ns                                                                                       130 ↵ ──(火,1207)─┘
NAME        READY   STATUS    RESTARTS   AGE
nats-js-0   1/1     Running   0          11m
nats-js-1   1/1     Running   0          10m
nats-js-2   1/1     Running   0          10m
nats-js-3   1/1     Running   0          10m
nats-js-4   1/1     Running   0          10m

時間が経って、全てのPodが立ち上がれば、問題はない。
本番デプロイの際はこの点を留意したい。

実際に動くかどうかを確認したい。

まずは、ポートフォワードする
└─(17:22:59)──> kubectl port-forward svc/nats-js 4222:4222  --namespace nats-ns                                                              1 ↵ ──(火,1207)─┘
Forwarding from 127.0.0.1:4222 -> 4222
Forwarding from [::1]:4222 -> 4222
次に、subscribeする
└─(16:38:58)──> nats sub sample                                                                                                                                ──(火,1207)─┘
17:24:28 Subscribing on sample
[#1] Received on "sample"
hello nats world
Publish する
└─(16:38:58)──> nats pub sample "hello nats world"                                                                                         127 ↵ ──(火,1207)─┘
17:24:37 Published 16 bytes to "sample"
様子を見る
└─(16:38:58)──> nats sub sample                                                                                                                                ──(火,1207)─┘
17:24:28 Subscribing on sample
[#1] Received on "sample"
hello nats world
1
2
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
1
2