More than 3 years have passed since last update.

株式会社ビービット

KubernetesのHPA with custom metricsをRabbitMQとPrometheusで試す

Last updated at 2021-05-19Posted at 2021-04-03

やりたいこと

Queueにメッセージが溜まったときに、ConsumerのPodの数を増やして処理速度をあげ、Queueにメッセージがなくなったときに、Podの数を減らしてリソースを開放したい。

考慮したこと

HPAでCPUやメモリの使用量でスケール方法もあるが、直接的に待ってるQueueの数を元にスケールしてみたい
HPA with custom metricsの練習
VPAは、PodがRestartしてしまうので、長い処理をするアプリケーションの場合、Restartするときに途中までの処理が無駄になってしまう

材料

方針

RabbitMQの messages_ready メトリクスをPrometheusで取得する
prometheus-adapter で custom-metricsでRabbitMQのMetricsを取得できるようにする
HPAを設定する

Code

https://github.com/nakamasato/kubernetes-training/tree/master/autoscaler/hpa/custom-metrics

ステップ

準備

1. Prometheusをデプロイ

prometheus-operatorをインストール

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml

monitoring namespaceを作成
```
kubectl create ns monitoring
```

Prometheusをデプロイ

kubectl apply -k ../../../prometheus-operator -n monitoring

http://localhost:30900 でUIをチェック

2. RabbitMQをデプロイ

RabbitMQ operatorをインストール

kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml

RabbitMQをデプロイ

rabbitmq/rabbitmq-cluster.yaml

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq

kubectl apply -f rabbitmq/rabbitmq-cluster.yaml

PodMonitorをデプロイ (RabbitMQをPrometheusがScrapeできるようにする)

rabbitmq/pod-monitor-rabbitmq.yaml

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  namespace: monitoring
  name: rabbitmq
spec:
  podMetricsEndpoints:
  - interval: 15s
    port: prometheus
    path: /metrics
  selector:
    matchLabels:
      app.kubernetes.io/component: rabbitmq
  namespaceSelector:
    any: true

kubectl apply -f rabbitmq/pod-monitor-rabbitmq.yaml

3. RabbitMQ producerをデプロイ

5分に一回20個RabbitMQのメッセージを送るCronJob

rabbitmq-producer

rabbitmq-producer-cronjob.yaml


apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: rabbitmq-producer
spec:
  schedule: '*/5 * * * *'
  successfulJobsHistoryLimit: 1
  jobTemplate:
    metadata:
      name: rabbitmq-producer
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - image: nakamasato/rabbitmq-producer
              name: rabbitmq-producer
              env:
                - name: RABBITMQ_USERNAME
                  valueFrom:
                    secretKeyRef:
                      name: rabbitmq-default-user
                      key: username
                - name: RABBITMQ_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: rabbitmq-default-user
                      key: password
                - name: RABBITMQ_HOST
                  value: rabbitmq
                - name: NUM_OF_MESSAGES
                  value: "20"

kubectl apply -f rabbitmq-producer-cronjob.yaml

4. RabbitMQ consumerをデプロイ

RabbitMQのメッセージを一つずつConsumeし、1つのメッセージにつき10秒で処理する。

rabbitmq-consumer

rabbitmq-consumer-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: rabbitmq-consumer
  name: rabbitmq-consumer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq-consumer
  template:
    metadata:
      labels:
        app: rabbitmq-consumer
    spec:
      containers:
        - image: nakamasato/rabbitmq-consumer
          name: rabbitmq-consumer
          env:
            - name: RABBITMQ_USERNAME
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-default-user
                  key: username
            - name: RABBITMQ_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-default-user
                  key: password
            - name: RABBITMQ_HOST
              value: rabbitmq
            - name: PROCESS_SECONDS
              value: "10"

kubectl apply -f rabbitmq-consumer-deployment.yaml

5. Grafana

(GrafanaはDashboardでMetricsを確認するためだけなので、必須ではない。)

grafana-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      containers:
        - name: grafana
          image: grafana/grafana:latest
          ports:
            - name: grafana
              containerPort: 3000
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-storage
      volumes:
        - name: grafana-storage
          emptyDir: {}

grafana-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  selector:
    app: grafana
  type: NodePort
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 32111

Grafanaをデプロイ

kubectl apply -f grafana-deployment.yaml,grafana-service.yaml

http://localhost:32111 に usernameとpasswordをともにadminでログインする
RabbitMQ-Overview Dashboard (10991)をimportする

本題

1. prometheus-adapterをデプロイ

prometheus-adapterをclone

git clone git@github.com:stefanprodan/k8s-prom-hpa.git && cd k8s-prom-hpa

certificateを準備

touch metrics-ca.key metrics-ca.crt metrics-ca-config.json
make certs

デプロイ
```
kubectl create -f ./custom-metrics-api
```

RabbitMQのMetricsがcustom metricsで取得できることを確認

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/rabbitmq_queue_messages_ready"| jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/rabbitmq_queue_messages_ready"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "rabbitmq-server-0",
        "apiVersion": "/v1"
      },
      "metricName": "rabbitmq_queue_messages_ready",
      "timestamp": "2021-03-27T12:01:15Z",
      "value": "1274"
    }
  ]
}

3. HorizontalPodAutoscalerをデプロイ

rabbitmq-consumer-hpa.yaml

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: rabbitmq-consumer
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rabbitmq-consumer
  minReplicas: 1
  maxReplicas: 20
  metrics:
    - type: Object
      object:
        metric:
          name: rabbitmq_queue_messages_ready
        describedObject:
          kind: Pod
          name: rabbitmq-server-0
          apiVersion: v1
        target:
          type: Value
          averageValue: 1

kubectl apply -f rabbitmq-consumer-hpa.yaml

4. Podの数がRabbitMQのQueueの数によって変化しているのをGrafanaで確認

Todo

HPAScaleToZero: 1.16で追加されたHPAでReplicaを0までできるAlphaのFeature (https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates)

You get articles that match your needs
You can efficiently read back useful information
You can use dark theme

What you can do with signing up