Kubernetes HPAの Alpha Feature HPAScaleToZeroをローカルで試す

背景

HPAではゼロにScaleIn出来ない。

kubectl explain HorizontalPodAutoscaler.spec.minReplicas
KIND:     HorizontalPodAutoscaler
VERSION:  autoscaling/v1

FIELD:    minReplicas <integer>

DESCRIPTION:
     minReplicas is the lower limit for the number of replicas to which the
     autoscaler can scale down. It defaults to 1 pod. minReplicas is allowed to
     be 0 if the alpha feature gate HPAScaleToZero is enabled and at least one
     Object or External metric is configured. Scaling is active as long as at
     least one metric value is available.

minReplicas is allowed to
be 0 if the alpha feature gate HPAScaleToZero is enabled and at least one
Object or External metric is configured.

• alpha feature gateを enableする必要がある
• Object or External metric少なくとも1つは必要

KubernetesのHPA with custom metricsをRabbitMQとPrometheusで試す で紹介したHPA with custom metricsのようなケースだと、Queueのメッセージがゼロのときは完全にスケールインしたい場合もある。今回は、HPAで0にまでスケールインを試す。

Feature Gateとは

feature-gates: 各Kubernetes FeatureのStageとDefaultでEnableされているか、どのバージョンから(どのバージョンまで)使用可能かがテーブルになっている。

Enableするには、 -feature-gates="...,DynamicKubeletConfig=true" 起動時のコマンドにEnableするFeatureペア (Feature名とtrue/false) をリストする必要がある

今回使いたいHPAScaleToZeroを見てみると、 HPAScaleToZero false Alpha 1.16 となっており、使用するためには明示的にenableする必要がある。

ローカルでHPAScaleToZero をEnableにしたKubernetesクラスタを作成

今回はローカル挙動の確認をしたいのでローカルで指定が簡単にできそうな kind を使用

Prerequisite

• go (1.11+)
• Docker

kindのインストール

Macの場合は、brewを使ってインストールできる (詳細は https://kind.sigs.k8s.io/docs/user/quick-start/#installation)

brew install kind

kindのクラスタ用のYamlを準備

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  # any feature gate can be enabled here with "Name": true
  # or disabled here with "Name": false
  # not all feature gates are tested, however
  "HPAScaleToZero": true

kindでKubernertes Clusterを作成

kind create cluster --config cluster-with-alpha-feature.yaml

kind create cluster --config cluster-with-alpha-feature.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.20.2) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊

HPAのための準備

前回と同じものをDeploy

（今回はGrafanaなどは省略）

1. Code をclone

   git clone https://github.com/nakamasato/kubernetes-training && cd kubernetes-training/autoscaler/hpa/custom-metrics
   

2. Prometheus with operator

   kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
   kubectl create ns monitoring
   kubectl apply -k ../../../prometheus-operator -n monitoring
   

3. RabbitMQ with operator

   kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
   kubectl apply -f rabbitmq/rabbitmq-cluster.yaml
   kubectl apply -f rabbitmq/pod-monitor-rabbitmq.yaml
   

4. RabbitMQConsumer & Producer

   kubectl apply -f rabbitmq-producer-cronjob.yaml
   kubectl apply -f rabbitmq-consumer-deployment.yaml
   

5. Prometheus Adapter をDeploy(https://github.com/stefanprodan/k8s-prom-hpa/)

   git clone [e-mail]:stefanprodan/k8s-prom-hpa.git && cd k8s-prom-hpa
   touch metrics-ca.key metrics-ca.crt metrics-ca-config.json
   make certs
   kubectl create -f ./custom-metrics-api
   

6. rabbitmq_queue_messages_ready が [custom.metrics.k8s.io](http://custom.metrics.k8s.io) から取得できるのを確認

   kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/rabbitmq_queue_messages_ready"| jq .
   {
     "kind": "MetricValueList",
     "apiVersion": "custom.metrics.k8s.io/v1beta1",
     "metadata": {
       "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/rabbitmq_queue_messages_ready"
     },
     "items": [
       {
         "describedObject": {
           "kind": "Pod",
           "namespace": "default",
           "name": "rabbitmq-server-0",
           "apiVersion": "/v1"
         },
         "metricName": "rabbitmq_queue_messages_ready",
         "timestamp": "2021-05-19T01:55:02Z",
         "value": "0"
       }
     ]
   }
   

HPAをデプロイ (minReplicas: 0)

rabbitmq-consumer-hpa-with-min-zero.yaml

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: rabbitmq-consumer
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rabbitmq-consumer
  minReplicas: 0
  maxReplicas: 20
  metrics:
    - type: Object
      object:
        metric:
          name: rabbitmq_queue_messages_ready
        describedObject:
          kind: Pod
          name: rabbitmq-server-0
          apiVersion: v1
        target:
          type: Value
          averageValue: 1

kubectl apply -f rabbitmq-consumer-hpa-with-min-zero.yaml

確認 New size: 0 となってることを確認!

Events:
  Type    Reason             Age                  From                       Message
  ----    ------             ----                 ----                       -------
  Normal  SuccessfulRescale  35m                  horizontal-pod-autoscaler  New size: 7; reason: All metrics below target
  Normal  SuccessfulRescale  30m (x2 over 40m)    horizontal-pod-autoscaler  New size: 16; reason: All metrics below target
  Normal  SuccessfulRescale  30m (x2 over 39m)    horizontal-pod-autoscaler  New size: 18; reason: All metrics below target
  Normal  SuccessfulRescale  25m (x2 over 35m)    horizontal-pod-autoscaler  New size: 14; reason: All metrics below target
  Normal  SuccessfulRescale  25m                  horizontal-pod-autoscaler  New size: 4; reason: All metrics below target
  Normal  SuccessfulRescale  20m                  horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
  Normal  SuccessfulRescale  10m (x4 over 40m)    horizontal-pod-autoscaler  New size: 4; reason: external metric rabbitmq_queue_messages_ready(nil) above target
  Normal  SuccessfulRescale  5m13s (x4 over 35m)  horizontal-pod-autoscaler  New size: 2; reason: All metrics below target
  Normal  SuccessfulRescale  4m58s                horizontal-pod-autoscaler  New size: 0; reason: All metrics below target
  Normal  SuccessfulRescale  37s (x9 over 20m)    horizontal-pod-autoscaler  (combined from similar events): New size: 4; reason: external metric rabbitmq_queue_messages_ready(nil) above target
  Normal  SuccessfulRescale  22s (x4 over 40m)    horizontal-pod-autoscaler  New size: 8; reason: external metric rabbitmq_queue_messages_ready(nil) above target
  Normal  SuccessfulRescale  6s                   horizontal-pod-autoscaler  New size: 16; reason: external metric rabbitmq_queue_messages_ready(nil) above targe

片付け

1. Kubernetesリソースの片付け

   kubectl delete -f rabbitmq-consumer-hpa-with-min-zero.yaml
   kubectl delete -f rabbitmq-producer-cronjob.yaml
   kubectl delete -f rabbitmq-consumer-deployment.yaml
   kubectl delete -f https://k8s.io/examples/application/php-apache.yaml
   kubectl delete -f rabbitmq/rabbitmq-cluster.yaml
   kubectl delete -k ../../../prometheus-operator -n monitoring
   kubectl delete -f ./k8s-prom-hpa/custom-metrics-api                                                           
   kubectl delete -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
   kubectl delete -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
   

2. kindクラスタの削除

   kind delete cluster --name kind
   

まとめ

Alpha Featureの HPAScaleToZero をローカルKubernetesクラスタkindでEnableして、Custom MetricsによるHPAで minReplicas: 0 の動作を確認できた。

参考リンク

• https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
• https://github.com/nakamasato/kubernetes-training/tree/master/autoscaler/hpa/custom-metrics
• https://qiita.com/gymnstcs/items/ae375899d254b7b36c80
• https://kind.sigs.k8s.io/docs/user/quick-start/#installation