KubernetesクラスタへPrometheus+GrafanaをOperatorで簡単にインストール

この資料の内容は最新版のドキュメントとは異なります．公式ドキュメントのQuickstartを参照してください．

概要

Kubernetesクラスタ上にPrometheusをインストールする方法を説明する．
ここではPrometheus用のKubernetes オペレータを使う．
prometheus-operator/prometheus-operatorはbetaであるため，prometheus-operator/kube-prometheusを使う．

環境

• Docker for Desktop 4.1.1 with K8s
  • vCPU: 4
  • Memory: 5GB
  • Swap: 1GB
  • Disk Image size: 59.6GB(21.5GB used)
  • Kubernetes v1.21.5
• M1 Macbook Air 2021

$ kubectl version                                                                                                      (git)[main]
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:59:07Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/arm64"}

手順

このURLに書いてあるQuickstartを読みながら作業する．

# リポジトリを取得
git clone --depth=1 [e-mail]:prometheus-operator/kube-prometheus.git
cd kube-prometheus/

# セットアップを実行
kubectl create -f manifests/setup

# 待つ
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done

# マニフェストを適用（リソースを追加）
kubectl create -f manifests/

リソースが不足している場合，PodがPendingになるのでリソースを増やす．本来はRequierementsを読んで適切なノードのスペックを調べてから作業を行う

kubectl get pod -n monitoring -l prometheus=k8s                                                                      (git)[main]
NAME               READY   STATUS    RESTARTS   AGE
prometheus-k8s-0   0/2     Pending   0          3m38s
prometheus-k8s-1   0/2     Pending   0          3m38s

原因を調べると，0/1 nodes are available: 1 Insufficient memory.とあるのでクラスタのメモリを増やす．

kubectl describe pod/prometheus-k8s-0 -n monitoring                                                                  (git)[main]
Name:         prometheus-k8s-0
Namespace:    monitoring
Priority:     0
Node:         docker-desktop/192.168.65.4
Start Time:   Wed, 10 Nov 2021 16:56:14 +0900
Labels:       app.kubernetes.io/component=prometheus
              app.kubernetes.io/instance=k8s
              app.kubernetes.io/managed-by=prometheus-operator
              app.kubernetes.io/name=prometheus
              app.kubernetes.io/part-of=kube-prometheus
              app.kubernetes.io/version=2.31.1
              controller-revision-hash=prometheus-k8s-7d994f449b
              operator.prometheus.io/name=k8s
              operator.prometheus.io/shard=0
              prometheus=k8s
              statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations:  kubectl.kubernetes.io/default-container: prometheus
Status:       Pending
IP:           10.1.0.75
IPs:
  IP:           10.1.0.75
Controlled By:  StatefulSet/prometheus-k8s
Init Containers:
  init-config-reloader:
    Container ID:  docker://d159301da6826148013e624a8e050f64ad4c8fcfd119a3ca7c2e7c19585edf6c
    Image:         quay.io/prometheus-operator/prometheus-config-reloader:v0.52.0
    Image ID:      docker-pullable://quay.io/prometheus-operator/prometheus-config-reloader@sha256:d8ceaa9878215db2c2e06e79ee8593a48186888ea48179793b9cc34801b6b486
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 10 Nov 2021 16:56:15 +0900
      Finished:     Wed, 10 Nov 2021 16:56:15 +0900
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jj68 (ro)
Containers:
  prometheus:
    Container ID:
    Image:         quay.io/prometheus/prometheus:v2.31.1
    Image ID:
    Port:          9090/TCP
    Host Port:     0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --storage.tsdb.retention.time=24h
      --web.enable-lifecycle
      --web.route-prefix=/
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      memory:     400Mi
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=120
    Environment:  <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-k8s-db (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jj68 (ro)
  config-reloader:
    Container ID:
    Image:         quay.io/prometheus-operator/prometheus-config-reloader:v0.52.0
    Image ID:
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --listen-address=:8080
      --reload-url=http://localhost:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jj68 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s
    Optional:    false
  tls-assets:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s-tls-assets
    Optional:    false
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional:  false
  web-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s-web-config
    Optional:    false
  prometheus-k8s-db:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-6jj68:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  5m57s  default-scheduler  0/1 nodes are available: 1 Insufficient memory.
  Warning  FailedScheduling  5m55s  default-scheduler  0/1 nodes are available: 1 Insufficient memory.
  Normal   Scheduled         12s    default-scheduler  Successfully assigned monitoring/prometheus-k8s-0 to docker-desktop

Webアクセス

Webからアクセスする方法として，次の2つを説明する．

1. kubectl port-forward & SSHポートフォワード
2. Ingressを作成

(1) kubectl port-forward & SSHポートフォワード

kubectl port-forwardでlocalhost:9090をServiceに対応付ける．

kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090

ブラウザから http://localhost:9090/ へアクセスする．

仮にVMからkubectlを実行している場合，sshのオプションでポートフォワードを行う．
以下は，自分のラップトップの9000番をssh先の9090に対応付けている例である．
この場合は，ブラウザから http://localhost:9000/ へアクセスする．

ssh -L 9000:localhost:9090 <YOUR_HOST_NAME>

Grafanaの場合，svc/prometheus-k8sをsvc/grafanaに変更する．

(2) Ingressを作成

kubectl port-forward以外でアクセスを実現する方法としてingressの利用がある．
外部から簡単にアクセスできるようingressで外部へ公開する設定を入れる．

ここではGrafanaをIngressでクラスタ外からアクセス可能にする．
以下の内容を grafana-publish.yml として /opt/prometheus に保存する．
<YOUR_HOSTNAME>をホスト名に置き換える．

grafana-publish.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-grafana
  namespace: monitoring
spec:
  ingressClassName: nginx
  rules:
  - host: <YOUR_HOSTNAME>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grafana
            port:
              number: 3000

kubectlコマンドでリソースを作成する．

kubectl apply -f /opt/prometheus/grafana-publish.yml

ブラウザから以下のURLにアクセスする．

http://<YOUR_HOSTNAME>/

以下はGrafanaからPodのCPUとMemoryを確認している例である．