LoginSignup
5

More than 1 year has passed since last update.

KubernetesクラスタへPrometheus+GrafanaをOperatorで簡単にインストール

Last updated at Posted at 2021-11-13

この資料の内容は最新版のドキュメントとは異なります.公式ドキュメントのQuickstartを参照してください.

概要

Kubernetesクラスタ上にPrometheusをインストールする方法を説明する.
ここではPrometheus用のKubernetes オペレータを使う.
prometheus-operator/prometheus-operatorはbetaであるため,prometheus-operator/kube-prometheusを使う.

環境

  • Docker for Desktop 4.1.1 with K8s
    • vCPU: 4
    • Memory: 5GB
    • Swap: 1GB
    • Disk Image size: 59.6GB(21.5GB used)
    • Kubernetes v1.21.5
  • M1 Macbook Air 2021
$ kubectl version                                                                                                      (git)[main]
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:59:07Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/arm64"}

手順

このURLに書いてあるQuickstartを読みながら作業する.

# リポジトリを取得
git clone --depth=1 git@github.com:prometheus-operator/kube-prometheus.git
cd kube-prometheus/

# セットアップを実行
kubectl create -f manifests/setup

# 待つ
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done

# マニフェストを適用(リソースを追加)
kubectl create -f manifests/

リソースが不足している場合,PodがPendingになるのでリソースを増やす.本来はRequierementsを読んで適切なノードのスペックを調べてから作業を行う

kubectl get pod -n monitoring -l prometheus=k8s                                                                      (git)[main]
NAME               READY   STATUS    RESTARTS   AGE
prometheus-k8s-0   0/2     Pending   0          3m38s
prometheus-k8s-1   0/2     Pending   0          3m38s

原因を調べると,0/1 nodes are available: 1 Insufficient memory.とあるのでクラスタのメモリを増やす.

kubectl describe pod/prometheus-k8s-0 -n monitoring                                                                  (git)[main]
Name:         prometheus-k8s-0
Namespace:    monitoring
Priority:     0
Node:         docker-desktop/192.168.65.4
Start Time:   Wed, 10 Nov 2021 16:56:14 +0900
Labels:       app.kubernetes.io/component=prometheus
              app.kubernetes.io/instance=k8s
              app.kubernetes.io/managed-by=prometheus-operator
              app.kubernetes.io/name=prometheus
              app.kubernetes.io/part-of=kube-prometheus
              app.kubernetes.io/version=2.31.1
              controller-revision-hash=prometheus-k8s-7d994f449b
              operator.prometheus.io/name=k8s
              operator.prometheus.io/shard=0
              prometheus=k8s
              statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations:  kubectl.kubernetes.io/default-container: prometheus
Status:       Pending
IP:           10.1.0.75
IPs:
  IP:           10.1.0.75
Controlled By:  StatefulSet/prometheus-k8s
Init Containers:
  init-config-reloader:
    Container ID:  docker://d159301da6826148013e624a8e050f64ad4c8fcfd119a3ca7c2e7c19585edf6c
    Image:         quay.io/prometheus-operator/prometheus-config-reloader:v0.52.0
    Image ID:      docker-pullable://quay.io/prometheus-operator/prometheus-config-reloader@sha256:d8ceaa9878215db2c2e06e79ee8593a48186888ea48179793b9cc34801b6b486
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 10 Nov 2021 16:56:15 +0900
      Finished:     Wed, 10 Nov 2021 16:56:15 +0900
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jj68 (ro)
Containers:
  prometheus:
    Container ID:
    Image:         quay.io/prometheus/prometheus:v2.31.1
    Image ID:
    Port:          9090/TCP
    Host Port:     0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --storage.tsdb.retention.time=24h
      --web.enable-lifecycle
      --web.route-prefix=/
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      memory:     400Mi
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=120
    Environment:  <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-k8s-db (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jj68 (ro)
  config-reloader:
    Container ID:
    Image:         quay.io/prometheus-operator/prometheus-config-reloader:v0.52.0
    Image ID:
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --listen-address=:8080
      --reload-url=http://localhost:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jj68 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s
    Optional:    false
  tls-assets:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s-tls-assets
    Optional:    false
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional:  false
  web-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s-web-config
    Optional:    false
  prometheus-k8s-db:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-6jj68:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  5m57s  default-scheduler  0/1 nodes are available: 1 Insufficient memory.
  Warning  FailedScheduling  5m55s  default-scheduler  0/1 nodes are available: 1 Insufficient memory.
  Normal   Scheduled         12s    default-scheduler  Successfully assigned monitoring/prometheus-k8s-0 to docker-desktop

Webアクセス

Webからアクセスする方法として,次の2つを説明する.

  1. kubectl port-forward & SSHポートフォワード
  2. Ingressを作成

(1) kubectl port-forward & SSHポートフォワード

kubectl port-forwardでlocalhost:9090をServiceに対応付ける.

kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090

ブラウザから http://localhost:9090/ へアクセスする.

仮にVMからkubectlを実行している場合,sshのオプションでポートフォワードを行う.
以下は,自分のラップトップの9000番をssh先の9090に対応付けている例である.
この場合は,ブラウザから http://localhost:9000/ へアクセスする.

ssh -L 9000:localhost:9090 <YOUR_HOST_NAME>

Grafanaの場合,svc/prometheus-k8ssvc/grafanaに変更する.

(2) Ingressを作成

kubectl port-forward以外でアクセスを実現する方法としてingressの利用がある.
外部から簡単にアクセスできるようingressで外部へ公開する設定を入れる.

ここではGrafanaをIngressでクラスタ外からアクセス可能にする.
以下の内容を grafana-publish.yml として /opt/prometheus に保存する.
<YOUR_HOSTNAME>をホスト名に置き換える.

grafana-publish.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-grafana
  namespace: monitoring
spec:
  ingressClassName: nginx
  rules:
  - host: <YOUR_HOSTNAME>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grafana
            port:
              number: 3000

kubectlコマンドでリソースを作成する.

kubectl apply -f /opt/prometheus/grafana-publish.yml

ブラウザから以下のURLにアクセスする.

http://<YOUR_HOSTNAME>/

以下はGrafanaからPodのCPUとMemoryを確認している例である.

image.png

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
5