Qiita Teams that are logged in
You are not logged in to any team

Log in to Qiita Team
Community
OrganizationEventAdvent CalendarQiitadon (β)
Service
Qiita JobsQiita ZineQiita Blog
1
Help us understand the problem. What are the problem?

posted at

updated at

Organization

KubernetesクラスタへPrometheus+GrafanaをOperatorで簡単にインストール

概要

Kubernetesクラスタ上にPrometheusをインストールする方法を説明する.
ここではPrometheus用のKubernetes オペレータを使う.
prometheus-operator/prometheus-operatorはbetaであるため,prometheus-operator/kube-prometheusを使う.

環境

  • Docker for Desktop 4.1.1 with K8s
    • vCPU: 4
    • Memory: 5GB
    • Swap: 1GB
    • Disk Image size: 59.6GB(21.5GB used)
    • Kubernetes v1.21.5
  • M1 Macbook Air 2021
$ kubectl version                                                                                                      (git)[main]
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:59:07Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/arm64"}

手順

このURLに書いてあるQuickstartを読みながら作業する.

# リポジトリを取得
git clone --depth=1 git@github.com:prometheus-operator/kube-prometheus.git
cd kube-prometheus/

# セットアップを実行
kubectl create -f manifests/setup

# 待つ
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done

# マニフェストを適用(リソースを追加)
kubectl create -f manifests/

リソースが不足している場合,PodがPendingになるのでリソースを増やす.本来はRequierementsを読んで適切なノードのスペックを調べてから作業を行う

kubectl get pod -n monitoring -l prometheus=k8s                                                                      (git)[main]
NAME               READY   STATUS    RESTARTS   AGE
prometheus-k8s-0   0/2     Pending   0          3m38s
prometheus-k8s-1   0/2     Pending   0          3m38s

原因を調べると,0/1 nodes are available: 1 Insufficient memory.とあるのでクラスタのメモリを増やす.

kubectl describe pod/prometheus-k8s-0 -n monitoring                                                                  (git)[main]
Name:         prometheus-k8s-0
Namespace:    monitoring
Priority:     0
Node:         docker-desktop/192.168.65.4
Start Time:   Wed, 10 Nov 2021 16:56:14 +0900
Labels:       app.kubernetes.io/component=prometheus
              app.kubernetes.io/instance=k8s
              app.kubernetes.io/managed-by=prometheus-operator
              app.kubernetes.io/name=prometheus
              app.kubernetes.io/part-of=kube-prometheus
              app.kubernetes.io/version=2.31.1
              controller-revision-hash=prometheus-k8s-7d994f449b
              operator.prometheus.io/name=k8s
              operator.prometheus.io/shard=0
              prometheus=k8s
              statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations:  kubectl.kubernetes.io/default-container: prometheus
Status:       Pending
IP:           10.1.0.75
IPs:
  IP:           10.1.0.75
Controlled By:  StatefulSet/prometheus-k8s
Init Containers:
  init-config-reloader:
    Container ID:  docker://d159301da6826148013e624a8e050f64ad4c8fcfd119a3ca7c2e7c19585edf6c
    Image:         quay.io/prometheus-operator/prometheus-config-reloader:v0.52.0
    Image ID:      docker-pullable://quay.io/prometheus-operator/prometheus-config-reloader@sha256:d8ceaa9878215db2c2e06e79ee8593a48186888ea48179793b9cc34801b6b486
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 10 Nov 2021 16:56:15 +0900
      Finished:     Wed, 10 Nov 2021 16:56:15 +0900
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jj68 (ro)
Containers:
  prometheus:
    Container ID:
    Image:         quay.io/prometheus/prometheus:v2.31.1
    Image ID:
    Port:          9090/TCP
    Host Port:     0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --storage.tsdb.retention.time=24h
      --web.enable-lifecycle
      --web.route-prefix=/
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      memory:     400Mi
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=120
    Environment:  <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-k8s-db (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jj68 (ro)
  config-reloader:
    Container ID:
    Image:         quay.io/prometheus-operator/prometheus-config-reloader:v0.52.0
    Image ID:
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --listen-address=:8080
      --reload-url=http://localhost:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jj68 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s
    Optional:    false
  tls-assets:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s-tls-assets
    Optional:    false
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional:  false
  web-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s-web-config
    Optional:    false
  prometheus-k8s-db:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-6jj68:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  5m57s  default-scheduler  0/1 nodes are available: 1 Insufficient memory.
  Warning  FailedScheduling  5m55s  default-scheduler  0/1 nodes are available: 1 Insufficient memory.
  Normal   Scheduled         12s    default-scheduler  Successfully assigned monitoring/prometheus-k8s-0 to docker-desktop

Webアクセス

PromethesへWebからアクセスしてみる.

kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090

SSHポートフォワード

仮にVMからkubectlを実行している場合,sshのオプションでポートフォワードを行う.
以下は,自分のラップトップの9000番をssh先の9090に対応付けている例である.

ssh -L 9000:localhost:9090 <YOUR_HOST_NAME>

GrafanaへWebからアクセスしてみる.

kubectl --namespace monitoring port-forward svc/grafana 3000

Service(type:Ingress)

Grafanaへアクセスするためにingressを作成

Grafanaへkubectl port-forward以外でアクセスを実現する方法としてingressの利用がある.
外部から簡単にアクセスできるようingressで外部へ公開する設定を入れる.

以下の内容を grafana-publish.yml として /opt/prometheus に保存する.
<YOUR_HOSTNAME>をホスト名に置き換える.

grafana-publish.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-grafana
  namespace: monitoring
spec:
  ingressClassName: nginx
  rules:
  - host: <YOUR_HOSTNAME>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grafana
            port:
              number: 3000

kubectlコマンドでリソースを作成する.

kubectl apply -f /opt/prometheus/grafana-publish.yml

以下はGrafanaからPodのCPUとMemoryを確認している例である.

image.png

Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
1
Help us understand the problem. What are the problem?