Prometheus OperatorでKubernetes Cluster外部をモニタリングする

はじめに

API等をKubernetesで構築していて、データベースは外部のVMを使用しているユースケースがあると思う。
そのVMのモニタリングをPrometheusのExporterを使用して実現する。

前提条件

KubernetesにPrometheus Operatorを導入済みである
- 今回はGKEにHelmでPrometheus Operatorをインストールしている
監視対象のVMにデータベースを構築済みである
- 今回はVM(GCE)にCassandra Clusterをインストールしている

Cassandra Exporterを導入する

Cassandraの監視はcassandra_exporterを使用する。
これは、JMX exporterをforkしたプロジェクト。

cassandra_exporterのユーザーを作成する

先ずはグループの作成

$ sudo groupadd -r cassandra_exporter

次にユーザーを作成
システムユーザーでログインできないようにしておく

$ sudo useradd -r -s /bin/false -g cassandra_exporter cassandra_exporter

ユーザー作成後確認

$ id cassandra_exporter
uid=997(cassandra_exporter) gid=994(cassandra_exporter) groups=994(cassandra_exporter)

cassandra_exporter(本体)のダウンロード

ここからダウンロードする

$ curl -L -O https://github.com/criteo/cassandra_exporter/releases/download/2.3.5/cassandra_exporter-2.3.5.jar

$ mv ./cassandra_exporter-2.3.5.jar /opt/cassandra_exporter.jar

$ chown cassandra_exporter:cassandra_exporter /opt/cassandra_exporter.jar

$ chmod 0755 /opt/cassandra_exporter.jar

cassandra_exporter(設定ファイル)の作成

ここを参考にする

$ vi /etc/cassandra_exporter_config.yml

$ chown cassandra_exporter:cassandra_exporter /etc/cassandra_exporter_config.yml

cassandra_exporter_config.yml

host: localhost:7199
ssl: False
user:
password:
listenAddress: 0.0.0.0
listenPort: 8080
# Regular expression to match environment variable names that will be added
# as labels to all data points. The name of the label will be either
# $1 from the regex below, or the entire environment variable name if no match groups are defined
#
# Example:
# additionalLabelsFromEnvvars: "^ADDL\_(.*)$"
additionalLabelsFromEnvvars:
blacklist:
   # To profile the duration of jmx call you can start the program with the following options
   # > java -Dorg.slf4j.simpleLogger.defaultLogLevel=trace -jar cassandra_exporter.jar config.yml --oneshot
   #
   # To get intuition of what is done by cassandra when something is called you can look in cassandra
   # https://github.com/apache/cassandra/tree/trunk/src/java/org/apache/cassandra/metrics
   # Please avoid to scrape frequently those calls that are iterating over all sstables

   # Unaccessible metrics (not enough privilege)
   - java:lang:memorypool:.*usagethreshold.*

   # Leaf attributes not interesting for us but that are presents in many path
   - .*:999thpercentile
   - .*:95thpercentile
   - .*:fifteenminuterate
   - .*:fiveminuterate
   - .*:durationunit
   - .*:rateunit
   - .*:stddev
   - .*:meanrate
   - .*:mean
   - .*:min

   # Path present in many metrics but uninterresting
   - .*:viewlockacquiretime:.*
   - .*:viewreadtime:.*
   - .*:cas[a-z]+latency:.*
   - .*:colupdatetimedeltahistogram:.*

   # Mostly for RPC, do not scrap them
   - org:apache:cassandra:db:.*

   # columnfamily is an alias for Table metrics
   # https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/metrics/TableMetrics.java#L162
   - org:apache:cassandra:metrics:columnfamily:.*

   # Should we export metrics for system keyspaces/tables ?
   - org:apache:cassandra:metrics:[^:]+:system[^:]*:.*

   # Logback doesn't have any useful metrics
   - ch:qos:logback:.*

   # Don't scrap us
   - com:criteo:nosql:cassandra:exporter:.*

maxScrapFrequencyInSec:
  50:
    - .*

  # Refresh those metrics only every hour as it is costly for cassandra to retrieve them
  3600:
    - .*:snapshotssize:.*
    - .*:estimated.*
    - .*:totaldiskspaceused:.*

systemd serviceとして登録する

cassandra_exporter.sh

vi /opt/cassandra_exporter.sh

#!/bin/bash
java -jar /opt/cassandra_exporter.jar /etc/cassandra_exporter_config.yml


$ chown cassandra_exporter:cassandra_exporter /opt/cassandra_exporter.sh

$ chmod 0755 /opt/cassandra_exporter.sh

$ sudo  vi /etc/systemd/system/cassandra_exporter.service

[Unit]
Description=Prometheus Cassandra Exporter
After=network.target

[Service]
Type=simple
User=cassandra_exporter
Group=cassandra_exporter
ExecStart=/opt/cassandra_exporter.sh

SyslogIdentifier=prometheus_cassandra_exporter
Restart=always

[Install]
WantedBy=multi-user.target

daemon-reloadして認識させる

sudo systemctl daemon-reload

cassandra_exporterを起動する

sudo systemctl start cassandra_exporter.service

cassandra_exporterの確認

systemctl status cassandra_exporter
● cassandra_exporter.service - Prometheus Cassandra Exporter
   Loaded: loaded (/etc/systemd/system/cassandra_exporter.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-04-26 16:41:46 JST; 10s ago
 Main PID: 16849 (cassandra_expor)
   CGroup: /system.slice/cassandra_exporter.service
           ├─16849 /bin/bash /opt/cassandra_exporter.sh
           └─16850 java -jar /opt/cassandra_exporter.jar /etc/cassandra_exporter_config.yml

Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at java.security.AccessController.doPrivileged(Native Method)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at java.security.AccessController.doPrivileged(Native Method)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Apr 26 16:41:55 blcloud-cassandra-01 prometheus_cassandra_exporter[16849]: at java.lang.Thread.run(Thread.java:748)

prometheus OperatorにService Monitorを追加する

上記のようにServiceMonitorは各Serviceを通してScrapeする。
そのため、今回作成したCassandraのExporterに対してKubernetesの Endpoint と Serviceを作成してKubernetes Clusterからアクセスする。

cassandra-exporter-01.yaml

---
apiVersion: v1
kind: Endpoints
metadata:
    name: cassandra-metrics-01
    labels:
        release: prometheus-operator
    namespace: default
subsets:
    - addresses:
      - ip: Cassandra NodeのIPアドレス
      ports:
      - name: metrics
        port: 8080
        protocol: TCP

---
apiVersion: v1
kind: Service
metadata:
    name: cassandra-metrics-01
    namespace: default
    labels:
        release: prometheus-operator
        k8s-app: cassandra-metrics
spec:
    type: ExternalName
    externalName: Cassandra NodeのIPアドレス
    ports:
    - name: metrics
      port: 8080
      protocol: TCP
      targetPort: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
    name: cassandra-metrics-sm-01
    labels:
        release: prometheus-operator
        prometheus: kube-prometheus
    namespace: default
spec:
    selector:
        matchLabels:
            release: prometheus-operator
            k8s-app: cassandra-metrics-01
    endpoints:
    - port: metrics
      interval: 10s
      honorLabels: true
      path: /metrics

$ kubectl apply -f cassandra-exporter-01.yaml
endpoints/cassandra-metrics-01 created
service/cassandra-metrics-01 created
servicemonitor.monitoring.coreos.com/cassandra-metrics-sm-01 created

確認

今回、GrafanaのIngressは作ってなかったので、port-forwardで確認する

kubectl port-forward svc/prometheus-operator-grafana 8080:80

6400番のこのdashboardをインポートする

そして、Data Sourceを --Grafana-- -> Prometheusに変更する

すると、以下のようにメトリクスが表示されるようになる