Prometheus+blackboxをgrafanaで可視化 #prometheus

目的

Prometheusで監視対象のホストのメトリクスを収集したり、死活監視をする場合に、grafanaで可視化したいと思う。

ディレクトリ構成

docker-compose.yml
blackbox_exporter
- config.yml
grafana
- Dockerfile
prometheus
- prometheus.yml

Docker系config

docker-compose.yml

version: '3'
services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - '9090:9090'


  blackbox_exporter:
    image: prom/blackbox-exporter:latest
    container_name: blackbox_exporter
    volumes:
      - ./blackbox_exporter/config.yml:/etc/blackbox_exporter/config.yml

  grafana:
#    image: grafana/grafana:latest
    build: ./grafana
    container_name: grafana
    hostname: grafana
    ports:
      - 3000:3000
    user: "$PID:$GID"
    volumes:
      - ./grafana:/var/lib/grafana

grafana/dockerfile
データソースにzabbixを使用する場合はこちらを導入する

FROM grafana/grafana
RUN grafana-cli plugins install magnesium-wordcloud-panel
RUN grafana-cli plugins install alexanderzobnin-zabbix-app

Prometheus系config

prometheus/prometheus.yml

global:
  scrape_interval: 15s # デフォルトの監視対象にpullする間隔
  external_labels: # 外部と通信するときのラベル
    monitor: 'codelab-monitor'

scrape_configs: # 監視対象ごとの設定
  - job_name: 'prometheus' # Prometheus自身の監視
    scrape_interval: 5s # デフォルトの間隔を上書き
    static_configs:
      - targets: ['prometheus:9090']


###### blackbox HTTP
  - job_name: 'blackbox_http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://google.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115


##### blackbox ICMP
  - job_name: 'blackbox_icmp'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
        - 8.8.8.8
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115


##### blackbox TCP
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
        - google.com:443
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115

blackbox_exporter/config.yml

modules:
  http_2xx:
    prober: http
    http:

  http_post_2xx:
    prober: http
    http:
      method: POST

  tcp_connect:
    prober: tcp

  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false

  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"

  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"

  icmp:
    prober: icmp

pluginの有効化

datasourceにzabbixを用いる場合は、grafanaのdashboardから有効化する必要がある
左側のパネルから「plugins」

下にスクロールするとzabbixがあるので、クリック

Enableをクリック

grafanaにpromqlを入力

graphはgauge等にすると分かりやすい

動作確認

動いてる