LoginSignup
4
3

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

@prodigy413(Obi)

Prometheus サンプル設定

Posted at

概要

  • k8sテスト環境構築

    Prometheus サンプル設定

構築目次

環境

  • Rancher: v2.4.8
  • kubernetes(Client): v1.19.1
  • kubernetes(Server): v1.18.8
  • kube-prometheus-stack Chart: v9.4.3
  • kube-prometheus-stack App: v0.38.1

サンプル設定サマリ

  1. Nginx metricsのexpose設定
  2. NginxをPrometheusのtargetに追加
  3. Nginxのhttp request countを表示するGrafana dashboard 作成
  4. http request countを使ってアラート作成 → アラートを外部転送

Nginx metrics 設定

server {
    listen   8080;
    location /metrics {
        stub_status;
    }
}
test-nginx.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configmap
  labels:
    app: nginx
data:
  nginx.conf: |2

    user  nginx;
    worker_processes  1;
    error_log  /var/log/nginx/error.log warn;
    pid        /var/run/nginx.pid;
    events {
        worker_connections  1024;
    }
    http {
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
        access_log  /var/log/nginx/access.log  main;

        server {
            listen   8080;
            location /metrics {
                stub_status;
            }
        }

        sendfile        on;
        keepalive_timeout  65;
        include /etc/nginx/conf.d/*.conf;
    }
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  ports:
  - name: nginx-http
    protocol: TCP
    port: 80
    targetPort: 80
  - name: nginx-exporter
    protocol: TCP
    port: 9113
    targetPort: 9113
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deploy
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.19.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-conf
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
      - name: nginx-exporter
        image: nginx/nginx-prometheus-exporter:0.8.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9113
        args:
          - -nginx.scrape-uri=http://localhost:8080/metrics
      volumes:
      - name: nginx-conf
        configMap:
          name: nginx-configmap
          items:
          - key: nginx.conf
            path: nginx.conf
  • Nginx 起動
$ kubectl apply -f test-nginx.yaml

Prometheus Target 追加

Prometheus Operatorのインストール時、追加されるCRDServiceMonitorを使ってターゲット追加

  • serviceMonitorSelector 確認

    serviceMonitorSelector.matchLabelsの値をServiceMonitorに追加する

    この環境ではrelease: prometheusを追加

    → 設定しないとPrometheusはServiceMonitorを追加しない
$ kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus -o yaml
..........
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      release: prometheus
..........
  • マニフェスト作成
nginx-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-servicemonitor
  namespace: monitoring
  labels:
    app: nginx
    release: prometheus # serviceMonitorSelectorで確認したラベル
spec:
  endpoints:
  - port: nginx-exporter
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app: nginx # Serviceのラベル
  • ServiceMonitor 作成
$ kubectl apply -f nginx-servicemonitor.yaml

## 確認 ##
$ kubectl get servicemonitor -n monitoring
NAME                             AGE
..........
nginx-servicemonitor             64s
..........

Nginxのmetricsも使用可能
5-1.png

Grafana Dashboard 作成

  • Grafanaにログイン → [+] > [Dashboard]選択
    6-1.png

  • [Dashboard settings]クリック
    13-1.png

  • Dashboard名設定
    [General] > [Name]設定
    ・Name: Nginx HTTP Request Count
    14-1.png

  • 変数設定
    [Variables] > [Add variable]クリック
    15-1.png
    以下変数を作成
    ※各変数設定後、[Add]クリックで保存 > [New]クリックで新規変数作成

Name Data source Refresh Query
cluster Prometheus On Dashboard Load label_values(kube_pod_info, cluster)
namespace Prometheus On Dashboard Load label_values(kube_pod_info{cluster="$cluster"}, namespace)

  • 確認
    [←]クリック
    7-1.png
    変数を確認し、Nginx Podを作成したnamespaceに設定しておく
    ・cluster: None
    ・namespace: default
    1-1.png

  • Panel 作成
    [Ad new panel]クリック
    8-2.png
    以下のように設定
    ・Panel title: HTTP Request Count
    9-3.png
    ・Metrics: sum(increase(nginx_http_requests_total{namespace="$namespace", cluster="$cluster"}[1m])) by (pod)
    　→ 各PodのHTTP Request Countの合計/1分
    ・Legend: {{pod}}
    2-1.png
    ・Decimals: 0
    9-2.png
    [Save]クリックして保存
    ・Dashboard name: Nginx HTTP Request Count
    9-4.png
    18.png

  • 動作確認

  • Nginx port forward 設定

    $ kubectl port-forward svc/nginx-svc 8080:80

  • 他のTerminalでアクセス

    $ while true ; do curl http://localhost:8080 ; sleep 5 ; done

  • Dashboard 確認

    3-1.png

Alert 設定

Prometheus Operatorのインストール時、追加されるCRDPrometheusRuleを使ってターゲット追加

  • ruleSelector 確認

    ruleSelector.matchLabelsの値をPrometheusRuleに追加する

    この環境ではapp: kube-prometheus-stack、release: prometheusを追加

    → 設定しないとPrometheusはPrometheusRuleを追加しない
$ kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus -o yaml
..........
  ruleNamespaceSelector: {}
  ruleSelector:
    matchLabels:
      app: kube-prometheus-stack
      release: prometheus
..........
  • マニフェスト作成
  • 作成Rule: HTTP Request Count値が1分間20を超えたらアラート発生
nginx-http-request-rule.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    app: kube-prometheus-stack # ruleSelectorで確認したラベル
    release: prometheus # ruleSelectorで確認したラベル
  name: http-request-rule.rules
  namespace: monitoring
spec:
  groups:
  - name: nginx-http-request.rules
    rules:
    - alert: NginxTooManyRequest
      annotations:
        message: HTTP Request to {{ $labels.namespace }}/{{ $labels.pod }} is over 20 / 1 minute.
      expr: sum(increase(nginx_http_requests_total{job="nginx-svc", namespace=~".*"}[1m])) by (namespace, pod) > 20
      for: 1m
      labels:
        severity: critical
  • PrometheusRule 作成
$ kubectl apply -f nginx-http-request-rule.yaml

## 確認 ##
$ kubectl get prometheusrule -n monitoring
NAME                                AGE
..........
http-request-rule.rules             27m
..........

Alert 外部転送

AlertをSlackに転送

  • Slack 設定 / Webhook URL 確認
    [Slackをブラウズする] > [App]クリック
    2-1.png
    incoming webhooks検索 > [追加]クリック
    3-1.png
    [Slackに追加]クリック
    4-1.png
    チャンネル選択 > [Incoming Webhook インテルレーションの追加]クリック
    6-1.png
    Webhook URL確認
    7-1.png

  • 転送設定マニフェスト作成

  • slack_api_urlは上記で確認したWebhook URL

  • channelはWebhook設定時、選択したチャンネル

alertmanager.yaml
global:
  resolve_timeout: 1m
  slack_api_url: 'https://hooks.slack.com/services/xxxxx..............................'
route:
  receiver: 'slack_notifications'
  group_interval: 5m
  group_wait: 30s
  repeat_interval: 12h
  routes:
  - match:
      alertname: 'NginxTooManyRequest'
    receiver: 'nginx_request_count_error'
receivers:
- name: 'slack_notifications'
- name: 'nginx_request_count_error'
  slack_configs:
  - channel: '#it-test'
    send_resolved: true
    icon_url: https://avatars3.githubusercontent.com/u/3380462
  • 設定ファイルのsecret化

    上記マニフェストのbase64値を確認
$ kubectl create secret generic test --from-file=./alertmanager.yaml --dry-run=client -o yaml
apiVersion: v1
data:
  alertmanager.yaml: {base64値}
kind: Secret
metadata:
  creationTimestamp: null
  name: test
  • secret 更新用マニフェスト作成

    上記のbase64値を使って更新用マニフェスト作成
alertmanager-slack.yaml
apiVersion: v1
kind: Secret
metadata:
  labels:
    release: prometheus
  name: alertmanager-prometheus-prometheus-oper-alertmanager
  namespace: monitoring
type: Opaque
data:
  alertmanager.yaml: {base64値}
  • secret 更新
## secret 確認 ##
$ kubectl get secret -n monitoring
NAME                                                          TYPE              DATA   AGE
..........
alertmanager-prometheus-kube-prometheus-alertmanager          Opaque            1      3d10h
..........

## 更新 ##
$ kubectl apply -f alertmanager-slack.yaml

  • 確認

    http://alertmanager.test.local/#/status
    6.png

  • 動作確認

  • Nginx port forward 設定

    $ kubectl port-forward svc/nginx-svc 8080:80

  • 他のTerminalでアクセス

    $ while true ; do curl http://localhost:8080 ; sleep 1 ; done

  • Slack 確認

    7.png

4
3
1

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
4
3

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?