Help us understand the problem. What is going on with this article?

Check! Node-RED で Prometheus のメトリクスを可視化する

More than 1 year has passed since last update.

こんにちは、 @dz_ こと大平かづみです。

Node-RED Advent Calendar 2017 11日目の投稿です。

Prologue - はじめに

Node-RED のダッシュボードがお手軽で便利なので、これはインフラのモニタリングにも使えるのではないかと試してみました!

題材は Prometheus のメトリクスを Node-RED で受け取り、可視化してみます。Prometheus は、Grafana と組み合わせるのがスタンダードですが、物は試しです (ノ*’ω’)ノ

コードサンプル

今回書いたコードはこちらに公開しました。このコードをベースに話を進めます。

準備

Docker で環境を用意する

Node-RED も Prometheus も Docker イメージが提供されているので、それを利用します。

これらを docker-compose で構成しました。

docker-compose.yml
version: "3"
services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./docker/prometheus/config/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
  nodered:
    image: nodered/node-red-docker
    container_name: nodered
    volumes:
      - ./data:/data
      - ./nodes:/nodes
    ports:
      - "1880:1880"
    links:
      - prometheus

prometheus.yml は、Prometheus の設定ファイルで、この Getthing Started の内容を参考にしています。

上記の準備ができたら、コンテナインスタンスを立ち上げます。

docker-compose up -d

これで、Node-RED エディタと、Prometheus の簡易GUIにアクセスできるようになりました。

項目 URL
Node-RED エディタ http://localhost:1880
Prometheus の簡易GUI http://localhost:9090

Node-RED ダッシュボードの準備

それから、Node-RED のダッシュボードが使えるように、「パレットの管理」> 「ノードを追加」から、 node-red-dashuboard を追加しておきます。

Prometheus のメトリクスについて理解する

Promeheus は、Exporter と呼ばれる別のコンポーネントがメトリクスを収集してきています。そのフォーマットについてはこちらに記載があります。

ざっくり理解するとこんな感じです。

metric_name{label_name1="label_value1", label_name2="label_value2", ... } value

ラベルが入れ子になったり、タイムスタンプがつくパターンもあるようですが、割愛します。

フローを作成する(簡易版)

まずは簡単なフローを組んでみました。任意のメトリクスを選択して可視化します。

  1. inject ノードで定期的に実行する
  2. http request ノードで Prometheus のメトリクスを取得する
  3. split ノードで行ごとに分割、 switch ノードで取得したいパラメータでメトリクスを選択する
  4. splitswitch ノードを組み合わせて、メトリクスデータを取得する
  5. ダッシュボードの ui_chart ノードに入力する

サンプルフロー

上記構成のサンプルコードです。Prometheus 自体が発行する go_goroutines というメトリクスをダッシュボードに表示しています。

[{"id":"e8b15b7b.923128","type":"inject","z":"8f3a7677.4b1d08","name":"","topic":"","payload":"","payloadType":"date","repeat":"10","crontab":"","once":false,"x":132.00000762939453,"y":239.00003242492676,"wires":[["7b270a3d.128be4"]]},{"id":"7b270a3d.128be4","type":"http request","z":"8f3a7677.4b1d08","name":"","method":"GET","ret":"txt","url":"http://prometheus:9090/metrics","tls":"","x":318.00001525878906,"y":238.76392364501953,"wires":[["af09abb4.c607f8"]]},{"id":"af09abb4.c607f8","type":"split","z":"8f3a7677.4b1d08","name":"split by \\n","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":147.0000114440918,"y":313.33336544036865,"wires":[["579d6691.276068"]]},{"id":"40e71a0.454c7e8","type":"debug","z":"8f3a7677.4b1d08","name":"","active":true,"console":"false","complete":"true","x":333.00000381469727,"y":513.444465637207,"wires":[]},{"id":"579d6691.276068","type":"switch","z":"8f3a7677.4b1d08","name":"get go_goroutiens","property":"payload","propertyType":"msg","rules":[{"t":"regex","v":"^go_goroutines .*$","vt":"str","case":false}],"checkall":"true","outputs":1,"x":336.38890838623047,"y":313.2222099304199,"wires":[["b058c01d.d3354"]]},{"id":"b058c01d.d3354","type":"split","z":"8f3a7677.4b1d08","name":"split by space","splt":" ","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":166.3888931274414,"y":388.2222385406494,"wires":[["cc0b7c3f.10cc6"]]},{"id":"cc0b7c3f.10cc6","type":"switch","z":"8f3a7677.4b1d08","name":"get index[1]","property":"parts.index","propertyType":"msg","rules":[{"t":"eq","v":"1","vt":"num"}],"checkall":"true","outputs":1,"x":156.38890838623047,"y":457.2222385406494,"wires":[["ddbb5c4f.b44eb","40e71a0.454c7e8"]]},{"id":"ddbb5c4f.b44eb","type":"ui_chart","z":"8f3a7677.4b1d08","name":"","group":"215d19a8.9edbb6","order":0,"width":0,"height":0,"label":"chart","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":"1","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":332.38891983032227,"y":456.7778215408325,"wires":[[],[]]},{"id":"215d19a8.9edbb6","type":"ui_group","z":"","name":"go_goroutines","tab":"356225f4.ea84ea","disp":true,"width":"10"},{"id":"356225f4.ea84ea","type":"ui_tab","z":"","name":"Prometheus","icon":"dashboard"}]

image.png
image.png

ポイント

http request ノードには Node-RED サーバーからアクセスできるURLを指定する

今回、http request ノードに指定するURLは、 http://localhost:9090/metrics ではなく、 http://prometheus:9090/metrics を指定します。

どういうことかというと、 localhost:9090 は、実行マシンから みた
Prometheus の URL なので、Node-RED からは見えません。そこで、前述の docker-compose.yml で、nodered コンテナから prometheus コンテナに対して link を貼って、 prometheus:9090 でアクセスできるようにしました。

今回は Docker の場合ですが、例えばクラウドであれば、仮想ネットワークやセキュリティグループなどの設定で、ルートを確保する必要がありそうです。

とはいえ、 switch ノードによるメトリクスの抽出がいまいち…

作ってみてなんですが、このフロー構成だと、複雑なメトリクスを扱うのが面倒ですね…。

よし!ノードを自作してみましょう!🙌

Prometheus のメトリクス解析ノードを作ってみる(改良版)

実はノードを自作したことなかったので、これを機にやってみます。ノードの自作については、こちらをご参照ください。

自作ノード node-red-contrib-parser-prom-metrics の仕様

自作ノードは、http_request ノードで取得した /metrics の結果を入力すると、下記のようなJSON構造に変換する仕様にしました。

{
    "name": "<metric_name>",
    "value": <value>,
    "labels": {
        "label_name1": "label_value1",
        "label_name2": "label_value2",
        ...
    }
}

コードは、前述のコードサンプルの nodes/parser-prom-metrics をご参照ください。

サンプルフロー

さて、サンプルとして、http_requests_total の複数データを1つのグラフに表示してみました。ノードの構成もシンプルになりました!

[{"id":"df1a33e9.993f5","type":"inject","z":"78d9cc24.3c55a4","name":"","topic":"","payload":"","payloadType":"date","repeat":"10","crontab":"","once":false,"x":154,"y":77.23610877990723,"wires":[["951f8fda.3b733"]]},{"id":"951f8fda.3b733","type":"http request","z":"78d9cc24.3c55a4","name":"","method":"GET","ret":"txt","url":"http://prometheus:9090/metrics","tls":"","x":340.00000762939453,"y":77,"wires":[["22dfb0e0.c8f01"]]},{"id":"22dfb0e0.c8f01","type":"parser-prom-metrics","z":"78d9cc24.3c55a4","name":"","x":151.388916015625,"y":153.23609733581543,"wires":[["209317b4.e62488"]]},{"id":"63747b53.8c22c4","type":"switch","z":"78d9cc24.3c55a4","name":"filter by http_requests_total","property":"payload.name","propertyType":"msg","rules":[{"t":"eq","v":"http_requests_total","vt":"str"}],"checkall":"true","outputs":1,"x":206.38889694213867,"y":235.01391696929932,"wires":[["fb01d069.8a2c3","9ffbd8dc.2f92d8"]]},{"id":"209317b4.e62488","type":"split","z":"78d9cc24.3c55a4","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":322.3888931274414,"y":152.01390266418457,"wires":[["63747b53.8c22c4"]]},{"id":"9f300d6.7394af","type":"ui_chart","z":"78d9cc24.3c55a4","name":"prometheus","group":"a815aaaf.67adb8","order":0,"width":0,"height":0,"label":"handler=pometheus only","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":548.3888893127441,"y":326.7917261123657,"wires":[[],[]]},{"id":"4702d48b.49a73c","type":"change","z":"78d9cc24.3c55a4","name":"","rules":[{"t":"move","p":"payload.labels.handler","pt":"msg","to":"topic","tot":"msg"},{"t":"move","p":"payload.value","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":335.388916015625,"y":326.1250047683716,"wires":[["9f300d6.7394af","6cd20ad5.0a2794"]]},{"id":"fb01d069.8a2c3","type":"switch","z":"78d9cc24.3c55a4","name":"prometheus","property":"payload.labels.handler","propertyType":"msg","rules":[{"t":"eq","v":"prometheus","vt":"str"}],"checkall":"true","outputs":1,"x":151.3888931274414,"y":326.66667079925537,"wires":[["4702d48b.49a73c"]]},{"id":"61338a91.013f44","type":"ui_chart","z":"78d9cc24.3c55a4","name":"http_requests_total","group":"9e8c057f.89f1f8","order":0,"width":0,"height":0,"label":"other handler","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":570.9999465942383,"y":461.0000066757202,"wires":[[],[]]},{"id":"a70e5931.6cd578","type":"change","z":"78d9cc24.3c55a4","name":"","rules":[{"t":"move","p":"payload.labels.handler","pt":"msg","to":"topic","tot":"msg"},{"t":"move","p":"payload.value","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":335.99999618530273,"y":461.3332862854004,"wires":[["61338a91.013f44","6cd20ad5.0a2794"]]},{"id":"6cd20ad5.0a2794","type":"debug","z":"78d9cc24.3c55a4","name":"","active":true,"console":"false","complete":"payload","x":549,"y":370.99997901916504,"wires":[]},{"id":"9ffbd8dc.2f92d8","type":"switch","z":"78d9cc24.3c55a4","name":"exclude prometheus","property":"payload.labels.handler","propertyType":"msg","rules":[{"t":"neq","v":"prometheus","vt":"str"}],"checkall":"true","outputs":1,"x":185.16666412353516,"y":396,"wires":[["a70e5931.6cd578"]]},{"id":"a815aaaf.67adb8","type":"ui_group","z":"","name":"http_requests_total (prometheus)","tab":"6832eea4.26d3","order":2,"disp":true,"width":"12"},{"id":"9e8c057f.89f1f8","type":"ui_group","z":"","name":"http_requests_total","tab":"6832eea4.26d3","order":1,"disp":true,"width":"12"},{"id":"6832eea4.26d3","type":"ui_tab","z":"","name":"Prometheus","icon":"dashboard"}]

image.png
image.png

ポイント

自作ノードを有効にする

今回、既成のコンテナイメージを使っているので、自作ノードを有効にするには少しコツがいりました。それは、root 権限で npm link を貼らないとならなかったので、下記のように対応しました。

docker exec -it --user=root nodered /bin/bash

# nodered コンテナの中で npm link で相互にリンクさせます
cd /nodes/parser-prom-metrics && npm link
cd /usr/src/node-red && npm link node-red-contrib-parser-prom-metrics
exit

docker restart nodered

nodered コンテナが再起動し終わったら、自作ノードが使えるようになってるはずです。

ダッシュボードの ui-chart で複数のグラフを表示させる

このような形式でデータを入力すると、複数のグラフを表示させることができます。詳細はドキュメントをご参照ください。

{
    "msg": {
        "topic": "<データのラベル>",
        "payload": <値>
    }
}

Epilogue - おわりに

既存のノードだけで作ってみたらあんまり格好良くなかったので、自作ノードに手を出してみました。簡易版よりスマートになって満足です!

ともあれ、こうして遊ぶのも勉強になっていいですね!🙌

dz_
Nice to meet you! :)
Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
Comments
No comments
Sign up for free and join this conversation.
If you already have a Qiita account
Why do not you register as a user and use Qiita more conveniently?
You need to log in to use this function. Qiita can be used more conveniently after logging in.
You seem to be reading articles frequently this month. Qiita can be used more conveniently after logging in.
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
ユーザーは見つかりませんでした