Go to Qiita Advent Calendar 2024 Top
LoginSignup
1
1

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

@nakacya(Nakatomi Katsutoshi)

Nginx Vhost Traffic Status のデータを ELK に取り込む

Last updated at Posted at 2020-02-05

Nginx Vhost Traffic Status のデータを ELK に取り込む

Nginx Vhost Traffic Status(以降 Nginx VTS) のデータを取り込む準備

nginx-vts-exporter のインストール

Nginx VTS の JSONデータを Prometeus用に出力してくれる便利なやつ

git-clone
$ git clone https://github.com/hnlq715/nginx-vts-exporter.git
golangのinstall
$ wget https://dl.google.com/go/go1.13.7.linux-amd64.tar.gz
$ sudo tar go1.13.7.linux-amd64.tar.gz -C /usr/local/go
$ sudo cat <<EOF > /etc/profile.d/goenv.sh 
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
EOF
$ source /etc/profile.d/goenv.sh
nginx-vts-exporterのCompile&Install
$ cd nginx-vts-exporter
$ make
$ sudo cp nginx-vts-exporter /usr/bin
$ sudo cp nginx_vts_exporter.default /etc/sysconfig/nginx_vts_exporter
$ sudo cat <<EOF > /etc/sysconfig/nginx_vts_exporter
NGINX_VTS_EXPORTER_OPTS='-telemetry.address="localhost:9913" -nginx.scrape_uri=http://localhost/status/json'
EOF
$ sudo cat <<EOF > /usr/lib/systemd/system/nginx_vts_exporter.service
[Unit]
Description=Prometheus exporter thar exports Nginx VTS stats.
Documentation=https://github.com/hnlq715/nginx-vts-exporter
After=network.target

[Service]
EnvironmentFile=-/etc/sysconfig/nginx_vts_exporter
User=root
ExecStart=/usr/bin/nginx-vts-exporter $NGINX_VTS_EXPORTER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
$ sudo systemctl enable nginx_vts_exporter
$ sudo systemctl start nginx_vts_exporter

データを取得してみる

expoter の Lisner は localhost の port 9913 なので接続してみる。

port9913接続
$ curl http://localhost:9913
<html>
			<head><title>Nginx Exporter</title></head>
			<body>
			<h1>Nginx Exporter</h1>
			<p><a href="/metrics">Metrics</a></p>
			</body>
			</html>

おや？なんか変・・・・・

nginx-vts-exporterのoption
$ nginx-vts-exporter -h
Usage of nginx-vts-exporter:
  -insecure
    	Ignore server certificate if using https (default true)
  -metrics.namespace string
    	Prometheus metrics namespace. (default "nginx")
  -nginx.scrape_timeout int
    	The number of seconds to wait for an HTTP response from the nginx.scrape_uri (default 2)
  -nginx.scrape_uri string
    	URI to nginx stub status page (default "http://localhost/status")
  -telemetry.address string
    	Address on which to expose metrics. (default ":9913")
  -telemetry.endpoint string
    	Path under which to expose metrics. (default "/metrics")
  -version
    	Print version information.

ほう、endpoint が /metrics の模様なので再度試験

endpointに接続
$ curl http://localhost:9913/metrics
# HELP nginx_cache_bytes cache request/response bytes
# TYPE nginx_cache_bytes counter
nginx_cache_bytes{direction="in",zone="cache-zone"} 33366
nginx_cache_bytes{direction="out",zone="cache-zone"} 551016
# HELP nginx_cache_requests cache requests counter
# TYPE nginx_cache_requests counter
nginx_cache_requests{status="bypass",zone="cache-zone"} 65
nginx_cache_requests{status="expired",zone="cache-zone"} 0
nginx_cache_requests{status="hit",zone="cache-zone"} 4
nginx_cache_requests{status="miss",zone="cache-zone"} 7
nginx_cache_requests{status="revalidated",zone="cache-zone"} 0
nginx_cache_requests{status="scarce",zone="cache-zone"} 0
nginx_cache_requests{status="stale",zone="cache-zone"} 0
nginx_cache_requests{status="updating",zone="cache-zone"} 0
<snip>

cache hit などは Virtual Server毎に観たいので以下のように取得してみる。

取得する内容をgrepで絞る
$ /usr/bin/curl -s http://localhost:9913/metrics | grep -v ^# | grep -e "_server_requests" -e "_server_bytes"
nginx_server_bytes{direction="in",host="*"} 1.1809658e+07
nginx_server_bytes{direction="in",host="elk.nakacya.jp"} 146149
nginx_server_bytes{direction="in",host="localhost"} 1.1582905e+07
nginx_server_bytes{direction="in",host="test1.nakacya.jp"} 46807
nginx_server_bytes{direction="in",host="www.nakacya.jp"} 33797
nginx_server_bytes{direction="out",host="*"} 2.554692267e+09
nginx_server_bytes{direction="out",host="elk.nakacya.jp"} 332947
nginx_server_bytes{direction="out",host="localhost"} 2.553677271e+09
nginx_server_bytes{direction="out",host="test1.nakacya.jp"} 597605
nginx_server_bytes{direction="out",host="www.nakacya.jp"} 84444
nginx_server_requests{code="1xx",host="*"} 0
nginx_server_requests{code="1xx",host="elk.nakacya.jp"} 0
nginx_server_requests{code="1xx",host="localhost"} 0
nginx_server_requests{code="1xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="1xx",host="www.nakacya.jp"} 0
nginx_server_requests{code="2xx",host="*"} 33502
nginx_server_requests{code="2xx",host="elk.nakacya.jp"} 73
nginx_server_requests{code="2xx",host="localhost"} 33309
nginx_server_requests{code="2xx",host="test1.nakacya.jp"} 83
nginx_server_requests{code="2xx",host="www.nakacya.jp"} 37
nginx_server_requests{code="3xx",host="*"} 14
nginx_server_requests{code="3xx",host="elk.nakacya.jp"} 14
nginx_server_requests{code="3xx",host="localhost"} 0
nginx_server_requests{code="3xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="3xx",host="www.nakacya.jp"} 0
nginx_server_requests{code="4xx",host="*"} 219
nginx_server_requests{code="4xx",host="elk.nakacya.jp"} 19
nginx_server_requests{code="4xx",host="localhost"} 133
nginx_server_requests{code="4xx",host="test1.nakacya.jp"} 33
nginx_server_requests{code="4xx",host="www.nakacya.jp"} 34
nginx_server_requests{code="5xx",host="*"} 2
nginx_server_requests{code="5xx",host="elk.nakacya.jp"} 0
nginx_server_requests{code="5xx",host="localhost"} 0
nginx_server_requests{code="5xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="5xx",host="www.nakacya.jp"} 2
nginx_server_requests{code="total",host="*"} 33737
nginx_server_requests{code="total",host="elk.nakacya.jp"} 106
nginx_server_requests{code="total",host="localhost"} 33442
nginx_server_requests{code="total",host="test1.nakacya.jp"} 116
nginx_server_requests{code="total",host="www.nakacya.jp"} 73

データ加工

direction のin/out が「指数表記」になっているのは困るので加工します。

ここの 1行perl が優秀

データ加工
#Server Metric GET
/usr/bin/curl -s http://localhost:9913/metrics | grep -v ^# | grep -e "_server_requests" -e "_server_bytes" > /var/log/nginx/vts_status.log

# an exponential -> real number
cat /var/log/nginx/vts_status.log | perl -anle 'printf "%s %d\n",$F[0],$F[1]' > /var/log/nginx/vts_status_number.log
mv -f /var/log/nginx/vts_status_number.log /var/log/nginx/vts_status.log

加工後

/var/log/nginx/vts_status.log
nginx_server_bytes{direction="in",host="*"} 11809961
nginx_server_bytes{direction="in",host="elk.nakacya.jp"} 146149
nginx_server_bytes{direction="in",host="localhost"} 11583208
nginx_server_bytes{direction="in",host="test1.nakacya.jp"} 46807
nginx_server_bytes{direction="in",host="www.nakacya.jp"} 33797
nginx_server_bytes{direction="out",host="*"} 2555083406
nginx_server_bytes{direction="out",host="elk.nakacya.jp"} 332947
nginx_server_bytes{direction="out",host="localhost"} 2554068410
nginx_server_bytes{direction="out",host="test1.nakacya.jp"} 597605
nginx_server_bytes{direction="out",host="www.nakacya.jp"} 84444
nginx_server_requests{code="1xx",host="*"} 0
nginx_server_requests{code="1xx",host="elk.nakacya.jp"} 0
nginx_server_requests{code="1xx",host="localhost"} 0
nginx_server_requests{code="1xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="1xx",host="www.nakacya.jp"} 0
nginx_server_requests{code="2xx",host="*"} 33505
nginx_server_requests{code="2xx",host="elk.nakacya.jp"} 73
nginx_server_requests{code="2xx",host="localhost"} 33312
nginx_server_requests{code="2xx",host="test1.nakacya.jp"} 83
nginx_server_requests{code="2xx",host="www.nakacya.jp"} 37
nginx_server_requests{code="3xx",host="*"} 14
nginx_server_requests{code="3xx",host="elk.nakacya.jp"} 14
nginx_server_requests{code="3xx",host="localhost"} 0
nginx_server_requests{code="3xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="3xx",host="www.nakacya.jp"} 0
nginx_server_requests{code="4xx",host="*"} 219
nginx_server_requests{code="4xx",host="elk.nakacya.jp"} 19
nginx_server_requests{code="4xx",host="localhost"} 133
nginx_server_requests{code="4xx",host="test1.nakacya.jp"} 33
nginx_server_requests{code="4xx",host="www.nakacya.jp"} 34
nginx_server_requests{code="5xx",host="*"} 2
nginx_server_requests{code="5xx",host="elk.nakacya.jp"} 0
nginx_server_requests{code="5xx",host="localhost"} 0
nginx_server_requests{code="5xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="5xx",host="www.nakacya.jp"} 2
nginx_server_requests{code="total",host="*"} 33740
nginx_server_requests{code="total",host="elk.nakacya.jp"} 106
nginx_server_requests{code="total",host="localhost"} 33445
nginx_server_requests{code="total",host="test1.nakacya.jp"} 116
nginx_server_requests{code="total",host="www.nakacya.jp"} 73

出た出た

sedで再加工&Host名でソート
$ cat /var/log/nginx/vts_status.log | sed -e s/\{/./g -e s/\}/" :"/g -e s/\"//g -e s/,/" "/g -e s/" host="/" "/g | sort -k2,2 > /tmp/temp.log
$ cat /tmp/temp.log
nginx_server_bytes.direction=in * : 11810163
nginx_server_bytes.direction=out * : 2555344161
nginx_server_requests.code=1xx * : 0
nginx_server_requests.code=2xx * : 33507
nginx_server_requests.code=3xx * : 14
nginx_server_requests.code=4xx * : 219
nginx_server_requests.code=5xx * : 2
nginx_server_requests.code=total * : 33742
nginx_server_bytes.direction=in elk.nakacya.jp : 146149
nginx_server_bytes.direction=out elk.nakacya.jp : 332947
nginx_server_requests.code=1xx elk.nakacya.jp : 0
nginx_server_requests.code=2xx elk.nakacya.jp : 73
nginx_server_requests.code=3xx elk.nakacya.jp : 14
nginx_server_requests.code=4xx elk.nakacya.jp : 19
nginx_server_requests.code=5xx elk.nakacya.jp : 0
nginx_server_requests.code=total elk.nakacya.jp : 106
nginx_server_bytes.direction=in localhost : 11583410
nginx_server_bytes.direction=out localhost : 2554329165
nginx_server_requests.code=1xx localhost : 0
nginx_server_requests.code=2xx localhost : 33314
nginx_server_requests.code=3xx localhost : 0
nginx_server_requests.code=4xx localhost : 133
nginx_server_requests.code=5xx localhost : 0
nginx_server_requests.code=total localhost : 33447
nginx_server_bytes.direction=in test1.nakacya.jp : 46807
nginx_server_bytes.direction=out test1.nakacya.jp : 597605
nginx_server_requests.code=1xx test1.nakacya.jp : 0
nginx_server_requests.code=2xx test1.nakacya.jp : 83
nginx_server_requests.code=3xx test1.nakacya.jp : 0
nginx_server_requests.code=4xx test1.nakacya.jp : 33
nginx_server_requests.code=5xx test1.nakacya.jp : 0
nginx_server_requests.code=total test1.nakacya.jp : 116
nginx_server_bytes.direction=in www.nakacya.jp : 33797
nginx_server_bytes.direction=out www.nakacya.jp : 84444
nginx_server_requests.code=1xx www.nakacya.jp : 0
nginx_server_requests.code=2xx www.nakacya.jp : 37
nginx_server_requests.code=3xx www.nakacya.jp : 0
nginx_server_requests.code=4xx www.nakacya.jp : 34
nginx_server_requests.code=5xx www.nakacya.jp : 2
nginx_server_requests.code=total www.nakacya.jp : 73

filebeat用に微調整

改行が邪魔なのと日時がないと困るので
・日時追加
・改行コード削除

$ date "+%Y/%m/%dT%H:%M:%S" |sed -z 's/\n/ /g' && cat /tmp/temp.log | sed -z 's/\n/ /g' |sed -e "s/\$/\n/"
2020/02/05T04:04:33 nginx_server_bytes.direction=in * : 11810163 nginx_server_bytes.direction=out * : 2555344161 nginx_server_requests.code=1xx * : 0 nginx_server_requests.code=2xx * : 33507 nginx_server_requests.code=3xx * : 14 nginx_server_requests.code=4xx * : 219 nginx_server_requests.code=5xx * : 2 nginx_server_requests.code=total * : 33742

最後に

これを filebeat に食わせて ELK の Logstash で Grok に食わせればデータの完成
Grok はこんな感じ

Grok
^%{DATA:Date} nginx_server_bytes.direction=in %{DATA:Host} : %{DATA:In-Byte} nginx_server_bytes.direction=out .*: %{DATA:Out_Byte} nginx_server_cache.status=bypass .*: %{NUMBER:Cache_Bypass} nginx_server_cache.status=expired .*: %{NUMBER:Cache_Expired} nginx_server_cache.status=hit .*: %{NUMBER:Cache_Hit} nginx_server_cache.status=miss .*: %{NUMBER:Cache_Miss} nginx_server_cache.status=revalidated .*: %{NUMBER:Cache_Revalidated} nginx_server_cache.status=scarce .*: %{NUMBER:Cache_Scarce} nginx_server_cache.status=stale .*: %{NUMBER:Cache_Stale} nginx_server_cache.status=updating .*: %{NUMBER:Cache_Updating} nginx_server_requests.code=1xx .*: %{NUMBER:Requests_Code_1xx} nginx_server_requests.code=2xx .*: %{NUMBER:Requests_Code_2xx} nginx_server_requests.code=3xx .*: %{NUMBER:Requests_Code_3xx} nginx_server_requests.code=4xx .*: %{NUMBER:Requests_Code_4xx} nginx_server_requests.code=5xx .*: %{NUMBER:Requests_Code_5xx} nginx_server_requests.code=total .*: %{NUMBER:Requests_Code_Total}
1
1
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
1
1

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?