Elasticsearch
Logstash
beats
kibana5

ELK5.2 + beats + jdbc による可視化 - 1

ELK 基本インストール編

  • ELK 公式マニュアルトップ: https://www.elastic.co/guide/index.html
  • 対象バージョン
    • elasticsearch 5.2.x
    • logstash 5.2.x
    • kibana 5.2.x
  • 事例ユースケース
    • windows server のイベントログ収集・集計・可視化
      • ざっくりモデル
        • winlogbeat -> logstash logbeat plugin + logstash -> elasticsearch -> kibana <- browser
    • RDBへのクエリ実行結果取得・集計・可視化
      • ざっくりモデル
        • RDB <- logstash jdbc plugin + logstash -> elasticsearch -> kibana <- browser
  • 事例環境
    • logstash, kibana @ debian jessie
    • winlogbeat, elasticsearch @ windows server 2012 R2

前提環境

  • curl インストール
$ apt-get install curl
  • oracle java8 インストール
## https://tecadmin.net/install-java-8-on-debian/
$ echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | sudo tee -a /etc/apt/sources.list.d/java-8-debian.list
$ echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | sudo tee -a /etc/apt/sources.list.d/java-8-debian.list
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer
$ java -version

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode, sharing)
$
  • ファイルディスクリプタのキャパ確認
## https://www.digitalocean.com/community/tutorials/how-to-setup-additional-entropy-for-cloud-servers-using-haveged
## 仮想マシン環境の場合、1024の場合があり、場合によりエントロピー枯渇する。65535ならOK。1024なら増やす。
## 増やさずにこのような解決法もあるようだけど、試していない。 http://inokara.hateblo.jp/entry/2014/03/08/083548
$ ulimit -n
1024
$ sudo aptitude install haveged
$ sudo reboot
## after relogin
$ ulimit -n
65535
  • ELK 公式パッケージリポジトリ利用準備
## logstash, elasticsearch, kibana 共通で必要な手順
$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ sudo apt-get install apt-transport-https
$ echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
$ sudo apt-get update
$
  • logstash インストール
## https://www.elastic.co/guide/en/logstash/current/installing-logstash.html#package-repositories
$ sudo apt-get install logstash
  • logstash 自動起動&起動操作
## detail: https://www.elastic.co/guide/en/logstash/current/running-logstash.html
$ sudo systemctl enable logstash.service
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.
$
$ sudo systemctl start logstash.service
$ ps -ef | grep logstash
<USER>       3257   713  0 13:12 pts/0    00:00:00 grep logstash
$ sudo systemctl start logstash.service
$ ps -ef | grep logstash
logstash  3272     1 99 13:12 ?        00:00:11 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
<USER>       3304   713  0 13:12 pts/0    00:00:00 grep logstash
$
$ sudo systemctl stop logstash.service
$ ps -ef | grep logstash
<USER>       3483   713  0 13:13 pts/0    00:00:00 grep logstash
$
  • ダミーパイプライン設定ファイル配置 (とりあえず起動させるためのミニマムダミー設定)
## 基本: https://www.elastic.co/guide/en/logstash/current/configuration.html
$ sudo vi /etc/logstash/conf.d/logstash-simple.conf
$ sudo cat /etc/logstash/conf.d/logstash-simple.conf
input { stdin { } }
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

$
  • logstash.yml 設定
## detail: https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html
$ sudo cp -p /etc/logstash/logstash.yml /etc/logstash/logstash.yml.org
$ sudo vi /etc/logstash/logstash.yml
$ diff -C1 logstash.yml.org logstash.yml
*** logstash.yml.org    2017-04-28 13:11:30.660159347 +0900
--- logstash.yml        2017-09-04 18:08:04.558868363 +0900
***************
*** 77,78 ****
--- 77,79 ----
  # config.reload.automatic: false
+ config.reload.automatic: true
  #
***************
*** 80,82 ****
  #
! # config.reload.interval: 3
  #
--- 81,83 ----
  #
! config.reload.interval: 30
  #
***************
*** 138,139 ****
--- 139,141 ----
  # http.host: "127.0.0.1"
+ http.host: "0.0.0.0"
  #
***************
*** 143,144 ****
--- 145,147 ----
  # http.port: 9600-9700
+ http.port: 9600-9700
  #
***************
*** 155,156 ****
--- 158,160 ----
  # log.level: info
+ log.level: warn
  path.logs: /var/log/logstash
$
$ sudo systemctl enable logstash.service
$ sudo service logstash start
$ ps -ef | grep logstash
logstash  1834     1 48 14:01 ?        00:00:19 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
$

elasticsearch インストール

今回の構成では、windows server に elasticsearch のインストーラで導入しましたが、
GUIオペレーションを貼ってもしょうがないので、当セクションには debian の場合の手順を記述しています。
読み替えて導入して下さい。

debianの場合
## https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html
$ sudo apt-get install elasticsearch
  • elasticsearch 自動起動&起動操作

windows server の場合はインストールした elasticsearch の root ディレクトリに
サービス登録スクリプトがあるので叩いて登録します。

debianの場合
## detail: https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html#deb-running-systemd
## https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html
$ sudo systemctl enable elasticsearch.service
Synchronizing state for elasticsearch.service with sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d elasticsearch defaults
Executing /usr/sbin/update-rc.d elasticsearch enable
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
$
$ sudo systemctl start elasticsearch.service
$ ps -ef | grep elasticsearch
elastic+  7257     1 99 13:36 ?        00:00:08 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.2.2.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch
USER       7321   818  0 13:36 pts/0    00:00:00 grep elasticsearch
$
$ sudo systemctl stop elasticsearch.service
$ ps -ef | grep elasticsearch
USER       7539   818  0 13:37 pts/0    00:00:00 grep elasticsearch
$
  • elasticsearch.yml 設定
debianの場合
$ sudo cp -p /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.org
$ sudo vi /etc/elasticsearch/elasticsearch.yml
$ sudo diff -C1 /etc/elasticsearch/elasticsearch.yml.org /etc/elasticsearch/elasticsearch.yml
*** elasticsearch.yml.org       2017-02-25 02:29:35.000000000 +0900
--- elasticsearch.yml   2017-03-04 14:55:11.303489207 +0900
***************
*** 32,34 ****
  #
! #path.data: /path/to/data
  #
--- 32,34 ----
  #
! path.data: /var/lib/elasticsearch
  #
***************
*** 36,38 ****
  #
! #path.logs: /path/to/logs
  #
--- 36,38 ----
  #
! path.logs: /var/log/elasticsearch
  #
***************
*** 54,56 ****
  #
! #network.host: 192.168.0.1
  #
--- 54,56 ----
  #
! network.host: 0.0.0.0
  #
***************
*** 58,60 ****
  #
! #http.port: 9200
  #
--- 58,60 ----
  #
! http.port: 9200
  #
***************
*** 67,69 ****
  #
! #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
  #
--- 67,69 ----
  #
! discovery.zen.ping.unicast.hosts: ["OWN_HOSTNAME:9300", "OWN_HOSTNAME", "localhost"]
  #
***************
*** 88 ****
--- 88,90 ----
  #action.destructive_requires_name: true
+
+ http.cors.enabled: true
$
$ sudo systemctl start elasticsearch.service
$ 
$ curl http://localhost:9200
{
  "name" : "node-1",
  "cluster_name" : "my-application",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "5.2.2",
    "build_hash" : "f9d9b74",
    "build_date" : "2017-02-24T17:26:45.835Z",
    "build_snapshot" : false,
    "lucene_version" : "6.4.1"
  },
  "tagline" : "You Know, for Search"
}
$

kibana インストール

kibana を導入します

## package インストール
## https://www.elastic.co/guide/en/kibana/current/deb.html
$ sudo apt-get install kibana
  • kibana 自動起動&起動操作
$ sudo systemctl enable kibana.service
Synchronizing state for kibana.service with sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d kibana defaults
Executing /usr/sbin/update-rc.d kibana enable
$ sudo systemctl start kibana.service
$ ps -ef | grep kibana
kibana   27373     1 50 15:48 ?        00:00:02 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
USER      27385   774  0 15:48 pts/0    00:00:00 grep kibana
$
$ sudo systemctl stop kibana.service
$ ps -ef | grep kibana
USER       1043   909  0 16:21 pts/0    00:00:00 grep kibana
$
  • kibana.yml 設定
$ sudo cp -p /etc/kibana/kibana.yml /etc/kibana/kibana.yml.org
$ sudo vi /etc/kibana/kibana.yml
$ sudo diff -C1 /etc/kibana/kibana.yml.org /etc/kibana/kibana.yml
*** kibana.yml.org      2017-05-22 21:13:19.000000000 +0900
--- kibana.yml  2017-09-19 16:57:10.065487855 +0900
***************
*** 1,3 ****
  # Kibana is served by a back end server. This setting specifies the port to use.
! #server.port: 5601

--- 1,3 ----
  # Kibana is served by a back end server. This setting specifies the port to use.
! server.port: 5601

***************
*** 6,8 ****
  # To allow connections from remote users, set this parameter to a non-loopback address.
! #server.host: "localhost"

--- 6,8 ----
  # To allow connections from remote users, set this parameter to a non-loopback address.
! server.host: "0.0.0.0"

***************
*** 17,22 ****
  # The Kibana server's name.  This is used for display purposes.
! #server.name: "your-hostname"

  # The URL of the Elasticsearch instance to use for all your queries.
! #elasticsearch.url: "http://localhost:9200"

--- 17,22 ----
  # The Kibana server's name.  This is used for display purposes.
! server.name: "OWN_HOSTNAME"

  # The URL of the Elasticsearch instance to use for all your queries.
! elasticsearch.url: "http://ELASTICSEARCH_IPADDR:9200"

***************
*** 43,45 ****
  # These settings enable SSL for outgoing requests from the Kibana server to the browser.
! #server.ssl.enabled: false
  #server.ssl.certificate: /path/to/your/server.crt
--- 43,45 ----
  # These settings enable SSL for outgoing requests from the Kibana server to the browser.
! server.ssl.enabled: false
  #server.ssl.certificate: /path/to/your/server.crt
***************
*** 58,59 ****
--- 58,60 ----
  #elasticsearch.ssl.verificationMode: full
+ elasticsearch.ssl.verificationMode: none

***************
*** 85,87 ****
  # Enables you specify a file where Kibana stores log output.
! #logging.dest: stdout

--- 86,88 ----
  # Enables you specify a file where Kibana stores log output.
! logging.dest: /var/log/kibana/kibana.log

$
$ sudo systemctl enable kibana.service

winlogbeat インストール

winlogbeat を利用して windows サーバの event log を取得し logstash へ送出する設定をします。

winlogbeat.yml
#======================= Winlogbeat specific options ==========================

# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig
winlogbeat.event_logs:
  - name: Application
    ignore_older: 1h
    level: critical, error, warning
  - name: Security
    ignore_older: 1h
    level: critical, error, warning, information
  - name: System
    ignore_older: 1h
    level: critical, error, warning

logstash サーバ pipeline 設定ファイル

  • logstash で受けた情報をどのプラグインでどのように整形して elasticsearch へ送り出すか。といった内容を /etc/logstash/conf.d/*.conf に記述する。
  • 今回の例では、 logstash が受けた情報を beats の input プラグインで受けて、elasticsearch へ output する config を書くことになる。
$ cd /usr/share/logstash
$ sudo ./bin/logstash-plugin install logstash-input-beats
$ sudo service logstash restart
$ 
$ sudo touch /etc/logstash/conf.d/01-winlogbeat.conf
$ sudo vi /etc/logstash/conf.d/01-winlogbeat.conf
$ sudo cat /etc/logstash/conf.d/01-winlogbeat.conf
input {
  beats {
    port => 5044
    client_inactivity_timeout => 3600
  }
}

output {
  elasticsearch {
    hosts => "ELASTICSEARCH_HOST:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

$
$ sudo service logstash restart
$

具体的なログ収集・集計・可視化の設定例は次記事にて。
つづく