Help us understand the problem. What is going on with this article?

Fluentd + Elasticsearch + Kibana を使って YAMAHA RTX1100 の syslog をそれっぽく可視化する

More than 3 years have passed since last update.

概要

fluentd + elasticsearch + kibanaを用いて、ルーターのパケットフィルタで落としたIPのログなどを可視化してみた。
fluentdのプラグイン(fluent-plugin-geoip)を使ってIPから位置情報を取得できた。

image

貼り付けた画像_2016_02_25_2_35.jpg

設定

前提条件

  • ルータ(RTX1100)IP: 192.168.1.1
  • サーバ
    • OS: Ubuntu 14.04.3 LTS Trusty
    • IP: 192.168.1.100
    • kibana, fluentd, elasticsearchを動かす
    • LANからのアクセスのみ想定故、サーバのFWはガラ空き

パッケージインストール

公式手順参考

Apache2リバースプロキシ設定

elasticsearchとkibanaを起動すると、それぞれ5601番と9200番ポートで待ち受けるので、便宜上、適当にリバプロ設定。

<Location /kibana>
        Require ip 127 192.168.1
        ProxyPass http://localhost:5601
        ProxyPassReverse http://localhost:5601
</Location>

<Location /elasticsearch>
        Require ip 127 192.168.1
        ProxyPass http://localhost:9200
        ProxyPassReverse http://localhost:9200
</Location>

RTX1100コンフィグ変更

サーバにルータのsyslogをぶっ飛ばす設定を追加

#
# SYSLOG configuration
#
syslog local address 192.168.1.1
syslog host 192.168.1.100
syslog info on
syslog notice on

elasticsearch設定

elasticsearchが自動的に文字列を要素解析するらしく、
kibanaで可視化する際に「not analyzedにしないとパフォーマンス悪くなるぜ」とか仰るので、not analyzedなfieldも用意する。
あと、地理情報を入れるフィールド(geo_location)の型をgeo_pointにしておく。

$ curl -XPUT 192.168.1.100:9200/rtx1100-*/ -d "`cat elasticsearch_default_template.json`"

# lasticsearch_default_template.json
{
    "template": "rtx1100-*",
    "mappings": {
        "_default_": {
            "dynamic_templates": [
                {
                    "string_template" : {
                        "match" : "*",
                        "mapping": {
                            "type": "string",
                            "fields": {
                                "full": {
                                    "type": "string",
                                    "index": "not_analyzed"
                                }
                            }
                        },
                        "match_mapping_type": "string"
                    }
                }
            ],
            "properties": {
                "@timestamp": { "type": "date", "index": "not_analyzed" },
                "geo_location": {"type" : "geo_point" }
            }
        }
    }
}

fluentd設定

fluentdプラグインインストール

以下のプラグインを使用。

fluentd設定ファイル

  • RTX1100から流れてきたsyslogを正規表現でマッチさせて分解
    • マッチしなかったそれ以外の形式のログは、分解しない
  • ログの種類ごとに(elasticsearchの)indexを分けて格納
# 動的フィルタ => index: rtx1100-inspect
- [INSPECT] PP[01][out][105] TCP xxx.xxx.xxx.xxx:xxxxx > yyy.yyy.yyy.yyy:yyyyy (yyyy/MM/dd hh:mm:ss)

# フィルタリジェクト => index: rtx1100-reject
# リジェクト系だけはfluent-plugin-geoipで発信源の地理情報を取得する
- PP[01] Rejected at IN(xxxx) filter: TCP xxx.xxx.xxx.xxx:xxxxx > yyy.yyy.yyy.yyy:yyyyy

# コンソールログイン/ログアウト => index: rtx1100-console
- Login succeeded for TELNET: xxx.xxx.xxx.xxx
- Logout from TELNET: xxx.xxx.xxx.xxx

# VPN接続/切断 => index: rtx1100-tunnel
- [L2TP] TUNNEL[1] connected from xxx.xxx.xxx.xxx
- [L2TP] TUNNEL[1] disconnect tunnel xxxxx complete

# その他 => rtx1100-other
# /etc/td-agent/td-agent.conf

####
## Source descriptions:
##

## syslog
<source>
  type syslog
  tag raw.rtx1100
  format none
</source>

####
## Output descriptions:
##

<match raw.rtx1100.**>
  type parser
  format multi_format
  key_name message
  remove_prefix raw
  add_prefix parsed
  <pattern>
      format with_extra_fields
      base_format /^\[INSPECT\]\s+(?<target>.+)\[(?<direction>.+)\]\[(?<filter_num>\d+)\]\s+(?<proto>.+)\s+(?<src_ip>.+):(?<src_port>.+)\s+>\s+(?<dest_ip>.+):(?<dest_port>.+)\s+\((?<time>.+)\)$/
      time_format '%Y/%m/%d %H:%M:%S'
      extra_fields { "log_type": "inspect" }
  </pattern>
  <pattern>
      format with_extra_fields
      base_format /^(?<target>.+)\s+Rejected\s+at\s+(?<direction>.+)\((?<filter_num>\d+)\)\s+filter:\s+(?<proto>.+)\s+(?<src_ip>.+):(?<src_port>.+)\s+>\s+(?<dest_ip>.+):(?<dest_port>.+)$/ 
      extra_fields { "log_type": "reject" }
  </pattern>
  <pattern>
      format with_extra_fields
      base_format /^Logout\s+from\s+(?<proto>.+):\s+(?<ip>.+)$/
      extra_fields { "log_type": "console_logout" }
  </pattern>
  <pattern>
      format with_extra_fields
      base_format /^Login\s+succeeded\s+for\s+(?<proto>.+):\s+(?<ip>.+)$/ 
      extra_fields { "log_type": "console_login" }
  </pattern>
  <pattern>
      format with_extra_fields
      base_format /^\[(?<proto>.+)\]\s+(?<tunnel>.+)\s+connected\s+from\s+(?<src_ip>.+)$/ 
      extra_fields { "log_type": "tunnel_connect" }
  </pattern>
  <pattern>
      format with_extra_fields
      base_format /^\[(?<proto>.+)\]\s+(?<tunnel>.+)\s+disconnect\s+tunnel\s+\d+\s+complete$/ 
      extra_fields { "log_type": "tunnel_disconnect" }
  </pattern>  
  <pattern>
      format with_extra_fields
      base_format /^(?<msg>.+)$/
      extra_fields { "log_type": "other" }
  </pattern>  
</match>

<match parsed.rtx1100.**>
  type rewrite_tag_filter
  rewriterule1 log_type   ^inspect$       rtx1100.inspect
  rewriterule2 log_type   ^reject$        temp.rtx1100.reject
  rewriterule3 log_type   ^console_(.+)$  rtx1100.console.$1
  rewriterule4 log_type   ^tunnel_(.+)$   rtx1100.tunnel.$1
  rewriterule5 log_type   ^other$         rtx1100.other
</match>

<match rtx1100.inspect.**>
  type elasticsearch
  logstash_format true
  logstash_prefix rtx1100-inspect
  include_tag_key true
  tag_key @log_name
  hosts localhost:9200
  buffer_type memory
  num_threads 1
  flush_interval 60
  retry_wait 1.0
  retry_limit 17
</match>

<match temp.rtx1100.reject.**>
  type  geoip
  geoip_lookup_key src_ip
  <record>
    geo_location  '{ "lat" : ${latitude["src_ip"]}, "lon" : ${longitude["src_ip"]} }'
    country_code  ${country_code["src_ip"]}
  </record>
  remove_tag_prefix temp.
  skip_adding_null_record  true
  flush_interval 1s
</match>
<match rtx1100.reject.**>
  type elasticsearch
  logstash_format true
  logstash_prefix rtx1100-reject
  include_tag_key true
  tag_key @log_name
  hosts localhost:9200
  buffer_type memory
  num_threads 1
  flush_interval 60
  retry_wait 1.0
  retry_limit 17
</match>

<match rtx1100.console.**>
  type elasticsearch
  logstash_format true
  logstash_prefix rtx1100-console
  include_tag_key true
  tag_key @log_name
  hosts localhost:9200
  buffer_type memory
  num_threads 1
  flush_interval 60
  retry_wait 1.0
  retry_limit 17
</match>

<match rtx1100.tunnel.**>
  type elasticsearch
  logstash_format true
  logstash_prefix rtx1100-tunnel
  include_tag_key true
  tag_key @log_name
  hosts localhost:9200
  buffer_type memory
  num_threads 1
  flush_interval 60
  retry_wait 1.0
  retry_limit 17
</match>

<match rtx1100.other.**>
  type elasticsearch
  logstash_format true
  logstash_prefix rtx1100-other
  include_tag_key true
  tag_key @log_name
  hosts localhost:9200
  buffer_type memory
  num_threads 1
  flush_interval 60
  retry_wait 1.0
  retry_limit 17
</match>

参考文献

Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
Comments
No comments
Sign up for free and join this conversation.
If you already have a Qiita account
Why do not you register as a user and use Qiita more conveniently?
You need to log in to use this function. Qiita can be used more conveniently after logging in.
You seem to be reading articles frequently this month. Qiita can be used more conveniently after logging in.
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
ユーザーは見つかりませんでした