Elasticsearch with Docker から Kibana へ (NetFlowの可視化) #docker-compose

Fluentd で取得した NetFlow v5 のデータを Kibana で表示するための環境を Docker を使って構築します。

1. 前段として、dockerとdocker−composeは入れておく

2. メモリマップの上限値を上げておく

/etc/sysctl.conf

vm.max_map_count=262144

3. docker-compose.yml を以下のように記述。

~/neflow-compose/docker-compose.yml

fluentd:
  build: fluentd
  ports:
    - 24224:24224
    - 24224:24224/udp
  volumes:
    - ./fluentd:/fluentd/etc
    - ./fluentd/plugins:/fluentd/plugins
  links:
      - elasticsearch

elasticsearch:
  image: elasticsearch:5.6.4
  ports:
    - 9200:9200
    - 9300:9300
  expose:
    - 9300

kibana:
  image: kibana:5.6.4
  ports:
    - 5601:5601
  environment:
      - ELASTICSEARCH_URL=http://172.17.0.2:9200
#      - ELASTICSEARCH_URL=http://elasticsearch:9200
  links:
      - elasticsearch

4. fluentd から elastic search へ送る設定を作る

~/netflow-compose/fluentd/Dockerfile

FROM fluent/fluentd:latest-onbuild

#USER fluent

WORKDIR ~/netflow-compose/fluentd
ENV PATH ~/.gem/ruby/2.3.0/bin:$PATH
RUN apk add --no-cache --update --virtual=build-dependencies ruby-dev build-base && \
  gem install fluent-plugin-rewrite-tag-filter && \
  apk del build-dependencies && \
  rm -rf /tmp/* /var/tmp/* /var/cache/apt/*
RUN gem install fluent-plugin-netflow
RUN gem install fluent-plugin-secure-forward
RUN gem install fluent-plugin-elasticsearch # Elasticsearch連携

#EXPOSE 24284
EXPOSE 24224

CMD fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins -vv

~/netflow-compose/fluentd/fluent.conf

<source>
  type netflow
  tag netflow.event
  port 24224
</source>
<match netflow.event>
  @type copy
  <store>
    @type file
    path         /fluentd/log/data.*.log
    symlink_path /fluentd/log/data.log
    append       true
  </store>
  <store>
    @type elasticsearch
#    host localhost
#    host elasticsearch
    host 172.17.0.2
    port 9200
    type_name netflow
    logstash_format true
    logstash_prefix flow
    logstash_dateformat %Y%m%d
  </store>
</match>

5. あとは、docker-compose up する

$ sudo docker-compose up

起動時ログが落ち着いたら、Kibanaを開いてみる

参考リンク

【Docker】FluentdでとElasticsearchとRe:dashでログ管理
https://qiita.com/narumi_/items/59237554ea1c0e713dd9

・Docker 公式
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html
http://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html#docker-prod-cluster-composefile

・enqueueing all chunks in bufferについて
https://qiita.com/tatsu-yam/items/bd7006e483f3b3c64309

・fluentdのRuby導入についてはこのへんを参考に、
http://cross-black777.hatenablog.com/entry/2017/11/15/190000