fluent-bit で log file を tail して elasticsearch に送る

初めて fluent-bit を使ってみた。軽量な fluentd だそうです。pluginの追加あたりで躓いたので、メモっておく

install fluent-bit

Ref https://docs.fluentbit.io/manual/installation/linux/redhat-centos#configure-yum

repoを登録する

/etc/yum.repos.d/fluent-bit.repo

[fluent-bit]
name = Fluent Bit
baseurl = https://packages.fluentbit.io/centos/$releasever/$basearch/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
repo_gpgcheck=1
enabled=1

install

$ yum install fluent-bit

systemctl status fluent-bit

config

elasticsearchに出力するpluginを有効にします

/etc/fluent-bit/plugins.conf

[PLUGINS]
    @INCLUDE output-elasticsearch.conf             <--- 追加
    # Path /path/to/out_gstdout.so

defaultのconfigを書き換えます

/etc/fluent-bit/fluent-bit.conf

[SERVICE]
    daemon       Off
    log_level    debug
    plugins_file plugins.conf

# https://docs.fluentbit.io/manual/pipeline/inputs/tail
[INPUT]
    Name   tail
    Path   /path/to/your/log

# https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch
[OUTPUT]
    Name  es
    Match *
    Host  <your.elasticsearch.endpoint>
    Port  9200
    HTTP_User <es_user>
    HTTP_Passwd <es_password>
    tls   On
    tls.verify  On
    Index <your-index-name>
    Type  _doc

動作確認

start

systemctl restart fluent-bit
systemctl enable fluent-bit

date > /path/to/your/log

kibanaのdevtoolsとかで叩いてみる

GET <your-index-name>/_search
{
  "query": {
    "match_all": {}
  }
}

Voilà!