Edited at

設定メモ Fluentd+Amazon Elasticserch+Kibana4+ELB s3 Log

More than 3 years have passed since last update.


準備

# AWS Ubuntu 14.04 LTS/Trusty

$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh
$ sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch
$ sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elb-log

参照: http://docs.fluentd.org/articles/install-by-deb


設定

$ sudo vi /etc/td-agent/td-agent.conf

<source>
type elb_log
access_key_id xxxxxx
secret_access_key xxxxxx
region ap-northeast-1
s3_bucketname xxxxxxx
s3_prefix xxxxxxxx
timestamp_file /tmp/elb_last_at.dat
buf_file /tmp/fluentd-elblog.tmpfile
refresh_interval 300
tag elb.access
</source>
<match **>
type elasticsearch
type_name access_log
host xxxxxxx.ap-northeast-1.es.amazonaws.com
port 80
logstash_format true
include_tag_key true
tag_key @log_name
buffer_type file
buffer_chunk_limit 5m
buffer_queue_limit 1280
flush_interval 5s
buffer_path /var/log/td-agent/buffer/access.buffer
disable_retry_limit false
retry_limit 17
retry_wait 1s
</match>
</ROOT>

Each supported instance type has a maximum supported payload for HTTP requests:

t2.micro.elasticsearch: 10 MB
t2.small.elasticsearch: 10 MB
t2.medium.elasticsearch: 10 MB
m3.medium.elasticsearch: 10 MB
m3.large.elasticsearch: 10 MB
m3.xlarge.elasticsearch: 100 MB
m3.2xlarge.elasticsearch: 100 MB
r3.large.elasticsearch: 100 MB
r3.xlarge.elasticsearch: 100 MB
r3.2xlarge.elasticsearch: 100 MB
r3.4xlarge.elasticsearch: 100 MB
r3.8xlarge.elasticsearch: 100 MB
i2.xlarge.elasticsearch: 100 MB
i2.2xlarge.elasticsearch: 100 MB


起動

$ sudo /etc/init.d/td-agent start