設定メモ Fluentd+Amazon Elasticserch+Kibana4+ELB s3 Log

準備

# AWS Ubuntu 14.04 LTS/Trusty
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh
$ sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch
$ sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elb-log

参照: http://docs.fluentd.org/articles/install-by-deb

設定

$ sudo vi /etc/td-agent/td-agent.conf
  <source>
    type elb_log
    access_key_id xxxxxx
    secret_access_key xxxxxx
    region ap-northeast-1
    s3_bucketname xxxxxxx
    s3_prefix xxxxxxxx
    timestamp_file /tmp/elb_last_at.dat
    buf_file /tmp/fluentd-elblog.tmpfile
    refresh_interval 300
    tag elb.access
  </source>
  <match **>
    type elasticsearch
    type_name access_log
    host xxxxxxx.ap-northeast-1.es.amazonaws.com
    port 80
    logstash_format true
    include_tag_key true
    tag_key @log_name
    buffer_type file
    buffer_chunk_limit 5m
    buffer_queue_limit 1280
    flush_interval 5s
    buffer_path /var/log/td-agent/buffer/access.buffer
    disable_retry_limit false
    retry_limit 17
    retry_wait 1s
  </match>
</ROOT>

• portは9200じゃなくて80で通る
• Amazon Elasticsearch のインスタンスサイズによって一度にアップできるpayloadのサイズが限られてい https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html

Each supported instance type has a maximum supported payload for HTTP requests:
t2.micro.elasticsearch: 10 MB
t2.small.elasticsearch: 10 MB
t2.medium.elasticsearch: 10 MB
m3.medium.elasticsearch: 10 MB
m3.large.elasticsearch: 10 MB
m3.xlarge.elasticsearch: 100 MB
m3.2xlarge.elasticsearch: 100 MB
r3.large.elasticsearch: 100 MB
r3.xlarge.elasticsearch: 100 MB
r3.2xlarge.elasticsearch: 100 MB
r3.4xlarge.elasticsearch: 100 MB
r3.8xlarge.elasticsearch: 100 MB
i2.xlarge.elasticsearch: 100 MB
i2.2xlarge.elasticsearch: 100 MB

起動

$ sudo /etc/init.d/td-agent start