fluentdで軽くs3にログを集積してみる(sinatra編)

fluentdのtailプラグインを使ってログを収集してみるを参考に、sinatraでログ集積を試します。

最低限の事だけまとめました。
必要に応じて付け足して行きます。

必要なファイルを揃える

app01.rb

app01.rbは単純なsinatraアプリケーションです。
sinatra01.logにログを溜めます。

require "sinatra"
require 'logger'

logger = Logger.new('sinatra01.log')

get '/' do
  logger.info "log from 01"
  'Hello app 01'
end

app02.rb

app01.rbと同様です。

require "sinatra"
require 'logger'

logger = Logger.new('sinatra02.log')

get '/' do
  logger.info "log from 02"
  'Hello app 02'
end

conf/sinatra01.conf (app01.rbのconfigファイル)

app01.rbのログ(sinatra01.log)をtailで読み込み、ログサーバーにforwardします。

<source>
  type tail
  path /path/to/sinatra01.log
  format /^(?<date>[^ ]+)\s+(?<duration>.*) (?<client address>.*) (?<result code>.*) (?<bytes>.*) (?<request method>.*) (?<url>.*) (?<rfc931>.*) (?<hierarchy code>.*) (?<type>.*)$/
  pos_file /var/log/fluent/messages.pos
  tag sinatra
</source>

<match sinatra>
  type forward
  send_timeout 60s
  recover_wait 10s
  heartbeat_interval 100s
  phi_threshold 32
  format json
  time_slice_format %Y%m%d%H
  hard_timeout 60s
  buffer_chunk_limit 8*1024*1024
  buffer_queue_limit 256
  retry_wait 1s
  retry_limit 10

  <server>
    name log.server
    host 0.0.0.0
    port 12345
  </server>
</match>

conf/sinatra02.conf (app02.rbのconfigファイル)

conf/sinatra01.confと同様です。

<source>
  type tail
  path /path/to/sinatra02.log
  format /^(?<date>[^ ]+)\s+(?<duration>.*) (?<client address>.*) (?<result code>.*) (?<bytes>.*) (?<request method>.*) (?<url>.*) (?<rfc931>.*) (?<hierarchy code>.*) (?<type>.*)$/
  pos_file /var/log/fluent/messages.pos
  tag sinatra
</source>

<match sinatra>
  type forward
  send_timeout 60s
  recover_wait 10s
  heartbeat_interval 100s
  phi_threshold 32
  format json
  time_slice_format %Y%m%d%H
  hard_timeout 60s
  buffer_chunk_limit 8*1024*1024
  buffer_queue_limit 256
  retry_wait 1s
  retry_limit 10

  <server>
    name log.server
    host 0.0.0.0
    port 12345
  </server>
</match>

conf/log.conf (ログサーバーのconfig)

sinatra01.conf, sinatra02.confから立ち上げたfluentdからforwardされたログを受け取り、s3にアップロードします。

matchディレクティブには3種類のアウトプットを指定しています。
デバッグも兼ねてmatchディレクティブの正規表現は全てに一致するようになっていますが、適宜変更してください。

• type stdout
  デバッグ用

• type file
  ファイルに書き込めているか確認用

• type s3
  s3にあげる用

<source>
  type forward
  port 12345
  bind 0.0.0.0
</source>

<match *>
  type copy
  <store>
    type file
    path /var/log/fluent/sinatra.log
    buffer_path /var/log/fluent/sinatra/
    time_slice_format %Y%m%d
    time_format %Y%m%dT%H%M%S%z
  </store>
  <store>
    type s3
    aws_key_id xxxxxxxx
    aws_sec_key xxxxxxxx
    s3_bucket xxxxxxxx
    s3_endpoint  xxxxxxxx
    path logs/td-agent/sinatra
    format json
    time_slice_format %Y%m%d%H
    flush_interval 1s
    append true
    buffer_path /var/log/td-agent/sinatra
    buffer_chunk_limit 8*1024*1024
    buffer_queue_limit 256
    retry_wait 1s
    retry_limit 5
  </store>
  <store>
    type stdout
  </store>
</match>

サーバーの起動

アプリケーションサーバー(×2台)を起動

$ ruby app01.rb -p 1111

$ ruby app02.rb -p 2222

それぞれのアプリケーションサーバーに対するfluentdの起動

$ fluentd -c config/sinatra01.conf

$ fluentd -c config/sinatra02.conf

ログサーバーの起動

$ fluentd -c config/log.conf

結果

stdout

2014-07-07 15:55:49 +0900 [info]: listening fluent socket on 0.0.0.0:12345
2014-07-07 15:55:59 +0900 sinatra: {"date":"I,","duration":"[2014-07-07T15:55:58.408192","client address":"#53359]","result code":"","bytes":"INFO","request method":"--","url":":","rfc931":"log","hierarchy code":"from","type":"01"}
2014-07-07 15:55:59 +0900 sinatra: {"date":"I,","duration":"[2014-07-07T15:55:59.376684","client address":"#53645]","result code":"","bytes":"INFO","request method":"--","url":":","rfc931":"log","hierarchy code":"from","type":"02"}
2014-07-07 15:56:12 +0900 sinatra: {"date":"I,","duration":"[2014-07-07T15:56:11.495798","client address":"#53645]","result code":"","bytes":"INFO","request method":"--","url":":","rfc931":"log","hierarchy code":"from","type":"02"}
2014-07-07 15:56:15 +0900 sinatra: {"date":"I,","duration":"[2014-07-07T15:56:14.386984","client address":"#53359]","result code":"","bytes":"INFO","request method":"--","url":":","rfc931":"log","hierarchy code":"from","type":"01"}
2014-07-07 15:56:21 +0900 sinatra: {"date":"I,","duration":"[2014-07-07T15:56:20.757525","client address":"#53359]","result code":"","bytes":"INFO","request method":"--","url":":","rfc931":"log","hierarchy code":"from","type":"01"}

ファイルへの書き込み

$ cat sinatra.20140707.b4fd94dc2cd24b79d.log
20140707T065559+0000	sinatra	{"date":"I,","duration":"[2014-07-07T15:55:58.408192","client address":"#53359]","result code":"","bytes":"INFO","request method":"--","url":":","rfc931":"log","hierarchy code":"from","type":"01"}
20140707T065559+0000	sinatra	{"date":"I,","duration":"[2014-07-07T15:55:59.376684","client address":"#53645]","result code":"","bytes":"INFO","request method":"--","url":":","rfc931":"log","hierarchy code":"from","type":"02"}
20140707T065612+0000	sinatra	{"date":"I,","duration":"[2014-07-07T15:56:11.495798","client address":"#53645]","result code":"","bytes":"INFO","request method":"--","url":":","rfc931":"log","hierarchy code":"from","type":"02"}
20140707T065615+0000	sinatra	{"date":"I,","duration":"[2014-07-07T15:56:14.386984","client address":"#53359]","result code":"","bytes":"INFO","request method":"--","url":":","rfc931":"log","hierarchy code":"from","type":"01"}
20140707T065621+0000	sinatra	{"date":"I,","duration":"[2014-07-07T15:56:20.757525","client address":"#53359]","result code":"","bytes":"INFO","request method":"--","url":":","rfc931":"log","hierarchy code":"from","type":"01"}

s3

ファイルへの書き込みと同様にファイルが作成されてました。

エラー関連

エラー関連はこちらの記事にお世話になりました。
http://blog.livedoor.jp/sonots/archives/36895001.html