virtualbox(centos)でのfluentdインストールメモ(apacheのアクセスログ収集まで)

環境 centos6.3（ゲストOS）macosx10.8(ホストOS)

• ruby1.9.1以上gem1.8.1以上が入ってる前提とする
• httpd(apache)がインストール済とする

fluentdとは

ログなどをjson化できるもので日本人の古橋さんが開発したものです。
http://d.hatena.ne.jp/viver/20110929/p1

有名企業でも導入がされています。

• slideshare
• GREE 等

公式ドキュメント
http://docs.fluentd.org/articles/quickstart
日本語版（非公式??）
http://documents.mazgi.com/fluentd/doc/index.html

開発でよくある「アクセスログを解析してくれーーー」って時に
jsonでオブジェクト化されたデータの精査ができるので便利そうです。
（まだ使い始めなのでよくわからんが。。。。）

gemのupdateをする

一番はじめ公式ドキュメント通りやろうと思ったらエラーになってしばし悩んだ。。。

gem install fluentd --no-ri --no-rdoc

WARNING:  Error fetching data: SocketError: getaddrinfo: Name or service not known (http://rubygems.org/latest_specs.4.8.gz)
ERROR:  Could not find a valid gem 'fluentd' (>= 0) in any repository
ERROR:  Possible alternatives: fluentd

悩むこと１時間まずはupdateかと気づく
なんかの依存モジュールのパッケージで引っ掛かんだと思う。

gem update

Updating installed gems
WARNING:  Error fetching data: SocketError: getaddrinfo: Name or service not known (http://rubygems.org/latest_specs.4.8.gz)
Updating io-console
WARNING:  Error fetching data: SocketError: getaddrinfo: Name or service not known (http://rubygems.org/specs.4.8.gz)
Fetching: io-console-0.4.1.gem (100%)
Building native extensions.  This could take a while...
Successfully installed io-console-0.4.1
Updating minitest
Fetching: minitest-4.5.0.gem (100%)
Successfully installed minitest-4.5.0
Updating rake
Fetching: rake-10.0.3.gem (100%)
Successfully installed rake-10.0.3
Updating rdoc
Fetching: rdoc-3.12.1.gem (100%)
Depending on your version of ruby, you may need to install ruby rdoc/ri data:

<= 1.8.6 : unsupported
 = 1.8.7 : gem install rdoc-data; rdoc-data --install
 = 1.9.1 : gem install rdoc-data; rdoc-data --install
>= 1.9.2 : nothing to do! Yay!
Successfully installed rdoc-3.12.1
Gems updated: io-console, minitest, rake, rdoc
Installing ri documentation for io-console-0.4.1...
Installing ri documentation for minitest-4.5.0...
Installing ri documentation for rake-10.0.3...
Installing ri documentation for rdoc-3.12.1...
Installing RDoc documentation for io-console-0.4.1...
Installing RDoc documentation for minitest-4.5.0...
Installing RDoc documentation for rake-10.0.3...
Installing RDoc documentation for rdoc-3.12.1...

もう一度リトライ

gem install fluentd

Fetching: msgpack-0.4.7.gem (100%)
Building native extensions.  This could take a while...
Fetching: iobuffer-1.1.2.gem (100%)
Building native extensions.  This could take a while...
Fetching: cool.io-1.1.0.gem (100%)
Building native extensions.  This could take a while...
Fetching: http_parser.rb-0.5.3.gem (100%)
Building native extensions.  This could take a while...
Fetching: fluentd-0.10.31.gem (100%)
Successfully installed msgpack-0.4.7
Successfully installed iobuffer-1.1.2
Successfully installed cool.io-1.1.0
Successfully installed http_parser.rb-0.5.3
Successfully installed fluentd-0.10.31
5 gems installed
Installing ri documentation for msgpack-0.4.7...
Installing ri documentation for iobuffer-1.1.2...
Installing ri documentation for cool.io-1.1.0...
Installing ri documentation for http_parser.rb-0.5.3...
Installing ri documentation for fluentd-0.10.31...
Installing RDoc documentation for msgpack-0.4.7...
Installing RDoc documentation for iobuffer-1.1.2...
Installing RDoc documentation for cool.io-1.1.0...
Installing RDoc documentation for http_parser.rb-0.5.3...
Installing RDoc documentation for fluentd-0.10.31... #入った

※コマンドで最初fluentdが見つからなくて困ったが一旦virtualboxからexitで抜けて入りなおしたら入っていた。
多分PATHの設定がrbenvだと一旦コンソールから抜けないとダメっぽい。。
（というかinitの処理が必要だったみたい）

fluentdのセットアップ

まあ/etc/以下にあればいいだろうということで

fluentd --setup /etc/fluent

installed ./fluent/fluent.conf.

動作検証

これもマニュアルどおりにやってみる

fluentd -c /etc/fluent/fluent.conf -vv &

[1] 4075
[root@localhost fluent]# 2013-02-15 11:45:51 +0900: fluent/supervisor.rb:153:supervise: starting fluentd-0.10.31
2013-02-15 11:45:51 +0900: fluent/supervisor.rb:235:read_config: reading config file path="fluent.conf"
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered buffer plugin 'file'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered buffer plugin 'memory'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'debug_agent'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'exec'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'forward'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'gc_stat'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'http'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'object_space'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'status'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'tcp'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'unix'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'syslog'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'tail'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'copy'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'exec'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'exec_filter'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'file'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'forward'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'null'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'roundrobin'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'stdout'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'tcp'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'unix'
2013-02-15 11:45:51 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'test'
2013-02-15 11:45:51 +0900: fluent/engine.rb:61:configure: using configuration file: <ROOT>
  <source>
    type forward
  </source>
  <source>
    type http
    port 8888
  </source>
  <source>
    type debug_agent
    port 24230
  </source>
  <match debug.**>
    type stdout
  </match>
</ROOT>
2013-02-15 11:45:51 +0900: fluent/engine.rb:70:block in configure: adding source type="forward"
2013-02-15 11:45:51 +0900: fluent/engine.rb:70:block in configure: adding source type="http"
2013-02-15 11:45:51 +0900: fluent/engine.rb:70:block in configure: adding source type="debug_agent"
2013-02-15 11:45:51 +0900: fluent/engine.rb:86:block in configure: adding match pattern="debug.**" type="stdout"
2013-02-15 11:45:51 +0900: plugin/in_forward.rb:63:listen: listening fluent socket on 0.0.0.0:24224
2013-02-15 11:45:51 +0900: plugin/in_http.rb:74:start: listening http on 0.0.0.0:8888
2013-02-15 11:45:51 +0900: plugin/in_debug_agent.rb:46:start: listening dRuby uri="druby://0.0.0.0:24230" object="Engine"

実際やってみる

echo '{"json":"message"}' | fluent-cat debug.test

2013-02-15 11:46:08 +0900: plugin/in_forward.rb:147:initialize: accepted fluent socket object_id=14395860
2013-02-15 11:46:08 +0900 debug.test: {"json":"message"}
2013-02-15 11:46:08 +0900: plugin/in_forward.rb:188:on_close: closed fluent socket object_id=14395860

動いてる

fluentdの起動と停止方法

起動

fluentd -c fluent.confのパス -vv &

停止

#デフォルトは24224っぽい
/usr/sbin/lsof | grep 24224

fluentd   6428      root    6u     IPv4              44926      0t0        TCP *:24224 (LISTEN)
fluentd   6428      root    7u     IPv4              44928      0t0        UDP *:24224 

プロセスが生きてることを確認して

pkill fluentd

殺す！！

※一部のブログで下記の方法で停止をしていますが、たまにプロセスが重複起動してプロセスが殺しきれない場合があるので注意してください。

#まずはjobの確認
jobs -l
[1]+  4075 Running                 fluentd -c fluent.conf -vv &


#そしてkill
kill 4075
# 2013-02-15 12:09:17 +0900: fluent/engine.rb:167:stop: shutting down fluentd
2013-02-15 12:09:18 +0900: fluent/supervisor.rb:170:supervise: process finished code=0

[1]+  Done                    fluentd -c fluent.conf -vv

apacheのアクセスログをjsonで出力する

/etc/fluent/fluent.conf

vim /etc/fluent/fluent.conf

#修正前
#<source>
#  type tail
#  format apache
#  path /var/log/httpd/access_log
#  tag apache.access
#</source>


#コメントアウトを外してpathの部分を変更する
#修正後
<source>
  type tail
  format apache
  path /var/log/httpd/access_log
  tag apache.access
</source>

# 修正前
#<match apache.access>
#  type file
#  path /var/log/fluent/access
#</match>

#修正後
<match apache.access>
  type file
  path /var/log/fluent/access
</match>

fluentdの再起動

再起動しないと読み込まなかったので

停止

pkill fluentd

再起動

fluentd -c /etc/fluent/fluent.conf -vv &


[1] 4171
[root@localhost fluent]# 2013-02-15 12:21:53 +0900: fluent/supervisor.rb:153:supervise: starting fluentd-0.10.31
2013-02-15 12:21:53 +0900: fluent/supervisor.rb:235:read_config: reading config file path="/etc/fluent/fluent.conf"
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered buffer plugin 'file'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered buffer plugin 'memory'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'debug_agent'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'exec'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'forward'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'gc_stat'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'http'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'object_space'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'status'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'tcp'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'unix'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'syslog'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered input plugin 'tail'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'copy'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'exec'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'exec_filter'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'file'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'forward'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'null'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'roundrobin'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'stdout'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'tcp'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'unix'
2013-02-15 12:21:53 +0900: fluent/plugin.rb:89:register_impl: registered output plugin 'test'
2013-02-15 12:21:53 +0900: fluent/engine.rb:61:configure: using configuration file: <ROOT>
  <source>
    type forward
  </source>
  <source>
    type http
    port 8888
  </source>
  <source>
    type tail
    format apache
    path /var/log/httpd/access_log
    tag apache.access
  </source>
  <source>
    type debug_agent
    port 24230
  </source>
  <match apache.access>
    type file
    path /var/log/fluent/access
  </match>
  <match debug.**>
    type stdout
  </match>
</ROOT>
2013-02-15 12:21:53 +0900: fluent/engine.rb:70:block in configure: adding source type="forward"
2013-02-15 12:21:53 +0900: fluent/engine.rb:70:block in configure: adding source type="http"
2013-02-15 12:21:53 +0900: fluent/engine.rb:70:block in configure: adding source type="tail"
2013-02-15 12:21:53 +0900: plugin/in_tail.rb:49:configure: 'pos_file PATH' parameter is not set to a 'tail' source.
2013-02-15 12:21:53 +0900: plugin/in_tail.rb:50:configure: this parameter is highly recommended to save the position to resume tailing.
2013-02-15 12:21:53 +0900: fluent/engine.rb:70:block in configure: adding source type="debug_agent"
2013-02-15 12:21:53 +0900: fluent/engine.rb:86:block in configure: adding match pattern="apache.access" type="file"
2013-02-15 12:21:53 +0900: fluent/engine.rb:86:block in configure: adding match pattern="debug.**" type="stdout"
2013-02-15 12:21:53 +0900: plugin/in_forward.rb:63:listen: listening fluent socket on 0.0.0.0:24224
2013-02-15 12:21:53 +0900: plugin/in_http.rb:74:start: listening http on 0.0.0.0:8888
2013-02-15 12:21:53 +0900: plugin/in_tail.rb:289:initialize: following tail of /var/log/httpd/access_log
2013-02-15 12:21:53 +0900: plugin/in_debug_agent.rb:46:start: listening dRuby uri="druby://0.0.0.0:24230" object="Engine"

エラーとかがでていないことを確認する

出力内容の確認

場所は/var/log/fluent/httpd/access.日付.トークン

tail -f /var/log/fluent/httpd/access.20130215.b4d5bdcc431c4eeec

開発環境等にアクセスして監視してみる

例） http://localhostなどに複数回アクセスしてみる

出力されていればOK!!

後は起動スクリプトとかあったら便利だけど、心が折れたのでここまで！！