暇つぶしに、Cygwin上にfluentdを入れて動作させるまでのメモ
- apacheはvirtualbox上
- cygwin側にrubyとfluentdを入れて動作させる(virtualboxの開発環境をあまり変更したくないため)
- ruby関連まったく知りません
参考にさせていただいたサイト
Cygwin側の作業
cygwinのsetup.exeでrubyを入れる
結果
$ ruby --version
ruby 1.9.3p448 (2013-06-27) [i386-cygwin]
gemとかいうのをupdateしてfluentdをインストール
update結果
$ gem update --http-proxy http://proxy.xxx.yyy.zzz:8080
Updating installed gems
Updating bigdecimal
Fetching: bigdecimal-1.2.5.gem (100%)
Building native extensions. This could take a while...
Successfully installed bigdecimal-1.2.5
Updating io-console
Fetching: io-console-0.4.2.gem (100%)
Building native extensions. This could take a while...
Successfully installed io-console-0.4.2
Updating json
Fetching: json-1.8.1.gem (100%)
Building native extensions. This could take a while...
Successfully installed json-1.8.1
Updating minitest
Fetching: minitest-5.2.2.gem (100%)
Successfully installed minitest-5.2.2
Updating rake
Fetching: rake-10.1.1.gem (100%)
Successfully installed rake-10.1.1
Updating rdoc
Fetching: rdoc-4.1.1.gem (100%)
Depending on your version of ruby, you may need to install ruby rdoc/ri data:
<= 1.8.6 : unsupported
= 1.8.7 : gem install rdoc-data; rdoc-data --install
= 1.9.1 : gem install rdoc-data; rdoc-data --install
>= 1.9.2 : nothing to do! Yay!
Successfully installed rdoc-4.1.1
Gems updated: bigdecimal, io-console, json, minitest, rake, rdoc
Installing ri documentation for bigdecimal-1.2.5...
Installing ri documentation for io-console-0.4.2...
Installing ri documentation for json-1.8.1...
Installing ri documentation for minitest-5.2.2...
Installing ri documentation for rake-10.1.1...
Installing ri documentation for rdoc-4.1.1...
Installing RDoc documentation for bigdecimal-1.2.5...
Installing RDoc documentation for io-console-0.4.2...
Installing RDoc documentation for json-1.8.1...
Installing RDoc documentation for minitest-5.2.2...
Installing RDoc documentation for rake-10.1.1...
Installing RDoc documentation for rdoc-4.1.1...
install結果
$ gem install fluentd --no-ri --no-rdoc --http-proxy http://proxy.xxx.yyy.zzz:8080
Fetching: msgpack-0.5.8.gem (100%)
Building native extensions. This could take a while...
Fetching: yajl-ruby-1.2.0.gem (100%)
Building native extensions. This could take a while...
Fetching: iobuffer-1.1.2.gem (100%)
Building native extensions. This could take a while...
Fetching: cool.io-1.1.1.gem (100%)
Building native extensions. This could take a while...
Fetching: http_parser.rb-0.5.3.gem (100%)
Building native extensions. This could take a while...
Fetching: fluentd-0.10.42.gem (100%)
Successfully installed msgpack-0.5.8
Successfully installed yajl-ruby-1.2.0
Successfully installed iobuffer-1.1.2
Successfully installed cool.io-1.1.1
Successfully installed http_parser.rb-0.5.3
Successfully installed fluentd-0.10.42
6 gems installed
$ fluentd --version
fluentd 0.10.42
fluentdのセットアップ（任意のディレクトリ指定で）
$ fluentd --setup ./fluentd/fluent
Installed ./fluentd/fluent/fluent.conf.
fluentdの起動＆停止
起動コマンド（confファイル指定で）
fluentd -c ./fluentd/fluent/fluent.conf -vv &
起動結果
$ fluentd -c ./fluentd/fluent/fluent.conf -vv &
[1] 6880
$ 2013-12-03 17:51:18 +0900 [info]: fluent/supervisor.rb:171:supervise: starting fluentd-0.10.42
2013-12-03 17:51:18 +0900 [info]: fluent/supervisor.rb:273:read_config: reading config file path="./fluentd/fluent/fluent.conf"
2013-12-03 17:51:18 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered buffer plugin 'file'
2013-12-03 17:51:18 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered buffer plugin 'memory'
2013-12-03 17:51:18 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'debug_agent'
2013-12-03 17:51:18 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'exec'
2013-12-03 17:51:18 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'forward'
2013-12-03 17:51:18 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'gc_stat'
2013-12-03 17:51:18 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'http'
2013-12-03 17:51:18 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'monitor_agent'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'object_space'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'status'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'tcp'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'unix'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'syslog'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'tail'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'copy'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'exec'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'exec_filter'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'file'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'forward'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'null'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'roundrobin'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'stdout'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'tcp'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'unix'
2013-12-03 17:51:19 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'test'
2013-12-03 17:51:19 +0900 [info]: fluent/engine.rb:85:block in configure: gem 'fluentd' version '0.10.42'
2013-12-03 17:51:19 +0900 [info]: fluent/engine.rb:89:configure: using configuration file: <ROOT>
<source>
type forward
</source>
<source>
type http
port 8888
</source>
<source>
type monitor_agent
port 24220
</source>
<source>
type debug_agent
port 24230
</source>
<match debug.**>
type stdout
</match>
<match system.**>
type forward
host 192.168.0.11
<secondary>
host 192.168.0.12
</secondary>
</match>
</ROOT>
2013-12-03 17:51:19 +0900 [info]: fluent/engine.rb:99:block in configure: adding source type="forward"
2013-12-03 17:51:19 +0900 [info]: fluent/engine.rb:99:block in configure: adding source type="http"
2013-12-03 17:51:19 +0900 [info]: fluent/engine.rb:99:block in configure: adding source type="monitor_agent"
2013-12-03 17:51:19 +0900 [info]: fluent/engine.rb:99:block in configure: adding source type="debug_agent"
2013-12-03 17:51:19 +0900 [info]: fluent/engine.rb:115:block in configure: adding match pattern="debug.**" type="stdout"
2013-12-03 17:51:19 +0900 [info]: fluent/engine.rb:115:block in configure: adding match pattern="system.**" type="forward"
2013-12-03 17:51:19 +0900 [warn]: plugin/out_forward.rb:57:configure: 'host' option in forward output is obsoleted. Use '<server> host xxx </server>' instead.
2013-12-03 17:51:19 +0900 [info]: plugin/out_forward.rb:87:block in configure: adding forwarding server '192.168.0.12:24224' host="192.168.0.12" port=24224 weight=60
2013-12-03 17:51:19 +0900 [warn]: plugin/out_forward.rb:57:configure: 'host' option in forward output is obsoleted. Use '<server> host xxx </server>' instead.
2013-12-03 17:51:19 +0900 [info]: plugin/out_forward.rb:87:block in configure: adding forwarding server '192.168.0.11:24224' host="192.168.0.11" port=24224 weight=60
2013-12-03 17:51:19 +0900 [debug]: plugin/out_forward.rb:170:rebuild_weight_array: rebuilding weight array lost_weight=0
2013-12-03 17:51:19 +0900 [debug]: plugin/out_forward.rb:170:rebuild_weight_array: rebuilding weight array lost_weight=0
2013-12-03 17:51:19 +0900 [info]: plugin/in_forward.rb:64:listen: listening fluent socket on 0.0.0.0:24224
2013-12-03 17:51:19 +0900 [debug]: plugin/in_http.rb:73:start: listening http on 0.0.0.0:8888
2013-12-03 17:51:19 +0900 [debug]: plugin/in_monitor_agent.rb:157:start: listening monitoring http server on http://0.0.0.0:24220/api/plugins
2013-12-03 17:51:19 +0900 [info]: plugin/in_debug_agent.rb:44:start: listening dRuby uri="druby://0.0.0.0:24230" object="Engine"
2013-12-03 17:51:56 +0900 [warn]: plugin/out_forward.rb:406:tick: detached forwarding server '192.168.0.12:24224' host="192.168.0.12" port=24224 phi=16.405017259743943
2013-12-03 17:51:56 +0900 [debug]: plugin/out_forward.rb:170:rebuild_weight_array: rebuilding weight array lost_weight=60
2013-12-03 17:51:56 +0900 [warn]: plugin/out_forward.rb:406:tick: detached forwarding server '192.168.0.11:24224' host="192.168.0.11" port=24224 phi=16.408709530721485
2013-12-03 17:51:56 +0900 [debug]: plugin/out_forward.rb:170:rebuild_weight_array: rebuilding weight array lost_weight=60
Ctrl+cで終了させて再度起動してみたら、再起動ループが発生
$ 2013-12-04 12:09:30 +0900 [info]: fluent/supervisor.rb:171:supervise: starting fluentd-0.10.42
2013-12-04 12:09:30 +0900 [info]: fluent/supervisor.rb:273:read_config: reading config file path="./fluentd/fluent/fluent.conf"
2013-12-04 12:09:30 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered buffer plugin 'file'
2013-12-04 12:09:30 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered buffer plugin 'memory'
2013-12-04 12:09:30 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'debug_agent'
2013-12-04 12:09:30 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'exec'
2013-12-04 12:09:30 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'forward'
・・・省略・・・
2013-12-04 12:09:31 +0900 [info]: plugin/in_forward.rb:64:listen: listening fluent socket on 0.0.0.0:24224
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:218:rescue in run: unexpected error error_class=Errno::EADDRINUSE error=#<Errno::EADDRINUSE: Address already in use - bind(2)>
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/server.rb:57:in `initialize'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/server.rb:57:in `new'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/server.rb:57:in `initialize'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/plugin/in_forward.rb:65:in `new'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/plugin/in_forward.rb:65:in `listen'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/plugin/in_forward.rb:40:in `start'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/engine.rb:250:in `block in start'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/engine.rb:249:in `each'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/engine.rb:249:in `start'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/engine.rb:204:in `run'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/supervisor.rb:386:in `run_engine'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/supervisor.rb:94:in `block in start'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/supervisor.rb:198:in `call'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/supervisor.rb:198:in `main_process'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/supervisor.rb:173:in `block in supervise'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/supervisor.rb:172:in `fork'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/supervisor.rb:172:in `supervise'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/supervisor.rb:87:in `start'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/lib/fluent/command/fluentd.rb:146:in `<top (required)>'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:55:in `require'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:55:in `require'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/lib/ruby/gems/1.9.1/gems/fluentd-0.10.42/bin/fluentd:6:in `<top (required)>'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/bin/fluentd:23:in `load'
2013-12-04 12:09:31 +0900 [error]: fluent/engine.rb:219:rescue in run: /usr/bin/fluentd:23:in `<main>'
2013-12-04 12:09:31 +0900 [info]: fluent/engine.rb:221:run: shutting down fluentd
2013-12-04 12:09:33 +0900 [info]: fluent/supervisor.rb:188:supervise: process finished code=0
2013-12-04 12:09:33 +0900 [error]: fluent/supervisor.rb:97:start: fluentd main process died unexpectedly. restarting.
以降最初に戻って以下ループ・・・
とりあえずrubyのプロセスを片っ端からkillしたら止まった。
$kill -9 xxxx xxxx xxxx xxxx
$ 2013-12-04 12:27:36 +0900 [debug]: fluent/supervisor.rb:354:block in install_main_process_signal_handlers: fluentd main process get SIGTERM
2013-12-04 12:27:36 +0900 [debug]: fluent/supervisor.rb:357:block in install_main_process_signal_handlers: getting start to shutdown main process
2013-12-04 12:27:36 +0900 [info]: fluent/engine.rb:221:run: shutting down fluentd
起動するとrubyのプロセスが2つ起ち上がるらしいので、
これを両方ともkillすれば再起動ループは発生しない(この止め方が正しいかは不明)
$ ps -a
PID PPID PGID WINPID TTY UID STIME COMMAND
7148 6768 7148 1608 pty1 21751 14:05:57 /usr/bin/ruby
2256 7148 7148 2256 pty1 21751 14:05:59 /usr/bin/ruby
VirtualBox側の作業
apacheの設定
apacheのaccessログをcygwinから参照できるところにも吐くように
CustomLog設定を追加してapache再起動
httpd.conf
CustomLog /media/sf_share/logs/access_log combined
以上。
apacheのアクセスログファイル→fluentで別ファイルに出力を試してみる
設定ファイルを調整(※コメントアウトはずしてパスを変えただけ)
fluent.conf
## File input
## read apache logs with tag=apache.access
<source>
type tail
format apache
path /cygdrive/c/share/logs/access_log
tag apache.access
</source>
## match tag=apache.access and write to file
<match apache.access>
type file
path /cygdrive/c/share/logs/fluent/access
</match>
例えばこれが、
10.0.2.2 - abc@aaa.bbb.ccc.co.jp [04/Dec/2013:11:41:12 +0900] "GET /testapp/controller_b/action_1 HTTP/1.1" 500 3280 "http://127.0.0.1/testapp/index.php/controller_a/action_1/param" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
こう出力される。
2013-12-04T11:41:12+09:00 apache.access {"host":"10.0.2.2","user":"abc@aaa.bbb.ccc.co.jp","method":"GET","path":"/testapp/controller_b/action_1","code":"500","size":"3280","referer":"http://127.0.0.1/testapp/index.php/controller_a/action_1/param","agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"}
で、何に使うんだろうね。