Go to Qiita Advent Calendar 2024 Top
LoginSignup
23
24

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

@rubytomato@github(渡邉 真)

JavaアプリケーションのログをFluentd(Windows)へ送信する

Last updated at Posted at 2015-10-20

概要

開発、検証目的としてWindows PCにインストールしたFluentdに、Javaアプリケーションが出力するログを送信する方法のまとめです。

環境

  • Windows7 (64bit)
  • Fluentd 0.12.16
  • Ruby 2.2.3
  • Java 1.8.0_60
  • Git 2.6

参考

Javaアプリケーションのログを送信する

1. [Fluent Logger for Java] (https://github.com/fluent/fluent-logger-java)を使用する方法

Fluent Loggerを使用してログを直接fluentdへ送信する方法です。

Maven

pom.xml
<dependency>
  <groupId>org.fluentd</groupId>
  <artifactId>fluent-logger</artifactId>
  <version>0.3.2</version>
</dependency>

FluentLogger

  • tagPrefixに任意のタグを指定します。
  • hostのデフォルトはlocalhostです。
  • portのデフォルトは24224です。
例
final static FluentLogger LOG = FluentLogger.getLogger("app");

hostとportに任意の値を指定することもできます。

例
final static FluentLogger LOG = FluentLogger.getLogger("app", "localhost", 24224);

LOG

key/valueの形式でログを送信します。

例
LOG.log("tag1", "key1", "value1");
log
2015-10-20 19:41:19 +0900 app.tag1: {"key1":"value1"}

timeに任意の時刻を指定することができます。

例
long epoch = System.currentTimeMillis()/1000L;
LOG.log("tag1", "key1", "value1", epoch + 10L);

Mapの値を送信します。

例
Map<String, Object> data = new HashMap<>();
data.put("key2-A", "value2-A");
data.put("key2-B", "value2-B");
data.put("key2-C", "value2-C");
LOG.log("tag2", data);

fluentdの設定

conf
<source>
  @type forward
</source>

<match app.*>
  @type stdout
</match>

上記のログを入力としたfluentdの出力結果

fluentd_output
2015-10-20 19:41:19 +0900 app.tag2: {"key2-C":"value2-C","key2-B":"value2-B","key2-A":"value2-A"}

close

アプリケーション終了時にcloseAllメソッドを呼ぶ必要があります。

例
FluentLogger.closeAll();

2. [logback] (http://logback.qos.ch/)のlogback-jsonを利用する方法

正確にはJavaアプリケーションから送信ではなくfluentdからログを読み取る方法になります。
logbackでjson形式のログを出力し、fluentdのin_tailプラグインでそのログを読み取ります。

Maven

pom.xml
<dependency>
  <groupId>ch.qos.logback</groupId>
  <artifactId>logback-classic</artifactId>
</dependency>
<dependency>
  <groupId>ch.qos.logback.contrib</groupId>
  <artifactId>logback-json-classic</artifactId>
  <version>0.1.2</version>
</dependency>
<dependency>
  <groupId>ch.qos.logback.contrib</groupId>
  <artifactId>logback-jackson</artifactId>
  <version>0.1.2</version>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-core</artifactId>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-databind</artifactId>
</dependency>

logback.xml

logback.xml
<appender name="FILE_FLUENTD" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <file>d:/logs/fluentd.log</file>
  <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
    <fileNamePattern>d:/logs/fluentd.log.%d{yyyy-MM-dd}.zip</fileNamePattern>
    <maxHistory>7</maxHistory>
  </rollingPolicy>
  <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
    <layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
      <jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
        <prettyPrint>false</prettyPrint>
      </jsonFormatter>
      <timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSXXX</timestampFormat>
      <timestampFormatTimezoneId>Asia/Tokyo</timestampFormatTimezoneId>
      <includeContextName>false</includeContextName>
      <appendLineSeparator>true</appendLineSeparator>
    </layout>
    <charset>UTF-8</charset>
  </encoder>
</appender>

<logger name="com.example" level="DEBUG">
  <appender-ref ref="FILE_FLUENTD" />
</logger>

ログ出力は通常の方法です。

Logger
final static Logger logger = LoggerFactory.getLogger(FluentController.class);
例
logger.debug("FluentController:[index] Passing through...");

logbackが出力するログ

logback_log
{"timestamp":"2015-10-20T23:18:07.573+09:00","level":"DEBUG","thread":"http-nio-9000-exec-8","logger":"com.example.sbtip.web.FluentController","message":"FluentController:[index] Passing through..."}

fluentdの設定

conf
<source>
  @type tail
  format json
  tag test
  path d:/logs/fluentd.log
  pos_file d:/logs/fluentd.log.pos
  rotate_wait 5
  read_from_head true
  refresh_interval 60
</source>

<match test>
  @type stdout
</match>

上記のログを入力としたfluentdの出力結果

fluentd_output
2015-10-20 23:18:07 +0900 test: {"timestamp":"2015-10-20T23:18:07.573+09:00","level":"DEBUG","thread":"http-nio-9000-exec-8","logger":"com.example.sbtip.web.FluentController","message":"FluentController:[index] Passing through..."}

Fluentdのインストール

Fluentdのインストールと実行にはRubyとGitが必要なため、まだ環境にインストールされていない場合は事前にインストールを行います。

Ruby

ダウンロードページよりインストーラーをダウンロードしインストールします。

ダウンロードファイル: rubyinstaller-2.2.3-x64.exe

version

version
> ruby --version
ruby 2.2.3p173 (2015-08-18 revision 51636) [x64-mingw32]

DevKit

ダウンロードページよりインストーラーをダウンロードしインストールします。
インストール先はc:\devkit\としました。

ダウンロードファイル: DevKit-mingw64-64-4.7.2-20130224-1432-sfx.exe

DevKitのインストールディレクトリへ移動します。

> cd c:\devkit

DevKitの初期化を行います。

init
> ruby dk.rb init
[INFO] found RubyInstaller v2.2.3 at C:/Ruby22-x64

Initialization complete! Please review and modify the auto-generated
'config.yml' file to ensure it contains the root directories to all
of the installed Rubies you want enhanced by the DevKit.

DevKitのインストールを行います。

install
> ruby dk.rb install
[INFO] Updating convenience notice gem override for 'C:/Ruby22-x64'
[INFO] Installing 'C:/Ruby22-x64/lib/ruby/site_ruby/devkit.rb'

Git for Windows

ダウンロードページよりインストーラーをダウンロードしてインストールします。

ダウンロードファイル: Git-2.6.2-64-bit.exe

Fluentd

Git for Windowsを立ち上げリポジトリをcloneします。

clone
$ git clone https://github.com/fluent/fluentd.git
Cloning into 'fluentd'...
remote: Counting objects: 12896, done.
remote: Total 12896 (delta 0), reused 0 (delta 0), pack-reused 12896
Receiving objects: 100% (12896/12896), 12.95 MiB | 424.00 KiB/s, done.
Resolving deltas: 100% (7282/7282), done.
Checking connectivity... done.

cloneしたローカルリポジトリへ移動します。

> cd fluentd

bundlerをインストールします。

bundler
> gem install bundler
Fetching: bundler-1.10.6.gem (100%)
Successfully installed bundler-1.10.6
Parsing documentation for bundler-1.10.6
Installing ri documentation for bundler-1.10.6
Done installing documentation for bundler after 7 seconds
1 gem installed

gemをアップデートします。

update
> gem update --system
Updating rubygems-update
Fetching: rubygems-update-2.4.8.gem (100%)
Successfully installed rubygems-update-2.4.8
Parsing documentation for rubygems-update-2.4.8
Installing ri documentation for rubygems-update-2.4.8
Installing darkfish documentation for rubygems-update-2.4.8
Done installing documentation for rubygems-update after 1 seconds
Parsing documentation for rubygems-update-2.4.8
Done installing documentation for rubygems-update after 0 seconds
Installing RubyGems 2.4.8
RubyGems 2.4.8 installed
Parsing documentation for rubygems-2.4.8
Installing ri documentation for rubygems-2.4.8

...省略...

RubyGems system software updated

依存パッケージをインストールします。

bundle
> bundle
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/..
Resolving dependencies...
Using rake 10.4.2
Installing cool.io 1.4.1
Installing ffi 1.9.10
Installing flexmock 1.3.3
Installing http_parser.rb 0.6.0 with native extensions
Installing json 1.8.3 with native extensions
Installing msgpack 0.5.12 with native extensions
Installing sigdump 0.2.3
Installing string-scrub 0.0.5 with native extensions
Installing thread_safe 0.3.5
Installing tzinfo 1.2.2
Installing tzinfo-data 1.2015.7
Installing win32-ipc 0.6.6
Installing win32-event 0.6.1
Installing win32-service 0.8.7
Installing win32-api 1.5.3
Installing windows-api 0.4.4
Installing windows-pr 1.2.4
Installing yajl-ruby 1.2.1 with native extensions
Using fluentd 0.12.16 from source at .
Using multi_json 1.11.2
Installing parallel 1.6.1
Installing parallel_tests 1.9.0
Installing power_assert 0.2.4
Installing rr 1.1.2
Installing simplecov-html 0.5.3
Installing simplecov 0.6.4
Installing test-unit 3.1.5
Installing test-unit-rr 1.0.3
Installing timecop 0.8.0
Installing bundler 1.10.6
Bundle complete! 9 Gemfile dependencies, 31 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.

fluentdをビルドします。

build
> bundle exec rake build
fluentd 0.12.16 built to pkg/fluentd-0.12.16.gem.

fluentdをインストールします。

install
> gem install pkg\fluentd-0.12.16.gem –no-ri –no-rdoc
Successfully installed fluentd-0.12.16
Parsing documentation for fluentd-0.12.16
Installing ri documentation for fluentd-0.12.16
Done installing documentation for fluentd after 6 seconds
1 gem installed

バージョンの確認

version
> ruby bin\fluentd --version
fluentd 0.12.16

動作確認

exampleで用意されているconfを使用して起動します。

> ruby bin\fluentd -c .\example\in_forward.conf -o fluent.log

ログファイルを参照して起動したことを確認します。

fluent.log
2015-10-20 16:17:54 +0900 [info]: reading config file path=".\\example\\in_forward.conf"
2015-10-20 16:17:54 +0900 [info]: starting fluentd-0.12.16
2015-10-20 16:17:54 +0900 [info]: spawn command to main (windows) : C:/Ruby22-x64/bin/ruby.exe 'C:/Ruby22-x64/bin/fluentd' -c .\example\in_forward.conf -o fluent.log --no-supervisor
2015-10-20 16:17:55 +0900 [info]: reading config file path=".\\example\\in_forward.conf"
2015-10-20 16:17:55 +0900 [info]: starting fluentd-0.12.16 without supervision
2015-10-20 16:17:55 +0900 [info]: gem 'fluentd' version '0.12.16'
2015-10-20 16:17:55 +0900 [info]: adding match pattern="test" type="stdout"
2015-10-20 16:17:55 +0900 [info]: adding source type="forward"
2015-10-20 16:17:55 +0900 [info]: using configuration file: <ROOT>
  <source>
    type forward
  </source>
  <match test>
    type stdout
  </match>
</ROOT>
2015-10-20 16:17:55 +0900 [info]: listening fluent socket on 0.0.0.0:24224

fluent-catを使用してfluentdへログを送信します。

> echo {"id":100, "message": "fluentd - installing on Windows", "ts":"2015-10-20 00:00:00.000"} | ruby bin\fluent-cat test

先ほどのログファイルに出力されていることを確認します。

2015-10-20 16:22:49 +0900 test: {"id":100,"message":"fluentd - installing on Windows","ts":"2015-10-20 00:00:00.000"}

Fluentd v0.14.0.pre1のインストール (2015/11/04追記)

v0.14.0.pre.1をチェックアウトします。

$ git checkout -b v0.14.0.pre.1 refs/tags/v0.14.0.pre.1
Switched to a new branch 'v0.14.0.pre.1'

依存パッケージをインストールします。

bundle
> bundle
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/..
Resolving dependencies...
Using rake 10.4.2
Using cool.io 1.4.1
Using ffi 1.9.10
Using flexmock 1.3.3
Using http_parser.rb 0.6.0
Using json 1.8.3
Installing msgpack 0.7.0
Using sigdump 0.2.3
Installing strptime 0.1.4
Using thread_safe 0.3.5
Using tzinfo 1.2.2
Using tzinfo-data 1.2015.7
Using win32-ipc 0.6.6
Installing win32-event 0.6.3
Using win32-service 0.8.7
Using win32-api 1.5.3
Using windows-api 0.4.4
Using windows-pr 1.2.4
Using yajl-ruby 1.2.1
Using fluentd 0.14.0.pre.1 from source at .
Using multi_json 1.11.2
Using parallel 1.6.1
Using parallel_tests 1.9.0
Installing power_assert 0.2.5
Using rr 1.1.2
Using simplecov-html 0.5.3
Using simplecov 0.6.4
Using test-unit 3.1.5
Using test-unit-rr 1.0.3
Using timecop 0.8.0
Using bundler 1.10.6
Bundle complete! 9 Gemfile dependencies, 31 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.

fluentdをビルドします。

> bundle exec rake build
fluentd 0.14.0.pre.1 built to pkg/fluentd-0.14.0.pre.1.gem.

fluentdをインストールします。

install
> gem install pkg\fluentd-0.14.0.pre.1.gem -no-ri -no-rdoc
Successfully installed fluentd-0.14.0.pre.1
Parsing documentation for fluentd-0.14.0.pre.1
Installing ri documentation for fluentd-0.14.0.pre.1
Done installing documentation for fluentd after 6 seconds
1 gem installed

バージョンの確認

version
> ruby bin\fluentd --version
fluentd 0.14.0.pre.1

メモ

fluentdの設定について

sourceディレクティブ

forward Input Plugin

The in_forward Input plugin listens to a TCP socket to receive the event stream.

<source>
  type forward
</source>
  • type : forward, (required)
  • port : The port to listen to. Default Value = 24224
  • bind : The bind address to listen to. Default Value = 0.0.0.0 (all addresses)

[tail Input Plugin] (http://docs.fluentd.org/articles/in_tail)

The in_tail Input plugin allows Fluentd to read events from the tail of text files.

<source>
  type tail
  tag test
  path d:/logs/fluentd.log
  format json
  pos_file d:/logs/fluentd.log.pos
</source>
  • type : tail, (required)
  • tag : The tag of the event. (required)
  • path : (required)
  • format : (required)
  • pos_file : (highly recommended)

format

  • regexp
  • apache2
  • apache_error
  • nginx
  • syslog
  • tsv
  • csv
  • ltsv (Labeled Tab-Separated Value)
  • json
  • none
  • multiline

[http Input Plugin] (http://docs.fluentd.org/articles/in_http)

The in_http Input plugin enables Fluentd to retrieve records from HTTP POST.

<source>
  type http
</source>
  • type : http, (required)
  • port : The port to listen to. Default Value = 9880
  • bind : The bind address to listen to. Default Value = 0.0.0.0(all addresses)
  • format : The format of the HTTP body. The default is default.

format

  • default
  • regexp
  • json
  • ltsv
  • tsv
  • csv
  • none

例

> curl -X POST -d "json={\"action\":\"login\",\"user\":2}" "http://localhost:9880/test"

matchディレクティブ

[forward Output Plugin] (http://docs.fluentd.org/articles/out_forward)

The out_forward Buffered Output plugin forwards events to other fluentd nodes.

<match pattern>
  type forward

  <server>
    name other-server
    host 192.168.10.2
    port 24224
  </server>

  <secondary>
    type file
    path d:/logs/forward-failed.log
  </secondary>
</match>
  • type : forward, (required)
  • <server> : at least one is required.
  • name : The name of the server. This parameter is used in error messages.
  • host : The IP address or host name of the server. (required)
  • port : The port number of the host. The default is 24224.
  • weight : The load balancing weight. The default weight is 60.
  • <secondary> : The backup destination that is used when all servers are unavailable. (optional)

[stdout Output Plugin] (http://docs.fluentd.org/articles/out_stdout)

The stdout output plugin prints events to stdout (or logs if launched with daemon mode).

<match pattern>
  type stdout
</match>
  • type : stdout, (required)

[file Output Plugin] (http://docs.fluentd.org/articles/out_file)

The out_file TimeSliced Output plugin writes events to files.

<match pattern>
  type file
  path d:/logs/fluent-output.log
</match>
  • type : file, (required)
  • path : (required)
  • format : The format of the file content. The default is out_file.

format

  • out_file
  • json
  • hash
  • ltsv
  • single_value
  • csv
  • stdout

fluentdのUsage

Usage
> ruby bin\fluentd -h
Usage: fluentd [options]
    -s, --setup [DIR=/etc/fluent]    install sample configuration file to the directory
    -c, --config PATH                config file path (default: /etc/fluent/fluent.conf)
        --dry-run                    Check fluentd setup is correct or not
        --show-plugin-config=PLUGIN  Show PLUGIN configuration and exit(ex: input:dummy)
    -p, --plugin DIR                 add plugin directory
    -I PATH                          add library path
    -r NAME                          load library
    -d, --daemon PIDFILE             daemonize fluent process
        --no-supervisor              run without fluent supervisor
        --user USER                  change user
        --group GROUP                change group
    -o, --log PATH                   log file path
    -i CONFIG_STRING,                inline config which is appended to the config file on-fly
        --inline-config
        --emit-error-log-interval SECONDS
                                     suppress interval seconds of emit error logs
        --suppress-repeated-stacktrace [VALUE]
                                     suppress repeated stacktrace
        --without-source             invoke a fluentd without input plugins
        --use-v1-config              Use v1 configuration format (default)
        --use-v0-config              Use v0 configuration format
    -v, --verbose                    increase verbose level (-v: debug, -vv: trace)
    -q, --quiet                      decrease verbose level (-q: warn, -qq: error)
        --suppress-config-dump       suppress config dumping when fluentd starts
    -g, --gemfile GEMFILE            Gemfile path
    -G, --gem-path GEM_INSTALL_PATH  Gemfile install path (default: $(dirname $gemfile)/vendor/bundle)
    -x, --signame INTSIGNAME         an object name which is used for Windows Service signal (Windows only)
        --reg-winsvc MODE            install/uninstall as Windows Service. (i: install, u: uninstall) (Windows only)
        --reg-winsvc-fluentdopt OPTION
                                     specify fluentd option paramters for Windows Service. (Windows only)
23
24
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
23
24

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?