LoginSignup
3
5

More than 3 years have passed since last update.

@migo

yumでLogstashをインストールして、困ったこと

Posted at

概要

Logstash をyum installしたときに、つまづいたことのメモ。

（自分が参照した）ネットや書籍では、Logstashインストール後の動作確認で、sudo logstash -e ...などとrootで起動させるような記述がなされているが、そうすると、その後のsystemctl start logstashでエラーになるよ、という話。

検証環境

  • CentOS 7.5 (VirtualBox)
  • Logstash 6.8.1
  • OpenJDK 1.8.0_161

yum install したときの環境

sudo yum install logstash で、つくられる環境は、下記のとおり。

OSユーザー、グループ

# id logstash
uid=986(logstash) gid=980(logstash) groups=980(logstash)
# grep logstash /etc/passwd
logstash:x:986:980:logstash:/usr/share/logstash:/sbin/nologin
# grep logstash /etc/group
logstash:x:980:

/etc/systemd/system/logstash.service

このように、logstash ユーザーで起動するように書かれている。

[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

ディレクトリ・ファイル

Directory/File name Permission User Group Memo
/usr/share/logstash/ drwxr-xr-x. logstash logstash LS_HOME
/usr/share/logstash/bin/logstash -rwxr-xr-x. logstash logstash アプリケーション本体
/etc/logstash/ drwxr-xr-x. root root path.settings
/etc/logstash/logstash.yml -rw-r--r--. root root
/etc/logstash/pipelines.yml -rw-r--r--. root root
/etc/logstash/logstash-sample.conf -rw-r--r--. root root 設定ファイルのサンプル
/etc/logstash/conf.d/ drwxrwxr-x. root root 設定ファイルをおくディレクトリ
/var/log/logstash/ drwxrwxr-x. logstash root path.logs
/var/lib/logstash/ drwxrwxr-x. logstash logstash path.data

どんな意図があるのか、設定まわりが root でつくられている。

startup.options

参考として、/etc/logstash/startup.options の抜粋。

LS_HOME=/usr/share/logstash
LS_SETTINGS_DIR=/etc/logstash
LS_OPTS="--path.settings ${LS_SETTINGS_DIR}"
LS_USER=logstash
LS_GROUP=logstash
SERVICE_NAME="logstash"
SERVICE_DESCRIPTION="logstash"

Error/Warningになるパターン

という環境で、自分が遭遇したケースは下記のとおり。

動作確認を root でおこなった場合

インストール直後の動作確認を root でおこなうと、/usr/share/logstash/data/ 内に root でディレクトリなどが作成される。

$ sudo /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2019-07-12 11:55:05.967 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2019-07-12 11:55:05.997 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2019-07-12 11:55:06.500 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-07-12 11:55:06.515 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.1"}
[INFO ] 2019-07-12 11:55:06.591 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"be79488d-c123-4b09-a080-72ce424307a2", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2019-07-12 11:55:15.668 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
The stdin plugin is now waiting for input:
[INFO ] 2019-07-12 11:55:15.968 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x72418697 sleep>"}
[INFO ] 2019-07-12 11:55:16.029 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-07-12 11:55:16.320 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
Hello Logstash!
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
      "@version" => "1",
       "message" => "Hello Logstash!",
          "host" => "node160.migo.jp",
    "@timestamp" => 2019-07-12T02:55:37.212Z
}
[INFO ] 2019-07-12 11:55:40.254 [[main]-pipeline-manager] pipeline - Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x72418697 run>"}
[INFO ] 2019-07-12 11:55:41.014 [LogStash::Runner] runner - Logstash shut down.

$ ls -l /usr/share/logstash/data
合計 4
drwxrwxr-x.  4 logstash logstash  69  7月 12 13:40 .
drwxr-xr-x. 11 logstash logstash 241  7月 12 11:19 ..
-rw-r--r--.  1 root     root       0  7月 12 11:55 .lock
drwxr-xr-x.  2 root     root       6  7月 12 11:55 dead_letter_queue
drwxr-xr-x.  2 root     root       6  7月 12 11:55 queue
-rw-r--r--.  1 root     root      36  7月 12 11:55 uuid

そのため、その後 logstash ユーザーで実行するとエラーになる。

$ sudo -u logstash  /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[FATAL] 2019-07-12 13:23:43.303 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:447:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:229:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:140:in `block in validate_all'", "org/jruby/RubyHash.java:1419:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:139:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:278:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:237:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[ERROR] 2019-07-12 13:23:43.324 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

こうなったら、/usr/share/logstash/data の中身をすべて（.lock も忘れずに）削除して、logstash ユーザーで起動しなおせばよい。すると、...

$ sudo -u logstash /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
...（省略）...
$ ls -la /usr/share/logstash/data
合計 4
drwxrwxr-x.  4 logstash logstash  69  7月 12 13:49 .
drwxr-xr-x. 11 logstash logstash 241  7月 12 11:19 ..
-rw-r--r--.  1 logstash logstash   0  7月 12 13:49 .lock
drwxr-xr-x.  2 logstash logstash   6  7月 12 13:49 dead_letter_queue
drwxr-xr-x.  2 logstash logstash   6  7月 12 13:49 queue
-rw-r--r--.  1 logstash logstash  36  7月 12 13:49 uuid

こうなります。

動作確認で --path.settings をつけなかった場合

起動オプションの --path.settings を指定しないと、こんな警告が表示されます。

$ sudo -u logstash  /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
...（省略）...

動作確認レベルでは別に構わないのですが、オプションをつけると警告は表示されなくなります。

$ sudo -u logstash /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }' --path.settings "/etc/logstash"

ただ、/etc/logstash/ 内の設定が使われるので、/var/log/logstash/ にログが出力されたり、queue/ などが /usr/share/logstash/data/ ではなく /var/lib/logstash/ に作成されたりします。

まとめ

  • インストール後の動作確認は、logstash ユーザーでおこない、--path.setting オプションも指定してあげる。
$ sudo -u logstash /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }' --path.settings "/etc/logstash"

付録

設定ファイルをつくって、試す場合。
これでエラーが無ければ、systemctl start logstashしても大丈夫。

$ pwd
/etc/logstash
$ ls -l
合計 36
drwxrwxr-x. 2 root root    6  6月 18 23:13 conf.d
-rw-r--r--. 1 root root 1829  6月 18 23:13 jvm.options
-rw-r--r--. 1 root root 4568  6月 18 23:13 log4j2.properties
-rw-r--r--. 1 root root  342  6月 18 23:13 logstash-sample.conf
-rw-r--r--. 1 root root 8236  7月 12 11:19 logstash.yml
-rw-r--r--. 1 root root  285  6月 18 23:13 pipelines.yml
-rw-------. 1 root root 1696  6月 18 23:13 startup.options
$ sudo cp logstash-sample.conf ./conf.d/logstash.conf
$ cd conf.d/
$ ls -l
合計 4
-rw-r--r--. 1 root root 342  7月 12 14:19 logstash.conf
$ sudo vi logstash.conf 
$ cat logstash.conf 
input {
  stdin { }
}
output {
  stdout { }
}
$ sudo -u logstash  /usr/share/logstash/bin/logstash --path.settings "/etc/logstash"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-07-12T14:25:39,164][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.8.1"}
[2019-07-12T14:25:47,865][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
The stdin plugin is now waiting for input:
[2019-07-12T14:25:48,189][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x73860c9c sleep>"}
[2019-07-12T14:25:48,246][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-07-12T14:25:48,521][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
Hello Logstash!
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
       "message" => "Hello Logstash!",
      "@version" => "1",
          "host" => "node160.migo.jp",
    "@timestamp" => 2019-07-12T05:26:14.655Z
}
[2019-07-12T14:26:17,853][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x73860c9c run>"}
[2019-07-12T14:26:18,358][INFO ][logstash.runner          ] Logstash shut down.
3
5
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
3
5