@RyuseiSato (SAI)posted at 2021-11-15

Amazon Linux2にてtd-agentが起動できない

Q&A

Closed

AWS Fluentd

解決したいこと

Amazon Linux2にてtd-agentが起動できません

こちらの記事を参考に
Twitter APIを取得し、

consumer_key
consumer_secret
oauth_token
oauth_token_secret

を入手しました。
次にtd-agent4をインストールし、以下の各種プラグインをインストールしました。

$ sudo yum -y install openssl-devel libcurl libcurl-devel gcc-c++
$ sudo td-agent-gem install eventmachine
$ sudo td-agent-gem install fluent-plugin-twitter
$ sudo yum install -y  mysql-devel
$ sudo td-agent-gem install fluent-plugin-mysql

そして以下のコードを入力し設定ファイルにコードを追加しました。

該当するソースコードに追加したコードが書いてあります。

$ sudo vim /etc/td-agent/td-agent.conf

ここでtd-agentを起動したのですが、エラーが起きてしまいました。
追加したコードの異常な点を教えていただきたいです。

発生している問題・エラー

Redirecting to /bin/systemctl start td-agent.service
Job for td-agent.service failed because the control process exited with error code. See "systemctl status td-agent.service" and "journalctl -xe" for details.

該当するソースコード

<source>
  @type twitter
  consumer_key 〇〇〇〇
  consumer_secret 〇〇〇〇
  oauth_token 〇〇〇〇
  oauth_token_secret 〇〇〇〇
  tag input.twitter
  timeline sampling
  lang ja
  output_format flat
</source>

<match input.twitter>
  @type mysql
  host 〇〇〇〇
  database textdata
  key_names id, user_id, text, created_at
  sql INSERT INTO tweet (id, user_id, text, created_at) VALUES (?, ?, ?, STR_TO_DATE(?, "%a %b %d %H:%i:%s +0000 %Y"))
  username 〇〇〇〇
  password 〇〇〇〇
  <buffer>
   flush_interval 10s
  </buffer>
</match>

4Answer

@uasi posted at 2021-11-15

まずエラーメッセージで指示されているように systemctl status td-agent.service と journalctl -xe を実行してログを確認してください。

また https://github.com/tagomoris/fluent-plugin-mysql を見る限り ~~query_columns~~ column_names パラメータが必要なようです。

1Like

@RyuseiSato posted at 2021-11-15

@uasi 回答ありがとうございます。

systemctl status td-agent.service
を実行したところ以下のコードが出されました。

● td-agent.service - td-agent: Fluentd based data collector for Treasure Data
Loaded: loaded (/usr/lib/systemd/system/td-agent.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since 月 2021-11-15 13:10:07 UTC; 22s ago
Docs: https://docs.treasuredata.com/display/public/PD/About+Treasure+Data%27s+Server-Side+Agent
Process: 827 ExecStop=/bin/kill -TERM ${MAINPID} (code=exited, status=0/SUCCESS)
Process: 23643 ExecReload=/bin/kill -HUP ${MAINPID} (code=exited, status=0/SUCCESS)
Process: 1263 ExecStart=/opt/td-agent/bin/fluentd --log $TD_AGENT_LOG_FILE --daemon /var/run/td-agent/td-agent.pid $TD_AGENT_OPTIONS (code=exited, status=1/FAILURE)
Main PID: 781 (code=exited, status=0/SUCCESS)

11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: td-agent.service: control process exited, code=exited status=1
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: Failed to start td-agent: Fluentd based data collector for Treasure Data.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: Unit td-agent.service entered failed state.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: td-agent.service failed.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: td-agent.service holdoff time over, scheduling restart.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: Stopped td-agent: Fluentd based data collector for Treasure Data.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: start request repeated too quickly for td-agent.service
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: Failed to start td-agent: Fluentd based data collector for Treasure Data.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: Unit td-agent.service entered failed state.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: td-agent.service failed.

次に
journalctl -xeを実行したところ

11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:44:in `lookup' 11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:160:in` new_impl'
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:105:in `new_input' 11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:312:in` add_source'
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:158:in `block in configure' 11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:152:in` each'
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:152:in `configure' 11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in` configure'
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in `run_configure' 11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:663:in` run_supervisor'
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:344:in `<top (required)>' 11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in` require'
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' 11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/bin/fluentd:15:in` '
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/bin/fluentd:23:in `load' 11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal fluentd[1263]: from /opt/td-agent/bin/fluentd:23:in` '
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: td-agent.service: control process exited, code=exited status=1
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: Failed to start td-agent: Fluentd based data collector for Treasure Data.
-- Subject: Unit td-agent.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit td-agent.service has failed.

-- The result is failed.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: Unit td-agent.service entered failed state.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: td-agent.service failed.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: td-agent.service holdoff time over, scheduling restart.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: Stopped td-agent: Fluentd based data collector for Treasure Data.
-- Subject: Unit td-agent.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit td-agent.service has finished shutting down.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: start request repeated too quickly for td-agent.service
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: Failed to start td-agent: Fluentd based data collector for Treasure Data.
-- Subject: Unit td-agent.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit td-agent.service has failed.

-- The result is failed.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: Unit td-agent.service entered failed state.
11月 15 13:10:07 ip-172-31-22-104.ap-northeast-1.compute.internal systemd[1]: td-agent.service failed.
11月 15 13:11:31 ip-172-31-22-104.ap-northeast-1.compute.internal dhclient[2748]: XMT: Solicit on eth0, interval 108850ms.
11月 15 13:11:58 ip-172-31-22-104.ap-northeast-1.compute.internal sshd[1269]: Received disconnect from 101.32.181.61 port 55866:11: Bye Bye [preauth]
11月 15 13:11:58 ip-172-31-22-104.ap-northeast-1.compute.internal sshd[1269]: Disconnected from 101.32.181.61 port 55866 [preauth]

と出されました。

ご指摘いただいたquery_columnsはどこのことでしょうか？
　column_names
table
この二つでしょうか？

0Like

@uasi posted at 2021-11-15

ご指摘いただいたquery_columnsはどこのことでしょうか？

すみません、 column_names の間違いでした。 table も必要ですね。逆に key_names や sql には対応していません。

fluent-plugin-mysql v0.0.9 以前には key_names や sql が使えたようです https://github.com/tagomoris/fluent-plugin-mysql/blob/v0.0.9/README_mysql.md

参考にされたページの記述が古いようですね。最新の README を参考に書き直してください。

0Like

@RyuseiSato posted at 2021-11-24

@uei ありがとうございます。もう一度試してみます。

0Like

Are you sure you want to delete the question?

Amazon Linux2にてtd-agentが起動できない

解決したいこと

発生している問題・エラー

該当するソースコード

4Answer

-- Unit td-agent.service has failed.

-- Unit td-agent.service has failed.

Your answer might help someone💌