Qiita Teams that are logged in
You are not logged in to any team

Log in to Qiita Team
Community
OrganizationAdvent CalendarQiitadon (β)
Service
Qiita JobsQiita ZineQiita Blog
1
Help us understand the problem. What is going on with this article?
@speedkingg

kubernetesのログをfluentdでCloudwatchに送ろうとしたら、UndefinedConversionErrorで死んだ

More than 1 year has passed since last update.

٩(๑`^´๑)۶

エラーの原因究明に3時間くらいかかってつらかったので、書きました。

目的

kubernetespodのログをいちいち見に行くのが面倒なので、fluentdCloudwatchで集約したい。

やったこと

こちらの方の手順を参考にymlを作成し、applyした。
- 使用したmanifestfluentd-kubernetes-daemonset/fluentd-daemonset-cloudwatch-rbac.yaml

起きたエラー:UndefinedConversionError

2019-03-26 08:21:06 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2019-03-26 08:21:08 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xE3\" from ASCII-8BIT to UTF-8" plugin_id="out_cloudwatch_logs"

公式さん曰く

このエラーは、文字列エンコーディングがASCII-8BITに設定されているが実際のコンテンツがUTF-8の場合に発生するらしい。

I got enconding error inside plugin. How to fix it?
You may hit "\xC3" from ASCII-8BIT to UTF-8" like UndefinedConversionError in the plugin. This error
happens when string encoding is set to ASCII-8BIT but actual content is UTF-8. Fluentd and almost plugins treat the logs as a ASCII-8BIT by default but some libraries assume the log encoding is UTF-8. This is why this error happens.

There are several approaches to avoid this problem.

Set encoding correctly.
tail input has encoding related parameters to change the log encoding
Use record_modifier filter to change the encoding. See fluent-plugin-record-modifier README
Use yajl instead of json when error happens inside JSON.parse/JSON.dump

fluent-plugin-record-modifier入れて<filter pattern>設定すれば行けそう

fluentd.conf
<filter pattern>
  @type record_modifier

  # set UTF-8 encoding information to string.
  char_encoding utf-8

  # change char encoding from 'UTF-8' to 'EUC-JP'
  char_encoding utf-8:euc-jp
</filter>

確認したら最新版のfluentd.confと差分があった

デフォルトのやつ
# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /templates/conf/fluent.conf.erb

@include kubernetes.conf

<match **>
  @type cloudwatch_logs
  @id out_cloudwatch_logs
  log_group_name "#{ENV['LOG_GROUP_NAME']}"
  auto_create_stream true
  use_tag_as_stream true
</match>
最新のやつ
# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /templates/conf/fluent.conf.erb

@include "#{ENV['FLUENTD_SYSTEMD_CONF'] || 'systemd'}.conf"
@include kubernetes.conf
@include conf.d/*.conf

<match **>
  @type cloudwatch_logs
  @id out_cloudwatch_logs
  log_group_name "#{ENV['LOG_GROUP_NAME']}"
  auto_create_stream true
  use_tag_as_stream true
  json_handler yajl # To avoid UndefinedConversionError
  log_rejected_request "#{ENV['LOG_REJECTED_REQUEST']}" # Log rejected request for missing parts
</match>

・・・ん!?
なんか治ってるらしい。
json_handler yajl # To avoid UndefinedConversionError

いろいろあったけど結論

manifestimagetagを最新のタグに書き換える。以上。

ここを

fluentd-daemonset-cloudwatch-rbac.yaml
image: fluent/fluentd-kubernetes-daemonset:cloudwatch

こうする

fluentd-daemonset-cloudwatch-rbac.yaml
image: fluent/fluentd-kubernetes-daemonset:v1.3-debian-cloudwatch-1

:cloudwatchタグでとってきてたやつが最新じゃなかったみたいです。

1
Help us understand the problem. What is going on with this article?
Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
speedkingg
馬鹿なことを真剣に。

Comments

No comments
Sign up for free and join this conversation.
Sign Up
If you already have a Qiita account Login
1
Help us understand the problem. What is going on with this article?