9
6

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

embulkでElasticsearchにOutputするときのTimezone指定

Last updated at Posted at 2015-09-08

背景

お仕事で貯まったcsvデータ(日付と時刻情報)をElasticsearchにoutputするときに、日付情報の列がUTCでしか入らないのでハマった

結論

  • default_timezone: 'Asia/Tokyo' を設定する
  • 表記は、JSTじゃなく、 UTCのまま なので注意

embulk version.

$ embulk -v
Embulk v0.7.7

embulk-output-elasticsearchのインストール

  • 2016/2/15 時点で 0.2.1がダウンロードできた。ES2.X系に対応してくれている。
$ embulk gem install embulk-output-elasticsearch
Fetching: embulk-output-elasticsearch-0.2.1.gem (100%)
Successfully installed embulk-output-elasticsearch-0.2.1
1 gem installed

データ:sample.csv

  • 業務で使うデータは以下のような時刻フォーマットが多い。
user,time
sato,2015/5/1 0:00
suzuki,2015/5/2 12:00
yoshida,2015/5/3 23:00

embulkのconfig.yml

  • guessコマンドで生成したやつの一例
    • default_timezone: 'Asia/Tokyo'は下記のように入れる
in:
  type: file
  path_prefix: sample.csv
  parser:
    charset: UTF-8
    newline: CRLF
    type: csv
    delimiter: ','
    quote: '"'
    escape: '"'
    trim_if_not_quoted: false
    skip_header_lines: 1
    allow_extra_columns: false
    allow_optional_columns: false
    default_timezone: 'Asia/Tokyo'
    columns:
    - {name: user, type: string}
    - {name: time, type: timestamp, format: '%Y/%m/%d %H:%M'}
out:
  type: elasticsearch
  index: sample
  index_type: sample
  cluster_name: sample
  nodes:
  - {host: example.ne.jp, port: XXXX}

default_timezone: 'Asia/Tokyo'がないとき

  • そのまま、UTC表記になってしまう。
$ embulk preview config.yml
2015-09-08 03:19:45.928 +0000: Embulk v0.7.4
2015-09-08 03:19:48.325 +0000 [INFO] (preview): Listing local files at directory '.' filtering filename by prefix 'sample.csv'
2015-09-08 03:19:48.338 +0000 [INFO] (preview): Loading files [sample.csv]
+-------------+-------------------------+
| user:string |          time:timestamp |
+-------------+-------------------------+
|        sato | 2015-05-01 00:00:00 UTC |
|      suzuki | 2015-05-02 12:00:00 UTC |
|     yoshida | 2015-05-03 23:00:00 UTC |
+-------------+-------------------------+

default_timezone: 'Asia/Tokyo'があるとき

  • 時刻が、-9:00されてる
$ embulk preview config.yml
2015-09-08 03:23:26.862 +0000: Embulk v0.7.4
2015-09-08 03:23:29.504 +0000 [INFO] (preview): Listing local files at directory '.' filtering filename by prefix 'sample.csv'
2015-09-08 03:23:29.518 +0000 [INFO] (preview): Loading files [sample.csv]
+-------------+-------------------------+
| user:string |          time:timestamp |
+-------------+-------------------------+
|        sato | 2015-04-30 15:00:00 UTC |
|      suzuki | 2015-05-02 03:00:00 UTC |
|     yoshida | 2015-05-03 14:00:00 UTC |
+-------------+-------------------------+

その他

  • UTCじゃなくて、JSTに表記できないのかな。
9
6
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
9
6

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?