Linux
LTSV

LTSVフォーマットから該当の項目のみを抽出

概要

アクセスログをLTSVフォーマットに変更した。
既存のログと大きな違いは以下の2点

  • タブ区切り
  • ラベルが付けられる

今までgrepやawkを駆使して該当項目を抽出していたが、
lltsvコマンドをインストールすることにより、作業がグッと楽になった。

使い方

例えば以下のようなアクセスログがあるとする。

time_local:11/Dec/2017:12:09:25 +0900   request:"GET /admin/widget/ HTTP/1.1"   http_x_forwarded_for:"-"    uid_got:"-" req_body_start:""   
time_local:11/Dec/2017:12:09:25 +0900   request:"GET /admin/widget/ HTTP/1.0"   http_x_forwarded_for:"192.168.120.120"  uid_got:"app_uid=060012AC5A99275A050070A502030303"  req_body_start:""   
time_local:11/Dec/2017:12:09:26 +0900   request:"GET /admin/event/ HTTP/1.1"    http_x_forwarded_for:"-"    uid_got:"-" req_body_start:""   
time_local:11/Dec/2017:12:09:26 +0900   request:"GET /admin/event/ HTTP/1.0"    http_x_forwarded_for:"192.168.120.120"  uid_got:"app_uid=060012AC5A99275A050070A502030303"  req_body_start:""

このログから

  • time_local
  • http_x_forwarded_for

のみを抽出したい場合、以下のようなコマンドで簡単に抽出できる

# tail access.log | lltsv -k time_local,http_x_forwarded_for

time_local:11/Dec/2017:12:09:22 +0900   http_x_forwarded_for:"-"
time_local:11/Dec/2017:12:09:22 +0900   http_x_forwarded_for:"192.168.120.120"
time_local:11/Dec/2017:12:09:22 +0900   http_x_forwarded_for:"-"
time_local:11/Dec/2017:12:09:22 +0900   http_x_forwarded_for:"192.168.120.120"
time_local:11/Dec/2017:12:09:24 +0900   http_x_forwarded_for:"-"
time_local:11/Dec/2017:12:09:24 +0900   http_x_forwarded_for:"192.168.120.120"
time_local:11/Dec/2017:12:09:25 +0900   http_x_forwarded_for:"-"
time_local:11/Dec/2017:12:09:25 +0900   http_x_forwarded_for:"192.168.120.120"
time_local:11/Dec/2017:12:09:26 +0900   http_x_forwarded_for:"-"
time_local:11/Dec/2017:12:09:26 +0900   http_x_forwarded_for:"192.168.120.120"

# 

インストール方法

yum install wget -y
wget https://github.com/sonots/lltsv/releases/download/v0.3.0/lltsv_linux_amd64 -O lltsv
chmod a+x lltsv
mv lltsv /usr/sbin/.

参考情報

http://blog.livedoor.jp/sonots/archives/39628163.html
http://tech.mercari.com/entry/2017/05/09/152428