awkとシェルでLTSVの取り扱いを簡単にするフィルタを書いてみた

なるほど。

上の記事を見てrubyでもやってみた。

# !/usr/bin/env ruby

def ltsv(line)
        log = {}
        line.strip.split("\t").each {|kv|
                k,v = kv.split(":", 2)
                log[k] = v
        }
        log
end

tail -f access.log | ruby -r./ltsv -ne 'log=ltsv($_);p log if log["status"]=="404"'

もっと短く書けないかと試行錯誤していたらこれでもよかった

tail -f access.log | ruby -ne 'log=Hash[$_.strip.split("\t").map{|kv|kv.split(":",2)}];p log if log["status"]=="404"'

流れるログを「トータルの処理時間が大きい順」に並べて見たい

# !/usr/bin/env ruby
# tail -f /var/log/nginx/access.log | ruby ltsvstream.rb

while true
        summary = {}
        time = nil
        while line = STDIN.gets
                log = Hash[line.strip.split("\t").map{|kv|kv.split(":",2)}]
                time = log['time'] if time.nil?
                break if time != log['time']
                req = log['method'] + ' ' + log['path']
                summary[req] = {count:0,time:0} if summary[req].nil?
                summary[req][:count] += 1
                summary[req][:time] += log['req_time'].to_f
        end

        puts "=== #{time} ==="
        summary.sort_by{|req, detail| detail[:time]}.reverse.each do |req,detail|
                puts "#{'%.2f'%detail[:time]}\t#{detail[:count]}\t#{req}"
        end
end

こんな感じ

=== 2013-11-06T02:02:00+09:00 ===
54.29   7       GET /ticket/2
21.92   8       GET /
11.12   4       POST /buy
10.96   4       GET /artist/1
=== 2013-11-06T02:02:01+09:00 ===
100.95  14      GET /ticket/2
15.83   6       GET /
8.27    3       GET /artist/1
=== 2013-11-06T02:02:02+09:00 ===
51.33   7       GET /ticket/2
32.36   12      GET /
10.67   4       POST /buy
8.76    3       GET /artist/1
=== 2013-11-06T02:02:03+09:00 ===
31.22   4       GET /ticket/2
28.16   11      GET /artist/1
15.61   7       GET /
8.61    4       POST /buy
=== 2013-11-06T02:02:04+09:00 ===
39.55   5       GET /ticket/2
23.31   11      GET /artist/1
11.96   6       GET /
6.04    3       POST /buy

ltsvの形式は昨日と同じ