Help us understand the problem. What is going on with this article?

Apache ログを awk と uniq だけで集計する

More than 3 years have passed since last update.

Apache生ログワンライナーで解析するパターン集。
例えば以下のようなリクエスト別のアクセス数など。

$ grep 01/Feb/2016:01 access_log | awk -F '"' '{print $2}' | awk '{print $2}' | sort | uniq -c | sort -n -r
     69 /
     48 /en/
     35 /css/base.css
     23 /favicon.ico
     23 /img/example.jpg

とりあえずコピペ用

時間単位別リクエスト数(日別、時間別、分別、秒別)

grep 'May/2016' /var/log/httpd/access_log | awk '{print $4}' | cut -b 2-12 | sort | uniq -c          # 日別
grep '01/May/2016' /var/log/httpd/access_log | awk '{print $4}' | cut -b 2-15 | sort | uniq -c       # 時間別
grep '01/May/2016:01' /var/log/httpd/access_log | awk '{print $4}' | cut -b 2-18 | sort | uniq -c    # 分別
grep '01/May/2016:01:00' /var/log/httpd/access_log | awk '{print $4}' | cut -b 2-21 | sort | uniq -c # 秒別

項目別リクエスト数(リクエスト別、UA別、ブラウザ別、リファラー別、IP別)

grep '01/May/2016:01' /var/log/httpd/access_log | awk -F \" '{print $2}' | awk '{print $2}' | sort | uniq -c | sort -nr # リクエスト別
grep '01/May/2016:01' /var/log/httpd/access_log | awk -F \" '{print $6}' | sort | uniq -c | sort -nr | head -n 5        # UserAgent別
for UA in MSIE Firefox Chrome Safari; do COUNT=`grep '01/May/2016:01' /var/log/httpd/access_log | grep "$UA" | wc -l`; echo "$UA: $COUNT"; done
grep '01/May/2016:01' /var/log/httpd/access_log | awk -F \" '{print $4}' | sort | uniq -c | sort -nr # リファラー別
grep '01/May/2016:01' /var/log/httpd/access_log | cut -d " " -f 1 | sort | uniq -c                   # IP別
静的ファイルの除外
 | grep -ive "GET /.*\.\(css\|js\|jpg\|gif\|png\|swf\|ico\)\ HTTP"
指定キーワードのリクエスト総数
grep '01/May/2016:01' /var/log/httpd/access_log | grep -c 'favicon.ico'

時間単位別リクエスト数

時間単位別リクエスト数を集計するために、まず grep コマンドで日時などのテキストで全体の行を絞り込んで取得、 awk コマンドで4番目の日時データを切り出し、さらに特定の時間単位で集計するために cut コマンドで範囲を指定して切り出し、sort コマンドで時間順に並び替えてから、最後に uniq コマンドで集計して終わり。

日別リクエスト数

grep 'Feb/2016' access_log | awk '{print $4}' | cut -b 2-12 | sort | uniq -c
   5960 01/Feb/2016
   7493 02/Feb/2016
   7023 03/Feb/2016
   7292 04/Feb/2016
   7144 05/Feb/2016

grep で年月 (ex.Feb/2016) を、cut に 2-12 を指定してやると日別のアクセス数が出力できる。

時間別リクエスト数

grep 01/Feb/2016 access_log | awk '{print $4}' | cut -b 2-15 | sort | uniq -c
    149 01/Feb/2016:00
    384 01/Feb/2016:01
    465 01/Feb/2016:02
    328 01/Feb/2016:03
    272 01/Feb/2016:04

grep で年月日 (ex.01/Feb/2016) を、cut に 2-15 を指定してやると時間別のアクセス数が出力できる。

分別リクエスト数

grep 01/Feb/2016:01 access_log | awk '{print $4}' | cut -b 2-18 | sort | uniq -c
      7 01/Feb/2016:01:00
     13 01/Feb/2016:01:01
      6 01/Feb/2016:01:02
      6 01/Feb/2016:01:03
      2 01/Feb/2016:01:04

grep で年月日と時 (ex.01/Feb/2016:01) を、cut に 2-18 を指定してやると分別のアクセス数が出力できる。

秒別リクエスト数

grep 01/Feb/2016:01:00 access_log | awk '{print $4}' | cut -b 2-21 | sort | uniq -c
      1 01/Feb/2016:01:00:00
      2 01/Feb/2016:01:00:10
      1 01/Feb/2016:01:00:12
      1 01/Feb/2016:01:00:23
      2 01/Feb/2016:01:00:25

grep で年月日と時分 (ex.01/Feb/2016:01:00) を、cut に 2-21 を指定してやると秒別のアクセス数が出力できる。

項目別リクエスト数

リクエスト別アクセス数

grep 01/Feb/2016:01 access_log | awk -F '"' '{print $2}' | awk '{print $2}' | sort | uniq -c | sort -n -r | head -n 5
     69 /
     48 /en/
     35 /css/base.css
     23 /favicon.ico
     23 /img/example.jpg

grep に絞り込みたい値を渡してやると、リクエスト別のアクセス数が出力できる。
上記の例では年月日と時 (01/Feb/2016:01) を渡しているので、2016年2月1日1〜2時のリクエスト別アクセス数ということになる。

ユーザーエージェント別リクエスト数

grep 01/Feb/2016:01 access_log | awk -F '"' '{print $6}' | sort | uniq -c | sort -n -r | head -n 5
     20 Mozilla/5.0 (iPhone; CPU iPhone OS 9_2_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13D15 Safari/601.1
     19 Mozilla/5.0 (Linux; U; Android 4.4.2; en-us; GT-N7100 Build/KOT49H) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
     18 Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
     17 Mozilla/5.0 (Windows NT 6.1; rv:43.0) Gecko/20100101 Firefox/43.0
     17 Mozilla/5.0 (Linux; Android 5.0; Lenovo A7000-a Build/LRX21M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.95 Mobile Safari/537.36

ブラウザ別リクエスト数

for UA in MSIE Firefox Chrome Safari; do COUNT=`grep '01/Feb/2016:01' /var/log/httpd/access_log | grep "$UA" | wc -l`; echo "$UA: $COUNT"; done
MSIE: 33
Firefox: 398
Chrome: 1163
Safari: 1996

その他

圧縮されたログファイルのアクセス数

ls /var/log/httpd/ 
access_log        access_log.4.gz   error_log        error_log.4.gz 
access_log.1      access_log.5.gz   error_log.1      error_log.5.gz 
access_log.2.gz   access_log.6.gz   error_log.2.gz   error_log.6.gz 
access_log.3.gz   access_log.7.gz   error_log.3.gz   error_log.7.gz 
zgrep 01/Feb/2016 /var/log/httpd/access_log* | awk -F '"' '{print $2}' | awk '{print $2}' | sort | uniq -c | sort -n -r
     69 /
     48 /en/
     35 /css/base.css
     23 /favicon.ico
     23 /img/example.jpg

Apache 関連記事

bezeklik
サーバー管理、CMS構築のお仕事募集中。 CentOS / Apache / Nginx / PHP / MySQL / Docker / CMS (eZ Platform, eZ Publish, WordPress, Drupal) / Redmine
Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
Comments
No comments
Sign up for free and join this conversation.
If you already have a Qiita account
Why do not you register as a user and use Qiita more conveniently?
You need to log in to use this function. Qiita can be used more conveniently after logging in.
You seem to be reading articles frequently this month. Qiita can be used more conveniently after logging in.
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
ユーザーは見つかりませんでした