Apache LogFormat

バージョン確認

$ httpd -v
Server version: Apache/2.4.37 (centos)
Server built:   May 20 2021 04:33:06

デフォルトでは /etc/httpd/conf/httpd.conf にて以下のようなフォーマットが使えます。

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

具体的には /etc/httpd/logs/access_log で、以下のような例を確認できます。

172.68.254.64 - - [13/Aug/2021:01:50:45 +0000] "GET / HTTP/1.1" 200 960 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36"

Cloudflare の HTTP リクエストログとのマッピング

%l, %u はほとんど使われていないフィールドかと思いますが、Cloudflare の HTTP リクエストログには該当する既定フィールドは見つけられていません。

mod_log_config - Apache HTTP Server Version 2.4

HTTP requests · Cloudflare Logs docs

	Apache	Example	Cloudflare	Example
アクセス元のホスト名	%h	172.68.254.64	ClientIP	172.68.254.64
クライアントの識別子	%l	-	該当する既定フィールドなし	該当する既定フィールドなし
認証ユーザー名	%u	-	該当する既定フィールドなし	該当する既定フィールドなし
リクエストを受け付けた時刻	%t	[13/Aug/2021:01:50:45 +0900]	EdgeStartTimestamp	1628819445609000000 (unixnano) 1629438503 (unix) 2021-08-20T05:29:38Z (rfc3339)
リクエストの最初の行の値	%r	GET / HTTP/1.1	ClientRequestMethod ClientRequestURI ClientRequestProtocol	GET / HTTP/1.1
最後のレスポンスのステータス	%>s	200	EdgeResponseStatus	200
送信されたバイト数	%b	960	EdgeResponseBytes	1336
リファラー	%{Referer}i	https://example.com	ClientRequestReferer	https://example.com
User Agent	%{User-Agent}i	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0	ClientRequestUserAgent	Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36

Cloudflare Log を取得し、Apache Combined Log Format 形式に変換

上記のマッピングを参考に、ログを取得・変換するコマンドです。

ClientRequestReferer で値がない場合は - を設定
EdgeStartTimestamp で sort
EdgeStartTimestamp を +32400 で JST に変換
などの処理をおこなって、それぞれのフィールド値を半角スペースで結合

export EMAIL='YOUR_EMAIL'
export APIKEY='YOUR_APIKEY'
export ZONE_ID='YOUR_ZONE_ID'
export FIELDS=ClientIP,EdgeStartTimestamp,ClientRequestMethod,ClientRequestURI,ClientRequestProtocol,EdgeResponseStatus,EdgeResponseBytes,ClientRequestReferer,ClientRequestUserAgent
export APACHE_FIELDS=.${FIELDS//,/,.}

START='2021-08-13T14:30:00%2B09:00'
END='2021-08-13T15:30:00%2B09:00'

curl -s \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logs/received?timestamps=unix&start=$START&end=$END&fields=$FIELDS" | jq -s -r '.|=map(select(.ClientRequestReferer=="").ClientRequestReferer|="-" // .) |sort_by(.EdgeStartTimestamp)|.[]|.EdgeStartTimestamp="- - [" + (.EdgeStartTimestamp|.+32400|strftime("%d/%b/%Y:%H:%M:%S +0900")) + "]"|.ClientRequestMethod="\"" + .ClientRequestMethod | .ClientRequestProtocol = .ClientRequestProtocol + "\"" |.ClientRequestReferer="\""+.ClientRequestReferer+"\""|.ClientRequestUserAgent="\""+.ClientRequestUserAgent+"\""|['"$APACHE_FIELDS"']|join(" ")'

変換後

このように変換しておくと、Apache ログ形式に対応した分析ツール等でそのまま解析できるので、既存のツールを活用できる可能性があります。

result.log

x.x.x.x - - [13/Aug/2021:14:57:48 +0900] "GET / HTTP/1.1" 200 1224 "-" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:14:58:25 +0900] "GET /favicon.ico HTTP/1.1" 404 659 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:58:25 +0900] "GET / HTTP/1.1" 200 1185 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:58:33 +0900] "GET / HTTP/2" 200 1218 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:58:33 +0900] "GET /favicon.ico HTTP/2" 404 775 "https://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:58:53 +0900] "GET / HTTP/2" 200 1237 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:59:51 +0900] "GET / HTTP/1.1" 200 1224 "-" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:14:59:07 +0900] "GET / HTTP/2" 200 1238 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:59:07 +0900] "GET /favicon.ico HTTP/2" 404 783 "https://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:15:06:43 +0900] "GET / HTTP/1.1" 200 1256 "-" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:07:11 +0900] "GET / HTTP/1.1" 200 1262 "-" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:07:14 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:13 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:15 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:16 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:17 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:18 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:21 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"

参考

参考：Apache アクセスログ

$ sudo tail /etc/httpd/logs/access_log
- - - [13/Aug/2021:06:22:15 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:22:16 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:22:17 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:22:18 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:22:21 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:49:29 +0000] "GET / HTTP/1.1" 200 832 "-" "Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers&#39; presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com"
- - - [13/Aug/2021:09:06:41 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "http://example.com/.htaccess" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
- - - [13/Aug/2021:09:08:36 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "https://example.com/.htaccess?param1=1234" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
- - - [13/Aug/2021:09:17:06 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "https://example.com/.htaccess?param1=1234" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
- - - [13/Aug/2021:09:34:30 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "http://example.com/.htaccess?param1=1234" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"

sudo tail -f /etc/httpd/logs/access_log
sudo tail -f /etc/httpd/logs/ssl_access_log

参考：GoAccess で可視化

GoAccess - Visual Web Log Analyzer

brew install goaccess
cat access.log | goaccess -a -o html --log-format COMBINED - > report.html