Cloudflare の HTTP リクエストログを Apache Combined Log Format 形式に変換する

Apache LogFormat

バージョン確認

$ httpd -v
Server version: Apache/2.4.37 (centos)
Server built:   May 20 2021 04:33:06

デフォルトでは /etc/httpd/conf/httpd.conf  にて以下のようなフォーマットが使えます。

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

具体的には /etc/httpd/logs/access_log で、以下のような例を確認できます。

172.68.254.64 - - [13/Aug/2021:01:50:45 +0000] "GET / HTTP/1.1" 200 960 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36"

Cloudflare の HTTP リクエストログとのマッピング

%l, %u はほとんど使われていないフィールドかと思いますが、Cloudflare の HTTP リクエストログには該当する既定フィールドは見つけられていません。

• mod_log_config - Apache HTTP Server Version 2.4
• HTTP requests · Cloudflare Logs docs

	Apache	Example	Cloudflare	Example
アクセス元のホスト名	%h	172.68.254.64	ClientIP	172.68.254.64
クライアントの識別子	%l	-	該当する既定フィールドなし	該当する既定フィールドなし
認証ユーザー名	%u	-	該当する既定フィールドなし	該当する既定フィールドなし
リクエストを受け付けた時刻	%t	[13/Aug/2021:01:50:45 +0900]	EdgeStartTimestamp	1628819445609000000 (unixnano) 1629438503 (unix) 2021-08-20T05:29:38Z (rfc3339)
リクエストの最初の行の値	%r	GET / HTTP/1.1	ClientRequestMethod ClientRequestURI ClientRequestProtocol	GET / HTTP/1.1
最後のレスポンスのステータス	%>s	200	EdgeResponseStatus	200
送信されたバイト数	%b	960	EdgeResponseBytes	1336
リファラー	%{Referer}i	https://example.com	ClientRequestReferer	https://example.com
User Agent	%{User-Agent}i	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0	ClientRequestUserAgent	Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36

Cloudflare Log を取得し、Apache Combined Log Format 形式に変換

上記のマッピングを参考に、ログを取得・変換するコマンドです。

• ClientRequestReferer で値がない場合は - を設定
• EdgeStartTimestamp で sort
• EdgeStartTimestamp を +32400 で JST に変換
• などの処理をおこなって、それぞれのフィールド値を半角スペース で結合

export EMAIL='YOUR_EMAIL'
export APIKEY='YOUR_APIKEY'
export ZONE_ID='YOUR_ZONE_ID'
export FIELDS=ClientIP,EdgeStartTimestamp,ClientRequestMethod,ClientRequestURI,ClientRequestProtocol,EdgeResponseStatus,EdgeResponseBytes,ClientRequestReferer,ClientRequestUserAgent
export APACHE_FIELDS=.${FIELDS//,/,.}

START='2021-08-13T14:30:00%2B09:00'
END='2021-08-13T15:30:00%2B09:00'

curl -s \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logs/received?timestamps=unix&start=$START&end=$END&fields=$FIELDS" | jq -s -r '.|=map(select(.ClientRequestReferer=="").ClientRequestReferer|="-" // .) |sort_by(.EdgeStartTimestamp)|.[]|.EdgeStartTimestamp="- - [" + (.EdgeStartTimestamp|.+32400|strftime("%d/%b/%Y:%H:%M:%S +0900")) + "]"|.ClientRequestMethod="\"" + .ClientRequestMethod | .ClientRequestProtocol = .ClientRequestProtocol + "\"" |.ClientRequestReferer="\""+.ClientRequestReferer+"\""|.ClientRequestUserAgent="\""+.ClientRequestUserAgent+"\""|['"$APACHE_FIELDS"']|join(" ")'

変換後

このように変換しておくと、Apache ログ形式に対応した分析ツール等でそのまま解析できるので、既存のツールを活用できる可能性があります。

result.log

x.x.x.x - - [13/Aug/2021:14:57:48 +0900] "GET / HTTP/1.1" 200 1224 "-" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:14:58:25 +0900] "GET /favicon.ico HTTP/1.1" 404 659 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:58:25 +0900] "GET / HTTP/1.1" 200 1185 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:58:33 +0900] "GET / HTTP/2" 200 1218 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:58:33 +0900] "GET /favicon.ico HTTP/2" 404 775 "https://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:58:53 +0900] "GET / HTTP/2" 200 1237 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:59:51 +0900] "GET / HTTP/1.1" 200 1224 "-" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:14:59:07 +0900] "GET / HTTP/2" 200 1238 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:59:07 +0900] "GET /favicon.ico HTTP/2" 404 783 "https://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:15:06:43 +0900] "GET / HTTP/1.1" 200 1256 "-" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:07:11 +0900] "GET / HTTP/1.1" 200 1262 "-" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:07:14 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:13 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:15 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:16 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:17 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:18 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:21 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"

参考

• Cloudflare の HTTP リクエストログ - Qiita
• Apache | LogFormatディレクティブ：ログファイルのフォーマットを定義する
• jq で エポックミリ秒を変換したい - Qiita
• jq で特定条件にマッチする要素を置換する - なにもわからない
• Date & time formats cheatsheet

参考：Apache アクセスログ

$ sudo tail /etc/httpd/logs/access_log
- - - [13/Aug/2021:06:22:15 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:22:16 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:22:17 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:22:18 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:22:21 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:49:29 +0000] "GET / HTTP/1.1" 200 832 "-" "Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: [e-mail]"
- - - [13/Aug/2021:09:06:41 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "http://example.com/.htaccess" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
- - - [13/Aug/2021:09:08:36 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "https://example.com/.htaccess?param1=1234" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
- - - [13/Aug/2021:09:17:06 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "https://example.com/.htaccess?param1=1234" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
- - - [13/Aug/2021:09:34:30 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "http://example.com/.htaccess?param1=1234" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"

sudo tail -f /etc/httpd/logs/access_log
sudo tail -f /etc/httpd/logs/ssl_access_log

参考：GoAccess で可視化

• GoAccess - Visual Web Log Analyzer

brew install goaccess
cat access.log | goaccess -a -o html --log-format COMBINED - > report.html