Cloudflare の HTTP リクエストログ

Cloudflare Logs で利用できる機能一覧

ログ取得には、Logpush の利用が推奨されます。
今回は Logpull を使って、簡単に手元で確認してみます。

Log Type	Download	Download Retention	Export to Cloud Storage / SIEM	Export Retention
Audit Logs	Audit Logs API	18 months	Logpush (ENT only)	No retention
HTTP requests	Logpull (ENT only)	at least 3 days (up to 7 days)	Logpush (ENT only)	No retention
Spectrum events	-	-	Logpush (ENT only)	No retention
Firewall events	-	-	Logpush (ENT only)	No retention
NEL reports	-	-	Logpush (ENT only)	No retention

HTTP requests ログ保持機能の有効化

デフォルトではログを保持しない設定になっているので、ドキュメントにしたがって有効化します。

Enabling log retention · Cloudflare Logs docs

export EMAIL='YOUR_EMAIL'
export APIKEY='YOUR_APIKEY'
export ZONE_ID='YOUR_ZONE_ID'

# 設定の確認
curl -s \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logs/control/retention/flag" | jq

# 有効化
curl -s \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logs/control/retention/flag" -d'{"flag":true}' | jq

HTTP requests ログフィールド一覧

以下のコマンドでフィールド一覧を取得できます。

# JSON で表示
curl -s \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logs/received/fields" | jq

# 全 Field 名をカンマ区切りで取得し変数に格納
export FIELDS=$(curl -s \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logs/received/fields" | jq -r 'keys | join(",")')
echo $FIELDS

fields.json

{
  "CacheCacheStatus": "string; unknown | miss | expired | updating | stale | hit | ignored | bypass | revalidated",
  "CacheResponseBytes": "int; number of bytes returned by the cache",
  "CacheResponseStatus": "int; HTTP status code returned by the cache to the edge; all requests (including non-cacheable ones) go through the cache; also see CacheStatus field",
  "CacheTieredFill": "bool; tiered Cache was used to serve this request",
  "ClientASN": "int; client AS number",
  "ClientCountry": "string; country of the client IP address",
  "ClientDeviceType": "string; client device type",
  "ClientIP": "string; IP address of the client",
  "ClientIPClass": "string; unknown | clean | badHost | searchEngine | allowlist | greylist | monitoringService | securityScanner | noRecord | scan | backupService | mobilePlatform | tor",
  "ClientRequestBytes": "int; number of bytes in the client request",
  "ClientRequestHost": "string; host requested by the client",
  "ClientRequestMethod": "string; HTTP method of client request",
  "ClientRequestPath": "string; URI path requested by the client",
  "ClientRequestProtocol": "string; HTTP protocol of client request",
  "ClientRequestReferer": "string; HTTP request referrer",
  "ClientRequestURI": "string; URI requested by the client",
  "ClientRequestUserAgent": "string; user agent reported by the client",
  "ClientSSLCipher": "string; client SSL cipher",
  "ClientSSLProtocol": "string; client SSL (TLS) protocol",
  "ClientSrcPort": "int; client source port",
  "ClientXRequestedWith": "string; X-Requested-With HTTP header",
  "EdgeColoCode": "string; IATA airport code of data center that received the request",
  "EdgeColoID": "int; Cloudflare edge colo id",
  "EdgeEndTimestamp": "int or string; timestamp at which the edge finished sending response to the client",
  "EdgePathingOp": "string; indicates what type of response was issued for this request (unknown = no specific action)",
  "EdgePathingSrc": "string; details how the request was classified based on security checks (unknown = no specific classification)",
  "EdgePathingStatus": "string; indicates what data was used to determine the handling of this request (unknown = no data)",
  "EdgeRateLimitAction": "string; the action taken by the blocking rule; empty if no action taken",
  "EdgeRateLimitID": "int; the internal rule ID of the rate-limiting rule that triggered a block (ban) or simulate action. 0 if no action taken.",
  "EdgeRequestHost": "string; host header on the request from the edge to the origin",
  "EdgeResponseBytes": "int; number of bytes returned by the edge to the client",
  "EdgeResponseCompressionRatio": "float; edge response compression ratio",
  "EdgeResponseContentType": "string; edge response Content-Type header value",
  "EdgeResponseStatus": "int; HTTP status code returned by Cloudflare to the client",
  "EdgeServerIP": "string; IP of the edge server making a request to the origin",
  "EdgeStartTimestamp": "int or string; timestamp at which the edge received request from the client",
  "FirewallMatchesActions": "array[string]; array of actions the Cloudflare firewall products performed on this request. The individual firewall products associated with this action be found in FirewallMatchesSources and their respective RuleIds can be found in FirewallMatchesRuleIDs. The length of the array is the same as FirewallMatchesRuleIDs and FirewallMatchesSources. Possible actions are allow | log | block | challenge | jschallenge | connectionClose | bypass",
  "FirewallMatchesRuleIDs": "array[string]; array of RuleIDs of the firewall product that has matched the request. The firewall product associated with the RuleID can be found in FirewallMatchesSources. The length of the array is the same as FirewallMatchesActions and FirewallMatchesSources.",
  "FirewallMatchesSources": "array[string]; the firewall products that matched the request. The same product can appear multiple times, which indicates different rules or actions that were activated. The RuleIDs can be found in FirewallMatchesRuleIDs, the actions can be found in FirewallMatchesActions. The length of the array is the same as FirewallMatchesRuleIDs and FirewallMatchesActions. Possible sources are asn | country | ip | ipRange | securityLevel | zoneLockdown | waf | firewallRules | uaBlock | rateLimit | bic | hot | l7ddos | validation | protect",
  "OriginIP": "string; IP of the origin server",
  "OriginResponseBytes": "int; number of bytes returned by the origin server",
  "OriginResponseHTTPExpires": "string; value of the origin 'expires' header in RFC1123 format",
  "OriginResponseHTTPLastModified": "string; value of the origin 'last-modified' header in RFC1123 format",
  "OriginResponseStatus": "int; status returned by the origin server",
  "OriginResponseTime": "int; number of nanoseconds it took the origin to return the response to edge",
  "OriginSSLProtocol": "string; SSL (TLS) protocol used to connect to the origin",
  "ParentRayID": "string; Ray ID of the parent request if this request was made using a Worker script",
  "RayID": "string; ID of the request",
  "SecurityLevel": "string; the security level configured at the time of this request. This is used to determine the sensitivity of the IP Reputation system.",
  "WAFAction": "string; action taken by the WAF, if triggered",
  "WAFFlags": "string; additional configuration flags: simulate (0x1) | null",
  "WAFMatchedVar": "string; the full name of the most-recently matched variable",
  "WAFProfile": "string; low | med | high",
  "WAFRuleID": "string; ID of the applied WAF rule",
  "WAFRuleMessage": "string; rule message associated with the triggered rule",
  "WorkerCPUTime": "int; amount of time in microseconds spent executing a worker, if any",
  "WorkerStatus": "string; status returned from worker daemon",
  "WorkerSubrequest": "bool; whether or not this request was a worker subrequest",
  "WorkerSubrequestCount": "int; number of subrequests issued by a worker when handling this request",
  "ZoneID": "int; internal zone ID"
}

HTTP requests ログ取得（Logpull）

• 最大で7日前まで日時が指定できます。
  • bad query: error parsing time: invalid time range: too early: logs older than 168h0m0s are not available
• 最大で1時間まで時間範囲が指定できます。
  • bad query: error parsing time: invalid time range: too long: maximum query range (difference between start and end) is 1h0m0s

# JST (+09:00) で時間範囲を指定
START='2021-07-29T11:00:00%2B09:00'
END='2021-07-29T12:00:00%2B09:00'

# 全フィールド値のログを取得
curl -s \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logs/received?start=$START&end=$END&fields=$FIELDS" | jq

example.json

{
  "CacheCacheStatus": "hit",
  "CacheResponseBytes": 1187,
  "CacheResponseStatus": 200,
  "CacheTieredFill": false,
  "ClientASN": 2527,
  "ClientCountry": "jp",
  "ClientDeviceType": "desktop",
  "ClientIP": "x.x.x.x",
  "ClientIPClass": "noRecord",
  "ClientRequestBytes": 2257,
  "ClientRequestHost": "xxx.example.com",
  "ClientRequestMethod": "HEAD",
  "ClientRequestPath": "/",
  "ClientRequestProtocol": "HTTP/2",
  "ClientRequestReferer": "",
  "ClientRequestURI": "/",
  "ClientRequestUserAgent": "curl/7.64.1",
  "ClientSSLCipher": "ECDHE-ECDSA-CHACHA20-POLY1305",
  "ClientSSLProtocol": "TLSv1.2",
  "ClientSrcPort": 64289,
  "ClientXRequestedWith": "",
  "EdgeColoCode": "NRT",
  "EdgeColoID": 22,
  "EdgeEndTimestamp": 1627524070604000000,
  "EdgePathingOp": "wl",
  "EdgePathingSrc": "macro",
  "EdgePathingStatus": "nr",
  "EdgeRateLimitAction": "",
  "EdgeRateLimitID": 0,
  "EdgeRequestHost": "xxx.example.com",
  "EdgeResponseBytes": 1224,
  "EdgeResponseCompressionRatio": 1,
  "EdgeResponseContentType": "image/png",
  "EdgeResponseStatus": 200,
  "EdgeServerIP": "",
  "EdgeStartTimestamp": 1627524070472000000,
  "FirewallMatchesActions": [],
  "FirewallMatchesRuleIDs": [],
  "FirewallMatchesSources": [],
  "OriginIP": "",
  "OriginResponseBytes": 0,
  "OriginResponseHTTPExpires": "",
  "OriginResponseHTTPLastModified": "",
  "OriginResponseStatus": 0,
  "OriginResponseTime": 0,
  "OriginSSLProtocol": "unknown",
  "ParentRayID": "00",
  "RayID": "6762c2006cad0ab4",
  "SecurityLevel": "med",
  "WAFAction": "unknown",
  "WAFFlags": "0",
  "WAFMatchedVar": "",
  "WAFProfile": "unknown",
  "WAFRuleID": "",
  "WAFRuleMessage": "",
  "WorkerCPUTime": 1651,
  "WorkerStatus": "ok",
  "WorkerSubrequest": false,
  "WorkerSubrequestCount": 1,
  "ZoneID": 111111111
}

jq を使って整形

こちらに例があります。

Parse Cloudflare Logs JSON data · Cloudflare Logs docs