LoginSignup
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

@khayama(Kyouhei Hayama)inCloudflare Japan 株式会社

Cloudflare の Audit Log を Google Cloud Storage に Logpush する

Last updated at Posted at 2021-10-28

Audit Log Retention

デフォルトで 18 ヶ月保持されますが、それ以上遡って保管したい場合は、Logpush で随時外部にエクスポートすることができます。

Understanding Cloudflare Audit Logs – Cloudflare Help Center

To maintain Audit Logs query performance, the public Audit Logs API was modified on Jun 30 2019 to return records with a maximum age of 18 months.

こちらのドキュメントを参考に進めます。

Manage Logpush with cURL · Cloudflare Logs docs

アカウントを特定する変数をセット

API 経由で Logpush Job を設定するために、変数をセットします。

Cloudflare API v4 Documentation

export EMAIL='YOUR_EMAIL'
export APIKEY='YOUR_APIKEY'
export ACCOUNT_ID='YOUR_ACCOUNT_ID'
export DATASET='audit_logs' # can be gateway_dns, gateway_http
export BUCKET='kyouhei'
export FOLDER='audit_log'
export JOB_NAME='audit-log' # cannot be changed after job creation. don't use "_"

Ownership challenge を取得

Google Cloud Storage を Logpush のターゲットにするための Ownership challenge を行います。

  • 事前に Google Cloud Storage でバケット kyouhei とディレクトリ audit_log を作成します。
  • {DATE} を指定することでディレクトリ内でログを日付ごとのフォルダに分けて保管できます。
export CHALLENGE_PATH=$(curl -s -X POST "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/logpush/ownership" \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
--data '{"destination_conf":"gs://'$BUCKET'/'$FOLDER'/{DATE}"}' | jq -r .result.filename)
# Ownership challenge を取得
export CHALLENGE=$(gsutil cat gs://$BUCKET/$CHALLENGE_PATH)
echo $CHALLENGE
result.txt
eyJhbGciOiJkaXIiLCJlbmMiOxxxxxxxxxx

Field を変数にセット

以下のコマンドでそれぞれセットします。

# 全フィールドを指定
export FIELDS=$(curl -s \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
"https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/logpush/datasets/$DATASET/fields" | jq -r '.result | keys | join(",")')
echo $FIELDS

Logpush Job を作成

dataset を audit_logs として、account レベルの Logpush Job を作成します。

curl -s -X POST \
https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/logpush/jobs \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
-d'{"name": "'$JOB_NAME'", "destination_conf":"gs://'$BUCKET'/'$FOLDER'/{DATE}", "dataset": "'$DATASET'", "logpull_options":"fields='$FIELDS'&timestamps=rfc3339", "ownership_challenge":"'$CHALLENGE'"}' | jq .
result.json
{
  "errors": [],
  "messages": [],
  "result": {
    "id": 42883,
    "dataset": "audit_logs",
    "frequency": "high",
    "kind": "",
    "enabled": false,
    "name": audit-log,
    "logpull_options": "fields=ActionResult,ActionType,ActorEmail,ActorID,ActorIP,ActorType,ID,Interface,Metadata,NewValue,OldValue,OwnerID,ResourceID,ResourceType,When&timestamps=rfc3339",
    "destination_conf": "gs://kyouhei/audit_log/{DATE}",
    "last_complete": null,
    "last_error": null,
    "error_message": null
  },
  "success": true
}

作成した Logpush Job を有効化

JobID を指定して、enabled が false になっているのを true にセットして有効化します。

curl -s -X PUT \
-d'{"enabled":true}' \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/logpush/jobs/42883 | jq .
result.json
{
  "errors": [],
  "messages": [],
  "result": {
    "id": 42883,
    "dataset": "audit_logs",
    "frequency": "high",
    "kind": "",
    "enabled": true,
    "name": null,
    "logpull_options": "fields=ActionResult,ActionType,ActorEmail,ActorID,ActorIP,ActorType,ID,Interface,Metadata,NewValue,OldValue,OwnerID,ResourceID,ResourceType,When&timestamps=rfc3339",
    "destination_conf": "gs://kyouhei/audit_log/{DATE}",
    "last_complete": null,
    "last_error": null,
    "error_message": null
  },
  "success": true
}

Logpush Job のリスト（アカウントレベル）

アカウントレベルでの Logpush Job リストを確認できます。

last_complete が最後にエクスポートが実行された時間です。

curl -X GET "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/logpush/jobs" \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" | jq .
result.json
{
  "errors": [],
  "messages": [],
  "result": [
    {
      "id": 42883,
      "dataset": "audit_logs",
      "frequency": "high",
      "kind": "",
      "enabled": true,
      "name": null,
      "logpull_options": "fields=ActionResult,ActionType,ActorEmail,ActorID,ActorIP,ActorType,ID,Interface,Metadata,NewValue,OldValue,OwnerID,ResourceID,ResourceType,When&timestamps=rfc3339",
      "destination_conf": "gs://kyouhei/audit_log/{DATE}",
      "last_complete": "2021-10-28T02:24:15Z",
      "last_error": null,
      "error_message": null
    }
  ],
  "success": true
}

Logpush の確認

UI

Account Home > Audit Log から確認できます。

image-20211028151833667
image-20211028151934446

image-20211028152035419

Google Cloud Storage

該当するログファイルを確認できました。

gsutil cat gs://kyouhei/audit_log/20211027/20211027T051004Z_20211027T051034Z_32068b24.log.gz | gunzip | jq .
result.json
{
  "ActionResult": true,
  "ActionType": "waiting_rooms_edit",
  "ActorEmail": "xxx",
  "ActorID": "xx",
  "ActorIP": "240d:xxx",
  "ActorType": "user",
  "ID": "0a2fc5f6-4900-46ed-9364-c31a7cd9a110",
  "Interface": "UI",
  "Metadata": {
    "actor_email": "xxx",
    "zone_name": "xxx",
    "zone_tag": "xxx"
  },
  "NewValue": {
    "id": "ea05cb62a47a8e66e387002eae48ab1d",
    "name": "waiting_room",
    "description": "",
    "host": "wait.xxx",
    "path": "/",
    "suspended": false,
    "queue_all": false,
    "new_users_per_minute": 200,
    "total_active_users": 200,
    "session_duration": 5,
    "custom_page_html": "",
    "disable_session_renewal": false,
    "json_response_enabled": false,
    "queueing_method": "fifo",
    "cookie_attributes": {
      "samesite": "auto",
      "secure": "auto"
    },
    "created_on": "2021-10-20T05:14:59.449217Z",
    "modified_on": "2021-10-27T05:08:09.273506Z"
  },
  "OldValue": {
    "id": "ea05cb62a47a8e66e387002eae48ab1d",
    "name": "wait",
    "description": "",
    "host": "wait.xxx",
    "path": "/",
    "suspended": false,
    "queue_all": false,
    "new_users_per_minute": 200,
    "total_active_users": 200,
    "session_duration": 5,
    "custom_page_html": "",
    "disable_session_renewal": false,
    "json_response_enabled": false,
    "queueing_method": "fifo",
    "cookie_attributes": {
      "samesite": "auto",
      "secure": "auto"
    },
    "created_on": "2021-10-20T05:14:59.449217Z",
    "modified_on": "2021-10-22T05:45:19.625349Z"
  },
  "OwnerID": "xxx",
  "ResourceID": "xxx",
  "ResourceType": "Waiting Room",
  "When": "2021-10-27T05:08:09Z"
}

参考：Logpush Job を削除

JobID を指定して Logpush Job を削除できます。

curl -s -X DELETE \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/logpush/jobs/42080 | jq .
result.json
{
  "errors": [],
  "messages": [],
  "result": null,
  "success": true
}

参考：Google Cloud Storage にある最新10個のログファイルの中身を表示

export BUCKET='kyouhei'
export FOLDER='audit_log'

gsutil list gs://$BUCKET/$FOLDER/$(date +"%Y%m%d") | grep .log.gz | tail -n 10 | while read line
do
  gsutil cat $line | gunzip | jq .
done

参考；ZONE ベースのスクリプト

export ZONE_ID='YOUR_ZONE_ID'
---
# Ownership challenge を取得
export CHALLENGE_PATH=$(curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/ownership" \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
--data '{"destination_conf":"gs://'$BUCKET'/'$FOLDER'/{DATE}"}' | jq -r .result.filename)

export CHALLENGE=$(gsutil cat gs://$BUCKET/$CHALLENGE_PATH)
echo $CHALLENGE
---
# 全フィールドを指定
export FIELDS=$(curl -s \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/datasets/$DATASET/fields" | jq -r '.result | keys | join(",")')
echo $FIELDS
---
# ジョブ作成
curl -s -X POST \
https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/jobs \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
-d'{"name": "'$JOB_NAME'", "destination_conf":"gs://'$BUCKET'/'$FOLDER'/{DATE}", "dataset": "'$DATASET'", "logpull_options":"fields='$FIELDS'&timestamps=rfc3339", "ownership_challenge":"'$CHALLENGE'"}' | jq .
---
# 有効化
curl -s -X PUT \
-d'{"enabled":true}' \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/jobs/42883 | jq .
---
# ジョブのリスト
curl -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/jobs" \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" | jq .
1
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?