Azure Storage Account の Blob コンテナーに Fluent Bit からデータを送る検証をしてみた

個人的には、オンプレミスや複数のパブリッククラウドから様々なログデータを保管する事を考えた場合、容量制限や保管期間や運用コストを考えると、オブジェクトストレージ一択なのではないかと思っています。

そこで今回は、Azure Storage Account の Blob コンテナーに Fluent Bit からデータを送る検証をしてみました。

ローカルで Fluent Bit の動作確認

Fluent Bit を MacOS にインストール

私の環境は Homebrew で Fluent Bit をインストールしました。

bash

$ brew install fluent-bit
==> Fetching fluent-bit
==> Downloading https://ghcr.io/v2/homebrew/core/fluent-bit/manifests/2.0.9
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/fluent-bit/blobs/sha256:0463fea8a38a358abccc5194f9fc829dd5e406f10f94201349c8df4ff112939c
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:0463fea8a38a358abccc5194f9fc829dd5e406f10f94201349c8
######################################################################## 100.0%
==> Pouring fluent-bit--2.0.9.arm64_ventura.bottle.tar.gz
🍺  /opt/homebrew/Cellar/fluent-bit/2.0.9: 170 files, 10.2MB
==> Running `brew cleanup fluent-bit`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

Fluent Bit の動作確認

dummy メッセージが何回か表示されたところで [Ctrl + C] で Fluent Bit を止めます。

bash

$ fluent-bit -i dummy -o stdout -f 1
Fluent Bit v2.0.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/02/25 09:15:07] [ info] [fluent bit] version=2.0.9, commit=, pid=7920
[2023/02/25 09:15:07] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/02/25 09:15:07] [ info] [cmetrics] version=0.5.8
[2023/02/25 09:15:07] [ info] [ctraces ] version=0.2.7
[2023/02/25 09:15:07] [ info] [input:dummy:dummy.0] initializing
[2023/02/25 09:15:07] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/02/25 09:15:07] [ info] [output:stdout:stdout.0] worker #0 started
[2023/02/25 09:15:07] [ info] [sp] stream processor started
[0] dummy.0: [1677284108.544548000, {"message"=>"dummy"}]
[0] dummy.0: [1677284109.542850000, {"message"=>"dummy"}]
[0] dummy.0: [1677284110.547118000, {"message"=>"dummy"}]
[0] dummy.0: [1677284111.546998000, {"message"=>"dummy"}]
[2023/02/25 09:15:12] [engine] caught signal (SIGINT)
[2023/02/25 09:15:12] [ info] [input] pausing dummy.0
[2023/02/25 09:15:12] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/02/25 09:15:12] [ info] [output:stdout:stdout.0] thread worker #0 stopped

検証用 Azure Storage Account を作成

bash

prefix=mnrfluentbit

az group create \
  --name ${prefix}-rg \
  --location japaneast

az storage account create \
  --name ${prefix}stor \
  --resource-group ${prefix}-rg \
  --sku Standard_LRS

storagekey=$(az storage account keys list \
  --account-name ${prefix}stor \
  --resource-group ${prefix}-rg \
  --query "[0].value" \
  --output tsv)

Fluent Bit 設定ファイルを作成

bash

cat <<EOF > azure-blog.conf
[SERVICE]
    flush     1
    log_level info

[INPUT]
    name      dummy
    dummy     {"name": "Fluent Bit", "year": 2020}
    samples   1
    tag       var.log.containers.app-default-96cbdef2340.log

[OUTPUT]
    name                  azure_blob
    match                 *
    account_name          ${prefix}stor
    shared_key            $storagekey
    path                  kubernetes
    container_name        logs
    auto_create_container on
    tls                   on
EOF

Blob コンテナーにデータを送る Fluent Bit を実行

content appended to blob successfully メッセージが表示されたところで [Ctrl + C] で Fluent Bit を止めます。

bash

$ fluent-bit -c azure-blog.conf
Fluent Bit v2.0.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/02/25 09:23:55] [ info] [fluent bit] version=2.0.9, commit=, pid=8772
[2023/02/25 09:23:55] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/02/25 09:23:55] [ info] [cmetrics] version=0.5.8
[2023/02/25 09:23:55] [ info] [ctraces ] version=0.2.7
[2023/02/25 09:23:55] [ info] [input:dummy:dummy.0] initializing
[2023/02/25 09:23:55] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/02/25 09:23:55] [ info] [output:azure_blob:azure_blob.0] account_name=mnrfluentbitstor, container_name=logs, blob_type=appendblob, emulator_mode=no, endpoint=mnrfluentbitstor.blob.core.windows.net
[2023/02/25 09:23:55] [ info] [sp] stream processor started
[2023/02/25 09:23:57] [ info] [output:azure_blob:azure_blob.0] container 'logs' not found, trying to create it
[2023/02/25 09:23:57] [ info] [output:azure_blob:azure_blob.0] container 'logs' created sucessfully
[2023/02/25 09:23:57] [ info] [output:azure_blob:azure_blob.0] blob not found: 
[2023/02/25 09:23:57] [ info] [output:azure_blob:azure_blob.0] blob created successfully: 
[2023/02/25 09:23:57] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2023/02/25 09:24:02] [engine] caught signal (SIGINT)
[2023/02/25 09:24:02] [ info] [input] pausing dummy.0

Blob コンテナーの状態確認

azure-blog.conf で指定した logs コンテナーの kubernetes パスの下にログファイルが作成されました。

ログファイルの中身

ar.log.containers.app-default-96cbdef2340.log

{"@timestamp":"2023-02-25T00:23:56.147395Z","name":"Fluent Bit","year":2020}

後片付け

rm azure-blog.conf

az group delete \
  --name ${prefix}-rg \
  --yes

参考