Datadog Logs Pipelinesの処理順序をTerraformで管理する

できるようになること

• Pipelineの順番をTerraformで管理できるようになる(プリセットのPipeline含む)
• Pipelineの有効化/無効化をTerraformで管理できるようになる(プリセットのPipeline含む)
  

手順

• DatadogAPIでpipelineの一覧を抽出する(pipeline_idを出力させるため)

curl -X GET https://api.datadoghq.com/api/v1/logs/config/pipelines \
-H "Content-Type: application/json" \
-H "DD-API-KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
-H "DD-APPLICATION-KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

• 目的のものがでてくる("id":"gfewuaiffwoafjowaeg9124")((idは適当な値に変更してます))

[{"id":"gfewuaiffwoafjowaeg9124","type":"pipeline","name":"Java","is_enabled":true,"is_read_only":true,"filter":{"query":"source:java"},"processors":[{"name":"Parsing Java Default formats","is_enabled":true,"source":"message","samples(省略)

• main.tfに以下の書き込み

resource "datadog_logs_integration_pipeline" "java" {
}

• terraform importでterraform.tfstateに落とし込む(backend "s3"はオフにしておく)

terraform import datadog_logs_integration_pipeline.java gfewuaiffwoafjowaeg9124

• terraform.tfstateの当該行に"module"を追加(moduleで構成している場合)

    {
      "module": "module.datadog",　←この行
      "mode": "managed",
      "type": "datadog_logs_integration_pipeline",
      "name": "java",
      "provider": "provider.datadog",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "gfewuaiffwoafjowaeg9124",
            "is_enabled": true
          },
          "private": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        }
      ]
    },

• 後は、datadog_logs_pipeline_orderで順番を変更する。

resource "datadog_logs_integration_pipeline" "python" {
    is_enabled = true
}
resource "datadog_logs_integration_pipeline" "java" {
    is_enabled = true
}
resource "datadog_logs_integration_pipeline" "nginx" {
    is_enabled = false
}

resource "datadog_logs_pipeline_order" "pipeline_order" {
    name = "pipeline_order"
    depends_on = [
      datadog_logs_custom_pipeline.java_test_custom,
      datadog_logs_integration_pipeline.java,
      datadog_logs_integration_pipeline.python,
    ]
    pipelines = [
      datadog_logs_custom_pipeline.java_test_custom.id,
      datadog_logs_integration_pipeline.java.id,
      datadog_logs_integration_pipeline.python.id,
    ]
}

結論

面倒なのでPipelinesの順序は流石に手で変更したほうがいい。
一番面倒なのはプリセットのPipelineは、ログを流し始めるまでpipeline_idが生成されない事。
環境ごとにJavaのログを流す→上記の工程を実施しないといけない。

参考

[https://www.terraform.io/docs/providers/datadog/r/logs_pipeline_order.html:embed:cite]
[https://www.terraform.io/docs/providers/datadog/r/logs_integration_pipeline.html:embed:cite]