Elastic Stack (Elasticsearch) Advent Calendar 2023Advent Calendar 2023

Elastic Stackに対してTerraformのImportをやってみた

Last updated at 2023-12-21Posted at 2023-12-21

はじめに

稼働中のElastic Stack環境 (ElasticsearchとKibana）に対して途中からTerraformを使って管理したい場合のTerraform importの仕方について試してみました。
最近はKibana AlertのルールもTerraformで管理できるようになったので、それを例としてやってみました。

なお、Elastic CloudデプロイメントのTerraform化に関しては以下の記事を参照ください。
Elastic Cloudに対してTerraformのImportをやってみた

前提条件

今回のテストはElastic Cloudで動いているElastic Stackを使いますが、オンプレのものでも同じようにできると思います。Elastic Stackはv8.11を使っていますが、管理下にするリソースによっては多少古いバージョンでもできると思います。

やってみること

稼働中のElastic StackをTerraform管理下にする
デフォルトで作られているDefault Kibana SpaceをImportし、Terraform管理下にする
既に作られたこちらの2つのAlertルールをImportし、Terraform管理下にする
新しい3つ目のAlertルールをTerraformから作成する

手順

Terraform tfファイルの作成

以下のファイルを作成します。

elastic-stack.tf

terraform {
  required_version = "~> 1.6"

  required_providers {
    elasticstack = {
      source = "elastic/elasticstack",
      version = "~>0.11"
    }

  }
}

variable "elastic-password" {
  type = string
}

variable "elasticsearch-endpoint" {
  type = string
}

variable "kibana-endpoint" {
  type = string
}

import {
  to = elasticstack_kibana_space.space-001
  id = "${data.elasticstack_elasticsearch_info.cluster_info.cluster_uuid}/default"
}

import {
  to = elasticstack_kibana_alerting_rule.alert-rule-001
  id = "default/2da310f0-9fc1-11ee-b4a5-79169becf22d"
}

import {
  to = elasticstack_kibana_alerting_rule.alert-rule-002
  id = "default/2da310f0-9fc1-11ee-b4a5-79169becf22d"
}

provider "elasticstack" {
  elasticsearch {
    username  = "elastic"
    password  = var.elastic-password
    endpoints = [var.elasticsearch-endpoint]
  }
  kibana {
    username  = "elastic"
    password  = var.elastic-password
    endpoints = [var.kibana-endpoint]
  }
}

data "elasticstack_elasticsearch_info" "cluster_info" {
}

上のファイルでは、Importする対象のAlert RuleのIDをベタで指定しています。ここは各自の環境の値で上書きしてください。この値はAlert Ruleを画面で開いた時のここのURLの部分の文字列となります。

次に、以下の環境変数ファイルを作り、自分の環境の値を設定ください。

terraform.tfvars

elastic-password=""
elasticsearch-endpoint=""
kibana-endpoint=""

今回はelastic-passwordはElastic Cloudの作成の時に作られたelasticユーザーのパスワードを設定します。
endpointはElastic Cloudのデプロイメント管理画面から確認できるエンドポイントURLです。

Terraform Importで取り込み

最初にterraform initコマンドで初期化します。

CLI

% terraform init

Initializing the backend...

Initializing provider plugins...
- Finding elastic/elasticstack versions matching "~> 0.11"...
- Installing elastic/elasticstack v0.11.0...
- Installed elastic/elasticstack v0.11.0 (signed by a HashiCorp partner, key ID 7FE579EDEC6DAA7B)

...

次に、terraform planを実行してみると、importのターゲットとなるリソースが記述されていないためにエラーとなります。このリソースを自動的に生成してくれる-generate-config-outオプションをコマンドを教えてくれます。

CLI

% terraform plan
data.elasticstack_elasticsearch_info.cluster_info: Reading...
data.elasticstack_elasticsearch_info.cluster_info: Read complete after 0s [id=3Q2f9ivpSE6M8TCj_Lnn_w]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Import block target does not exist
│ 
│   on elsaticstack.tf line 25:
│   25: import {
│ 
│ The target for the given import block does not exist. If you wish to automatically generate config for this resource, use the
│ -generate-config-out option within terraform plan. Otherwise, make sure the target resource exists within your configuration.
│ For example:
│ 
│   terraform plan -generate-config-out=generated.tf
╵
...

terraform plan -generate-config-out=generated.tfを実行すると、Importするリソースファイルを生成してくれます。いくつかエラーが表示されているものの、ちゃんとgenerated.tfファイルが作成されているはずです。

CLI

% terraform plan -generate-config-out=generated.tf
elasticstack_kibana_alerting_rule.alert-rule-002: Preparing import... [id=default/c84d79a0-9fbd-11ee-b4a5-79169becf22d]
elasticstack_kibana_alerting_rule.alert-rule-001: Preparing import... [id=default/2df8a7d0-9fbd-11ee-b4a5-79169becf22d]
data.elasticstack_elasticsearch_info.cluster_info: Reading...
elasticstack_kibana_alerting_rule.alert-rule-001: Refreshing state... [id=default/2df8a7d0-9fbd-11ee-b4a5-79169becf22d]
elasticstack_kibana_alerting_rule.alert-rule-002: Refreshing state... [id=default/c84d79a0-9fbd-11ee-b4a5-79169becf22d]
data.elasticstack_elasticsearch_info.cluster_info: Read complete after 0s [id=3Q2f9ivpSE6M8TCj_Lnn_w]
elasticstack_kibana_space.space-001: Preparing import... [id=3Q2f9ivpSE6M8TCj_Lnn_w/default]
elasticstack_kibana_space.space-001: Refreshing state... [id=3Q2f9ivpSE6M8TCj_Lnn_w/default]

Terraform planned the following actions, but then encountered a problem:

  # elasticstack_kibana_space.space-001 will be imported
  # (config will be generated)
    resource "elasticstack_kibana_space" "space-001" {
        color             = "#00bfb3"
        description       = "This is your default space!"
        disabled_features = []
        id                = "3Q2f9ivpSE6M8TCj_Lnn_w/default"
        name              = "Default"
        space_id          = "default"
    }

Plan: 1 to import, 0 to add, 0 to change, 0 to destroy.
╷
│ Warning: Config generation is experimental
│ 
│ Generating configuration during import is currently experimental, and the generated configuration format may change in future
│ versions.
╵
╷
│ Error: expected notify_when to be one of ["onActionGroupChange" "onActiveAlert" "onThrottleInterval"], got 
│ 
│   with elasticstack_kibana_alerting_rule.alert-rule-001,
│   on generated.tf line 5:
│   (source code not available)
│ 
╵
╷
│ Error: expected notify_when to be one of ["onActionGroupChange" "onActiveAlert" "onThrottleInterval"], got 
│ 
│   with elasticstack_kibana_alerting_rule.alert-rule-002,
│   on generated.tf line 5:
│   (source code not available)
│

生成されたファイルはこちら。

generated.tf

# __generated__ by Terraform
# Please review these resources and move them into your main configuration files.

# __generated__ by Terraform
resource "elasticstack_kibana_alerting_rule" "alert-rule-001" {
  consumer     = "alerts"
  enabled      = true
  interval     = "1m"
  name         = "alert-rule-001"
  notify_when  = ""
  params       = "{\"count\":{\"comparator\":\"more than\",\"value\":75},\"criteria\":[{\"comparator\":\"does not match\",\"field\":\"agent\",\"value\":\"\\\"aa\\\"\"}],\"logView\":{\"logViewId\":\"default\",\"type\":\"log-view-reference\"},\"timeSize\":5,\"timeUnit\":\"m\"}"
  rule_id      = "2df8a7d0-9fbd-11ee-b4a5-79169becf22d"
  rule_type_id = "logs.alert.document.count"
  space_id     = "default"
  tags         = []
  throttle     = null
  actions {
    group  = "logs.threshold.fired"
    id     = "elastic-cloud-email"
    params = "{\"message\":\"{{context.reason}}\\n\\n{{rule.name}} is active.\\n\\n{{^context.isRatio}}{{#context.group}}{{context.group}} - {{/context.group}}{{context.matchingDocuments}} log entries have matched the following conditions: {{context.conditions}}{{/context.isRatio}}\\n{{#context.isRatio}}{{#context.group}}{{context.group}} - {{/context.group}} Ratio of the count of log entries matching {{context.numeratorConditions}} to the count of log entries matching {{context.denominatorConditions}} was {{context.ratio}}{{/context.isRatio}}\\n\\n[View alert details]({{context.alertDetailsUrl}})\\n\",\"subject\":\"Test Alert for alert-001\",\"to\":[\"xxx@example.com\"]}"
  }
}

# __generated__ by Terraform
resource "elasticstack_kibana_alerting_rule" "alert-rule-002" {
  consumer     = "alerts"
  enabled      = true
  interval     = "1m"
  name         = "alert-rule-002"
  notify_when  = ""
  params       = "{\"count\":{\"comparator\":\"more than\",\"value\":75},\"criteria\":[{\"comparator\":\"does not match\",\"field\":\"agent\",\"value\":\"\\\"xyz\\\"\"}],\"logView\":{\"logViewId\":\"default\",\"type\":\"log-view-reference\"},\"timeSize\":5,\"timeUnit\":\"m\"}"
  rule_id      = "c84d79a0-9fbd-11ee-b4a5-79169becf22d"
  rule_type_id = "logs.alert.document.count"
  space_id     = "default"
  tags         = []
  throttle     = null
  actions {
    group  = "logs.threshold.fired"
    id     = "elastic-cloud-email"
    params = "{\"message\":\"{{context.reason}}\\n\\n{{rule.name}} is active.\\n\\n{{^context.isRatio}}{{#context.group}}{{context.group}} - {{/context.group}}{{context.matchingDocuments}} log entries have matched the following conditions: {{context.conditions}}{{/context.isRatio}}\\n{{#context.isRatio}}{{#context.group}}{{context.group}} - {{/context.group}} Ratio of the count of log entries matching {{context.numeratorConditions}} to the count of log entries matching {{context.denominatorConditions}} was {{context.ratio}}{{/context.isRatio}}\\n\\n[View alert details]({{context.alertDetailsUrl}})\\n\",\"subject\":\"Test Alert for alert 002\",\"to\":[\"xxx@example.com\"]}"
  }
}

# __generated__ by Terraform from "3Q2f9ivpSE6M8TCj_Lnn_w/default"
resource "elasticstack_kibana_space" "space-001" {
  color             = "#00bfb3"
  description       = "This is your default space!"
  disabled_features = []
  initials          = null
  name              = "Default"
  space_id          = "default"
}

上のファイルのnotify_when = ""の部分だけnotify_when = "onActiveAlert"など値を設定してあげてください。それをしないと、次のplan時に怒られます。

% terraform plan
╷
│ Error: expected notify_when to be one of ["onActionGroupChange" "onActiveAlert" "onThrottleInterval"], got 
│ 
│   with elasticstack_kibana_alerting_rule.alert-rule-001,
│   on generated.tf line 10, in resource "elasticstack_kibana_alerting_rule" "alert-rule-001":
│   10:   notify_when  = ""
│ 
╵
╷
│ Error: expected notify_when to be one of ["onActionGroupChange" "onActiveAlert" "onThrottleInterval"], got 
│ 
│   with elasticstack_kibana_alerting_rule.alert-rule-002,
│   on generated.tf line 30, in resource "elasticstack_kibana_alerting_rule" "alert-rule-002":
│   30:   notify_when  = ""
│ 
╵

terraform planを実行します。

CLI

% terraform plan
elasticstack_kibana_alerting_rule.alert-rule-002: Preparing import... [id=default/c84d79a0-9fbd-11ee-b4a5-79169becf22d]
elasticstack_kibana_alerting_rule.alert-rule-001: Preparing import... [id=default/2df8a7d0-9fbd-11ee-b4a5-79169becf22d]
elasticstack_kibana_alerting_rule.alert-rule-002: Refreshing state... [id=default/c84d79a0-9fbd-11ee-b4a5-79169becf22d]
elasticstack_kibana_alerting_rule.alert-rule-001: Refreshing state... [id=default/2df8a7d0-9fbd-11ee-b4a5-79169becf22d]
data.elasticstack_elasticsearch_info.cluster_info: Reading...
data.elasticstack_elasticsearch_info.cluster_info: Read complete after 0s [id=3Q2f9ivpSE6M8TCj_Lnn_w]
elasticstack_kibana_space.space-001: Preparing import... [id=3Q2f9ivpSE6M8TCj_Lnn_w/default]
elasticstack_kibana_space.space-001: Refreshing state... [id=3Q2f9ivpSE6M8TCj_Lnn_w/default]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  ~ update in-place

Terraform will perform the following actions:

  # elasticstack_kibana_alerting_rule.alert-rule-001 will be updated in-place
  # (imported from "default/2df8a7d0-9fbd-11ee-b4a5-79169becf22d")
  ~ resource "elasticstack_kibana_alerting_rule" "alert-rule-001" {
        consumer              = "alerts"
        enabled               = true
        id                    = "default/2df8a7d0-9fbd-11ee-b4a5-79169becf22d"
        interval              = "1m"
...(省略）...
Plan: 3 to import, 0 to add, 2 to change, 0 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run
"terraform apply" now.

次、terraform applyを実行してImportを実施します。

CLI

% terraform apply
elasticstack_kibana_alerting_rule.alert-rule-001: Preparing import... [id=default/2df8a7d0-9fbd-11ee-b4a5-79169becf22d]
elasticstack_kibana_alerting_rule.alert-rule-002: Preparing import... [
...(省略）...

Plan: 3 to import, 0 to add, 2 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes
...(省略）...

Apply complete! Resources: 3 imported, 0 added, 2 changed, 0 destroyed.

AlertルールをTerraformから変更

2点変更をしてみます。
まず、既存のルールのinterval間隔を1m->5mに変更します。

generated.tf

# __generated__ by Terraform
resource "elasticstack_kibana_alerting_rule" "alert-rule-002" {
  consumer     = "alerts"
  enabled      = true
  interval     = "5m"

次に、新しい3つめのAlert Ruleを作成します（ほぼ他のからコピペ。nameを変更し、rule_idは自動作成されるのでコメントアウト）

generated.tf

resource "elasticstack_kibana_alerting_rule" "alert-rule-003" {
  consumer     = "alerts"
  enabled      = true
  interval     = "1m"
  name         = "alert-rule-003"
  notify_when  = "onActiveAlert"
  params       = "{\"count\":{\"comparator\":\"more than\",\"value\":75},\"criteria\":[{\"comparator\":\"does not match\",\"field\":\"agent\",\"value\":\"\\\"xyz\\\"\"}],\"logView\":{\"logViewId\":\"default\",\"type\":\"log-view-reference\"},\"timeSize\":5,\"timeUnit\":\"m\"}"
  # rule_id      = "c84d79a0-9fbd-11ee-b4a5-79169becf22d"
  rule_type_id = "logs.alert.document.count"
  space_id     = "default"
  tags         = []
  throttle     = null
  actions {
    group  = "logs.threshold.fired"
    id     = "elastic-cloud-email"
    params = "{\"message\":\"{{context.reason}}\\n\\n{{rule.name}} is active.\\n\\n{{^context.isRatio}}{{#context.group}}{{context.group}} - {{/context.group}}{{context.matchingDocuments}} log entries have matched the following conditions: {{context.conditions}}{{/context.isRatio}}\\n{{#context.isRatio}}{{#context.group}}{{context.group}} - {{/context.group}} Ratio of the count of log entries matching {{context.numeratorConditions}} to the count of log entries matching {{context.denominatorConditions}} was {{context.ratio}}{{/context.isRatio}}\\n\\n[View alert details]({{context.alertDetailsUrl}})\\n\",\"subject\":\"Test Alert for alert 002\",\"to\":[\"xxx@example.com\"]}"
  }
}

terraform applyで変更を反映します。

CLI

% terraform apply
...(省略）...
Terraform will perform the following actions:

  # elasticstack_kibana_alerting_rule.alert-rule-002 will be updated in-place
  ~ resource "elasticstack_kibana_alerting_rule" "alert-rule-002" {
        id                    = "default/c84d79a0-9fbd-11ee-b4a5-79169becf22d"
      ~ interval              = "1m" -> "5m"
        name                  = "alert-rule-002"
        tags                  = []
        # (10 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # elasticstack_kibana_alerting_rule.alert-rule-003 will be created
  + resource "elasticstack_kibana_alerting_rule" "alert-rule-003" {
      + consumer              = "alerts"
      + enabled               = true
      + id                    = (known after apply)
      + interval              = "1m"
 ...(省略）...
 
Plan: 1 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

elasticstack_kibana_alerting_rule.alert-rule-002: Modifying... [id=default/c84d79a0-9fbd-11ee-b4a5-79169becf22d]
elasticstack_kibana_alerting_rule.alert-rule-003: Creating...
elasticstack_kibana_alerting_rule.alert-rule-002: Modifications complete after 2s [id=default/c84d79a0-9fbd-11ee-b4a5-79169becf22d]
elasticstack_kibana_alerting_rule.alert-rule-003: Creation complete after 3s [id=default/296e8270-9fd6-11ee-b4a5-79169becf22d]

Apply complete! Resources: 1 added, 1 changed, 0 destroyed.

Terraformによる変更の結果

ちゃんとTerraformからの5minのintervalとなり、rule-003も作成されました！

You get articles that match your needs
You can efficiently read back useful information
You can use dark theme

What you can do with signing up