Elastic Cloudに対してTerraformのImportをやってみた

はじめに

Elastic CloudのデプロイメントをTerraformを使って管理したいけど、既に動いているElastic Cloudだから中々できていない人もいるのではないでしょうか？
幸いTerraformにはimportという仕組みがあり、既に動いている環境に対応する術がありますが、それでも手間importで作られるtfstateからtfファイルのリソース定義を作るといった手間があり微調整基本のterraform applyだけではTerraform化できません。・管理は以前から実践していたのですが、既に作られたデプロイメントを途中からTerraformで管理したいときは？　TerraformのImportでやってみました。

なお、Elastic Stackの方のTerraform化については以下の記事を参照ください。
Elastic Stackに対してTerraformのImportをやってみた

手順

Elastic CloudのAPIキーの取得

最初にElastic Cloudの管理画面のこのページからAPIキーを作成してください。

Elastic CloudのデプロイメントのIDの取得

作成済みのElastic CloudデプロイメントのIDを以下のURL部分から取得します。

Terraform importでデプロイメントをTerraform管理下にする

今回のテスト用フォルダを適当作り、そこに以下のelastic-cloud.tf(名前は何でも良い）を作成します。
バージョンに関しては必ずしもこれである必要はありませんが、他の下位バージョンでどうなるかはテストしてみないとわかりません。Terraformのバージョンはimport blockがサポートされた最低1.5以上が必要です（実際には1.5ではテストしておらず、1.6でテストしています）

elastic-cloud.tf

terraform {
  required_version = "~> 1.6"

  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.8.0"
    }

  }
}

variable "ec-apikey" {
  type = string
}

variable "deployment-id" {
  type = string
}

import {
  to = ec_deployment.custom-deployment-1220
  id = var.deployment-id
}

provider "ec" {
    apikey = var.ec-apikey
}

変数ファイルは別に以下のように作成しましょう。xxxのところは上の手順で取得した値を入れてください。

terraform.tfvars

deployment-id = "xxx"
ec-apikey = "xxx"

terraform initを実行します。

CLI

% terraform init

Initializing the backend...

Initializing provider plugins...
- Finding elastic/elasticstack versions matching "0.6.2"...
- Finding elastic/ec versions matching "0.8.0"...
- Installing elastic/elasticstack v0.6.2...
- Installed elastic/elasticstack v0.6.2 (signed by a HashiCorp partner, key ID 7FE579EDEC6DAA7B)
- Installing elastic/ec v0.8.0...
- Installed elastic/ec v0.8.0 (signed by a HashiCorp partner, key ID 7FE579EDEC6DAA7B)

...(省略)...

次に、terraform planを実行してみると、importのターゲットとなるリソースが記述されていないためにエラーとなります。このリソースを自動的に生成してくれる-generate-config-outオプションをコマンドを教えてくれます。

CLI

% terraform plan

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Import block target does not exist
│ 
│   on elastic-cloud.tf line 26:
│   26: import {
│ 
│ The target for the given import block does not exist. If you wish to automatically generate config for this resource, use the
│ -generate-config-out option within terraform plan. Otherwise, make sure the target resource exists within your configuration. For
│ example:
│ 
│   terraform plan -generate-config-out=generated.tf

terraform plan -generate-config-out=generated.tfとコマンドを打ちます。（まだこのgenerateコマンドはexperimentalですね）

CLI

% terraform plan -generate-config-out=generated.tf
ec_deployment.custom-deployment-1220: Preparing import... [id=a74bd4bd0b574c26badba5624fdbd58d]
ec_deployment.custom-deployment-1220: Refreshing state... [id=a74bd4bd0b574c26badba5624fdbd58d]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Warning: Config generation is experimental
│ 
│ Generating configuration during import is currently experimental, and the generated configuration format may change in future
│ versions.
╵
╷
│ Error: attribute "elasticsearch": attributes "cloud_id", "http_endpoint", "https_endpoint", "region", and "resource_id" are required
│ 
│ 

エラーが出力されていますが、それでもちゃんとgenerated.tfファイルは以下の通り作成されていました。

generated.tf

# __generated__ by Terraform
# Please review these resources and move them into your main configuration files.

# __generated__ by Terraform
resource "ec_deployment" "custom-deployment-1220" {
  alias                  = "nobu-terraform-test1220"
  apm                    = null
  deployment_template_id = "gcp-storage-optimized"
  elasticsearch = {
    autoscale = false
    cold = {
      autoscaling = {
        max_size          = "64g"
        max_size_resource = "memory"
        min_size          = null
        min_size_resource = null
      }
      node_type_data   = null
      node_type_ingest = null
      node_type_master = null
      node_type_ml     = null
      size             = "0g"
      size_resource    = "memory"
      zone_count       = 1
    }
    config = {
      docker_image                = null
      plugins                     = []
      user_settings_json          = null
      user_settings_override_json = null
      user_settings_override_yaml = null
      user_settings_yaml          = null
    }
    coordinating = {
      autoscaling = {
        max_size          = null
        max_size_resource = null
        min_size          = null
        min_size_resource = null
      }
      node_type_data   = null
      node_type_ingest = null
      node_type_master = null
      node_type_ml     = null
      size             = "0g"
      size_resource    = "memory"
      zone_count       = 2
    }
    extension = null
    frozen = {
      autoscaling = {
        max_size          = "128g"
        max_size_resource = "memory"
        min_size          = null
        min_size_resource = null
      }
      node_type_data   = null
      node_type_ingest = null
      node_type_master = null
      node_type_ml     = null
      size             = "0g"
      size_resource    = "memory"
      zone_count       = 1
    }
    hot = {
      autoscaling = {
        max_size          = "128g"
        max_size_resource = "memory"
        min_size          = null
        min_size_resource = null
      }
      node_type_data   = null
      node_type_ingest = null
      node_type_master = null
      node_type_ml     = null
      size             = "2g"
      size_resource    = "memory"
      zone_count       = 1
    }
    master = {
      autoscaling = {
        max_size          = null
        max_size_resource = null
        min_size          = null
        min_size_resource = null
      }
      node_type_data   = null
      node_type_ingest = null
      node_type_master = null
      node_type_ml     = null
      size             = "0g"
      size_resource    = "memory"
      zone_count       = 3
    }
    ml = {
      autoscaling = {
        max_size          = "64g"
        max_size_resource = "memory"
        min_size          = "0g"
        min_size_resource = "memory"
      }
      node_type_data   = null
      node_type_ingest = null
      node_type_master = null
      node_type_ml     = null
      size             = "0g"
      size_resource    = "memory"
      zone_count       = 1
    }
    ref_id         = "main-elasticsearch"
    remote_cluster = null
    snapshot = {
      enabled    = true
      repository = null
    }
    snapshot_source = null
    strategy        = null
    trust_account = [
      {
        account_id      = "3293981696"
        trust_all       = true
        trust_allowlist = null
      },
    ]
    trust_external = [
    ]
    warm = {
      autoscaling = {
        max_size          = "128g"
        max_size_resource = "memory"
        min_size          = null
        min_size_resource = null
      }
      node_type_data   = null
      node_type_ingest = null
      node_type_master = null
      node_type_ml     = null
      size             = "0g"
      size_resource    = "memory"
      zone_count       = 2
    }
  }
  enterprise_search = null
  integrations_server = {
    config                       = null
    elasticsearch_cluster_ref_id = "main-elasticsearch"
    endpoints = {
      apm   = "https://nobu-terraform-test1220.apm.asia-northeast1.gcp.cloud.es.io"
      fleet = "https://nobu-terraform-test1220.fleet.asia-northeast1.gcp.cloud.es.io"
    }
    instance_configuration_id = "gcp.integrationsserver.n2.68x32x45"
    ref_id                    = "main-integrations_server"
    size                      = "1g"
    size_resource             = "memory"
    zone_count                = 1
  }
  kibana = {
    config                       = null
    elasticsearch_cluster_ref_id = "main-elasticsearch"
    instance_configuration_id    = "gcp.kibana.n2.68x32x45"
    ref_id                       = "main-kibana"
    size                         = "1g"
    size_resource                = "memory"
    zone_count                   = 1
  }
  name                         = "nobu-terraform-test1220"
  observability                = null
  region                       = "gcp-asia-northeast1"
  request_id                   = null
  reset_elasticsearch_password = null
  tags                         = null
  traffic_filter               = []
  version                      = "8.11.3"
}

リソース定義がこれでできたので、再びplanを実行します。

CLI

 % terraform plan                                  
ec_deployment.custom-deployment-1220: Preparing import... [id=a74bd4bd0b574c26badba5624fdbd58d]
ec_deployment.custom-deployment-1220: Refreshing state... [id=a74bd4bd0b574c26badba5624fdbd58d]

Terraform will perform the following actions:

  # ec_deployment.custom-deployment-1220 will be imported
    resource "ec_deployment" "custom-deployment-1220" {
        alias                  = "nobu-terraform-test1220"
        deployment_template_id = "gcp-storage-optimized"
        elasticsearch          = {
            autoscale      = false
...(省略)...

Plan: 1 to import, 0 to add, 0 to change, 0 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run
"terraform apply" now.

Plan: 1 to import, 0 to add, 0 to change, 0 to destroy.とあるので、このままapplyで反映してもImportがされるだけでその他の影響は受けません。

では、Terraform applyをします。

CLI

% terraform apply
ec_deployment.custom-deployment-1220: Preparing import... [id=a74bd4bd0b574c26badba5624fdbd58d]
ec_deployment.custom-deployment-1220: Refreshing state... [id=a74bd4bd0b574c26badba5624fdbd58d]

Terraform will perform the following actions:

  # ec_deployment.custom-deployment-1220 will be imported
    resource "ec_deployment" "custom-deployment-1220" {
        alias                  = "nobu-terraform-test1220"
        deployment_template_id = "gcp-storage-optimized"
...(省略)...

Plan: 1 to import, 0 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

ec_deployment.custom-deployment-1220: Importing... [id=a74bd4bd0b574c26badba5624fdbd58d]
ec_deployment.custom-deployment-1220: Import complete [id=a74bd4bd0b574c26badba5624fdbd58d]

Apply complete! Resources: 1 imported, 0 added, 0 changed, 0 destroyed.

無事、importされました。

Terraformでデプロイメントの設定変更をしてみる

これでTerraform管理ができるようになったので、実際に一部設定変更をTerraformから行ってみます。
generated.tfを開き、Hotティアのメモリsizesを2gから4gに変更します。

generated.tfの抜粋

    hot = {
      autoscaling = {
        max_size          = "128g"
        max_size_resource = "memory"
        min_size          = null
        min_size_resource = null
      }
      node_type_data   = null
      node_type_ingest = null
      node_type_master = null
      node_type_ml     = null
      size             = "4g"
      size_resource    = "memory"
      zone_count       = 1
    }

terraform applyをします。

CLI

% terraform apply
ec_deployment.custom-deployment-1220: Refreshing state... [id=a74bd4bd0b574c26badba5624fdbd58d]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  ~ update in-place

Terraform will perform the following actions:

  # ec_deployment.custom-deployment-1220 will be updated in-place
  ~ resource "ec_deployment" "custom-deployment-1220" {
      + apm_secret_token       = (sensitive value)
      ~ elasticsearch          = {
          ~ hot            = {
              ~ node_roles                = [
                  - "data_content",
                  - "data_hot",
                  - "ingest",
                  - "master",
                  - "remote_cluster_client",
                  - "transform",
                ] -> (known after apply)
              ~ size                      = "2g" -> "4g"
                # (4 unchanged attributes hidden)
            }
          ~ snapshot       = {
              + repository = (known after apply)
                # (1 unchanged attribute hidden)
            }
            # (16 unchanged attributes hidden)
        }
        id                     = "a74bd4bd0b574c26badba5624fdbd58d"
        name                   = "nobu-terraform-test1220"
        # (7 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

ec_deployment.custom-deployment-1220: Modifying... [id=a74bd4bd0b574c26badba5624fdbd58d]
ec_deployment.custom-deployment-1220: Still modifying... [id=a74bd4bd0b574c26badba5624fdbd58d, 10s elapsed]

UIの方で確認すると、ちゃんと設定変更が始まってますね！

最後、私のこのテストにおいてはdestroyコマンドでデプロイメントを削除しましたが、みなさんの環境ではやらないように気をつけましょう。（デプロイメントが削除されてしまうので）