terraformでeksを管理しつつ、managed nodegroupをspot起動させる

前提

• managed nodegroup自体がspot起動をサポートしていない
• eksctlで作るnodegroupはspot起動がサポートされている。
  • https://eksctl.io/usage/spot-instances/

以下のような設定でspot起動できた。(要awscli)

resource "aws_eks_cluster" "default" {
  name     = "default"
  role_arn = module.aws-iam-role-eks-cluster.aws_iam_role.this.arn

  vpc_config {
    subnet_ids = [
      module.vpc.aws_subnet.private[0].id,
      module.vpc.aws_subnet.private[1].id,
    ]
  }

  version = "1.16"
}

resource "aws_eks_node_group" "default" {
  cluster_name    = aws_eks_cluster.default.name
  node_group_name = "default"
  node_role_arn   = module.aws-iam-role-eks-worker.aws_iam_role.this.arn
  instance_types = [
    "t2.small",
  ]
  subnet_ids      = [
    module.vpc.aws_subnet.private[0].id,
    module.vpc.aws_subnet.private[1].id,
  ]

  scaling_config {
    desired_size = 1
    min_size     = 1
    max_size     = 10
  }

  release_version = "1.16.8-20200507"
  version = "1.16"
  lifecycle {
    ignore_changes = [
      scaling_config[0].desired_size
    ]
  }
}

data "aws_launch_template" "default" {
  filter {
    name   = "tag:eks:cluster-name"
    values = [split(":", aws_eks_node_group.default.id)[0]]
  }
}

locals {
  mixed_instances_policy = {
    LaunchTemplate = {
      LaunchTemplateSpecification = {
        LaunchTemplateId = data.aws_launch_template.default.id
        Version = tostring(data.aws_launch_template.default.latest_version)
      }
    }
    InstancesDistribution = {
      OnDemandBaseCapacity= 0
      OnDemandPercentageAboveBaseCapacity = 0
    }
  }
}

resource "null_resource" "this" {
  triggers = {
    launch_template_change = data.aws_launch_template.default.latest_version
    autoscaling_group_change = aws_eks_node_group.default.resources[0].autoscaling_groups[0].name
  }
  provisioner "local-exec" {
    command = <<-EOT
      aws autoscaling update-auto-scaling-group \
      --mixed-instances-policy '${jsonencode(local.mixed_instances_policy)}' \
      --auto-scaling-group-name ${aws_eks_node_group.default.resources[0].autoscaling_groups[0].name}
    EOT

    interpreter = ["/bin/bash", "-c"]
  }
}

説明

• aws_eks_node_groupはmanaged nodegroup
• 実態はautoscaling groupとlaunch template

awscliで自動作成されるautoscaling groupでspotを使うようにmixed-instances-policyを更新している

resource "null_resource" "this" {
  triggers = {
    launch_template_change = data.aws_launch_template.default.latest_version
    autoscaling_group_change = aws_eks_node_group.default.resources[0].autoscaling_groups[0].name
  }
  provisioner "local-exec" {
    command = <<-EOT
      aws autoscaling update-auto-scaling-group \
      --mixed-instances-policy '${jsonencode(local.mixed_instances_policy)}' \
      --auto-scaling-group-name ${aws_eks_node_group.default.resources[0].autoscaling_groups[0].name}
    EOT

    interpreter = ["/bin/bash", "-c"]
  }
}

null_resourceのtriggersを使うことで更新された場合にコマンドを実行できる。

備考

• managed nodegroupのspotの公式サポートはよこい、、