モチベーション

業務で k8s のマルチクラスタ構成について検討する機会があったのですが、設計段階でも実際に環境を触りたいケースがあると思います。
そんな時にパッと環境構築出来たら検証作業が捗るなーという思いから、Terraform で自動化してみました。

やること

Terraform でマルチクラスタを構築する
- EKSクラスタ & GKEクラスタの構成
クラスタ間をVPNで接続する
- Internal Load Balancer 経由でクラスタ間通信

やりたいことは、ざっくり下図のようなイメージとなります。
VPNの箇所はだいぶ端折ってますが、悪しからず、、

ディレクトリ構成

当初はクラウドリソース毎にモジュール化しようと思っていたのですが、途中で力尽きました。

.
|-- Dockerfile
|-- docker-compose.yaml
|-- .env
|-- credentials
|   `-- t-matsuno-xxxxxx.json
|-- backend
|   `-- main.tf
|-- environments
|   `-- test
|       |-- backend.tf
|       |-- main.tf
|       |-- provider.tf
|       |-- terraform.tfvars
|       |-- variables.tf
|       `-- versions.tf
`-- modules
    |-- aws
    |   |-- eks.tf
    |   |-- main.tf
    |   |-- multicloud_vpn.tf
    |   |-- outputs.tf
    |   |-- variables.tf
    |   `-- vpc.tf
    `-- gcp
        |-- bastion.tf
        |-- gke.tf
        |-- nat.tf
        |-- outputs.tf
        |-- variables.tf
        `-- vpc.tf

実行環境

docker-compose で実行環境をコンテナとして起動します。
用途に応じたコンテナイメージを用意することで、ローカル環境を汚さずに済みます。

Dockerfile

FROM gcr.io/google.com/cloudsdktool/google-cloud-cli:alpine

ENV TF_CLI_ARGS_plan   "--parallelism=30"
ENV TF_CLI_ARGS_apply  "--parallelism=100"

# Update and install packages
RUN apk update && \
  apk add --no-cache \
  bash \
  unzip \
  curl \
  git \
  vim \
  jq \
  aws-cli

# Install kubectl
RUN curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl" && \
  chmod +x kubectl && \
  mv kubectl /usr/local/bin/

# Install Helm
ARG HELM_VERSION=3.13.2
RUN curl https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz -o helm.tar.gz && \
  tar -zxvf helm.tar.gz && \
  mv linux-amd64/helm /usr/local/bin/helm && \
  rm -rf linux-amd64 && \
  rm helm.tar.gz

# Install Terraform
ARG TERRAFORM_VERSION=1.6.3
RUN curl -LO https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \
  unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \
  mv terraform /usr/bin/ && \
  rm terraform_${TERRAFORM_VERSION}_linux_amd64.zip

WORKDIR /work

ENTRYPOINT bash
CMD ["-c", "sleep", "infinity"]

.env に各クラウド用の環境変数を定義します。
認証情報もここに含めてしまいます。

.env

AWS_ACCESS_KEY_ID=xxxxxxxxxx
AWS_SECRET_ACCESS_KEY=yyyyyyyyyy
AWS_DEFAULT_REGION=ap-northeast-1
AWS_DEFAULT_OUTPUT=json

CLOUDSDK_CORE_PROJECT=<GCP プロジェクト名>
CLOUDSDK_CORE_ACCOUNT=<GCPアカウントのメールアドレス>
CLOUDSDK_COMPUTE_REGION=asia-northeast2
CLOUDSDK_COMPUTE_ZONE=asia-northeast2-c

# GCPサービスアカウントキーを配置する
GOOGLE_CREDENTIALS=/work/credentials/t-matsuno-xxxxxx.json

先ほどのコンテナイメージを起動するためのdocker-compose.yamlを作成します。
.env の内容はコンテナの環境変数として参照可能となります。

docker-compose.yaml

version: '3'

services:
  terraform:
    build:
      context: .
      dockerfile: ./Dockerfile
    env_file:
      - .env
    volumes:
      - ./credentials:/work/credentials
      - ./backend:/work/backend
      - ./modules:/work/modules
      - ./environments:/work/environments
      - tfdata:/tfdata
    tty: true
    environment:
      - TF_DATA_DIR=/tfdata

volumes:
  tfdata:

起動してコンテナにログインします。

$ podman compose up -d
...
[+] Running 1/1
 ✔ Container terraform-multicloud-terraform-1  Started
 
$ podman exec -it terraform-multicloud-terraform-1 sh
 
 /work # terraform --version
Terraform v1.6.3
on linux_amd64

Your version of Terraform is out of date! The latest version
is 1.6.5. You can update by downloading from https://www.terraform.io/downloads.html

backend 作成

Terraformの tfstateファイルを GCS で管理するため、最初に Cloud Storage バケットを作成します。
今回は検証目的なので、無料枠が適用される条件でバケットを作成します。

backend/main.tf

provider "google" {
  project = <GCP プロジェクト名>
}

resource "google_storage_bucket" "terraform-state-store" {
  name     = "tmatsuno-gke-test-tfstate"
  location = "us-west1"
  storage_class = "REGIONAL"
  versioning {
    enabled = true
  }
  lifecycle_rule {
    action {
      type = "Delete"
    }
    condition {
      num_newer_versions = 5
    }
  }
}

上記の内容で、Terraformを実行します。

/work # cd backend
# ワークスペースを初期化
/work/backend # terraform init
# 変更内容を確認
/work/backend # terraform plan
# ↑で確認した内容を実行
/work/backend # terraform apply -auto-approve

以後の Terraform 実行手順は割愛します。

GCP リソースの定義

GCP の各種リソース定義を modules/gcp 配下に作成します。

modules/gcp/vpc.tf

VPCとサブネットを定義します。
VPN 接続時の通信を許可するための Firewall Rule も先に作成してしまいます。

modules/gcp/vpc.tf

# VPC
resource "google_compute_network" "vpc_network" {
  project = var.gcp_common.project
  name    = "${var.common.prefix}-${var.common.env}-gke-vpc"
  auto_create_subnetworks = false
  routing_mode = "GLOBAL"
}

# サブネット
resource "google_compute_subnetwork" "subnet_gke" {
  project       = var.gcp_common.project
  name          = "${var.common.prefix}-${var.common.env}-subnet"
  ip_cidr_range = var.gcp_vpc.subnet_cidr
  region        = var.gcp_common.region
  network       = google_compute_network.vpc_network.id
  private_ip_google_access = true

  secondary_ip_range {
    range_name    = "service-ranges-${var.common.prefix}-${var.common.env}"
    ip_cidr_range = var.gcp_vpc.secondary_ip_service_ranges
  }

  secondary_ip_range {
    range_name    = "pod-ranges-${var.common.prefix}-${var.common.env}"
    ip_cidr_range = var.gcp_vpc.secondary_ip_pod_ranges
  }
}

# AWS からのインバウンド通信を許可
resource "google_compute_firewall" "fr_to_aws" {
  name    = "${var.common.prefix}-${var.common.env}-firewall-up-to-aws"
  network = google_compute_network.vpc_network.id
  direction     = "INGRESS"
  priority      = 900
  source_ranges = [var.aws_vpc.vpc_cidr]
  allow {
    protocol = "udp"
    ports    = ["0-65535"]
  }
  allow {
    protocol = "tcp"
    ports    = ["0-65535"]
  }
  allow {
    protocol = "icmp"
  }
}

# AWS へのアウトバウンド通信を許可
resource "google_compute_firewall" "fr_from_aws" {
  name    = "${var.common.prefix}-${var.common.env}-firewall-down-from-aws"
  network = google_compute_network.vpc_network.id
  direction          = "EGRESS"
  priority           = 900
  destination_ranges = [var.aws_vpc.vpc_cidr]
  allow {
    protocol = "udp"
    ports    = ["0-65535"]
  }
  allow {
    protocol = "tcp"
    ports    = ["0-65535"]
  }
  allow {
    protocol = "icmp"
  }
}

modules/gcp/gke.tf

ip_allocation_policy.{cluster,services}_secondary_range_name と VPC の secondary_ip_range.range_name を一致させる必要があります。

modules/gcp/gke.tf

# GKE 限定公開クラスタ
resource "google_container_cluster" "primary" {
  project          = var.gcp_common.project
  name             = "${var.common.prefix}-${var.common.env}-cluster"
  location         = var.gcp_common.region
  network          = google_compute_network.vpc_network.id
  subnetwork       = google_compute_subnetwork.subnet_gke.id
  deletion_protection      = false
  initial_node_count       = 1
  min_master_version       = var.gcp_gke.cluster_version
  networking_mode          = "VPC_NATIVE"
  remove_default_node_pool = true
  enable_l4_ilb_subsetting = true
  ip_allocation_policy {
    cluster_secondary_range_name  = "pod-ranges-${var.common.prefix}-${var.common.env}"
    services_secondary_range_name = "service-ranges-${var.common.prefix}-${var.common.env}"
  }
  release_channel {
    channel = "STABLE"
  }

  # 限定公開クラスタの設定
  private_cluster_config {
    enable_private_nodes    = true
    enable_private_endpoint = true
    master_ipv4_cidr_block  = var.gcp_gke.master_cidr
    master_global_access_config {
      enabled = false
    }
  }
  master_authorized_networks_config {
    # コントロールプレーンへのアクセスを許可する IP 範囲
    cidr_blocks {
      # ノードと踏み台が作られるサブネットからのアクセスを許可
      cidr_block = var.gcp_vpc.subnet_cidr
    }
  }
  maintenance_policy {
    recurring_window {
      start_time = "2023-11-01T00:00:00Z"
      end_time   = "2023-11-01T04:00:00Z"
      recurrence = "FREQ=WEEKLY;BYDAY=FR,SA,SU"
    }
  }
}

# ノードプール
resource "google_container_node_pool" "primary_nodes" {
  name       = "${var.common.prefix}-${var.common.env}-node-pool"
  location   = var.gcp_common.region
  cluster    = google_container_cluster.primary.name
  node_count = 1
  autoscaling {
    min_node_count = 1
    max_node_count = 3
  }
  upgrade_settings {
    max_surge       = 1   
    max_unavailable = 0
  }
  management {
    auto_repair  = true
    auto_upgrade = true
  }

  node_config {
    preemptible  = var.gcp_gke.preemptible
    machine_type = var.gcp_gke.machine_type
    disk_size_gb = 20
    service_account = var.gcp_common.email
    tags            = ["gke-node", "${var.gcp_common.project}-gke"]
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
    ]
    labels = {
      env = var.common.env
    }
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }
}

modules/gcp/nat.tf

GKE からインターネットにアクセスできるようにするため、Cloud NAT を作成します。

modules/gcp/nat.tf

# Cloud Router
resource "google_compute_router" "nat_router" {
  project = var.gcp_common.project
  name    = "${var.common.prefix}-${var.common.env}-nat-router"
  region  = var.gcp_common.region
  network = google_compute_network.vpc_network.id
}

# Cloud NAT
resource "google_compute_router_nat" "nat" {
  project = var.gcp_common.project
  name    = "${var.common.prefix}-${var.common.env}-nat"
  router  = google_compute_router.nat_router.name
  region  = google_compute_router.nat_router.region
  nat_ip_allocate_option             = "AUTO_ONLY"
  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"

  log_config {
    enable = true
    filter = "ERRORS_ONLY"
  }
}

modules/gcp/bastion.tf

GKE クラスタにアクセスするための踏み台サーバを用意します。
metadata_startup_script で必要なツールを予めインストールできて便利です。

modules/gcp/bastion.tf

# 踏み台 VM
resource "google_compute_instance" "bastion" {
  project      = var.gcp_common.project
  name         = "${var.common.prefix}-${var.common.env}-bastion"
  machine_type = var.gcp_bastion.machine_type
  zone         = var.gcp_common.zone
  tags         = ["ssh"]

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"
    }
  }
  network_interface {
    subnetwork_project = var.gcp_common.project
    network    = google_compute_network.vpc_network.name
    subnetwork = google_compute_subnetwork.subnet_gke.name
    access_config {}
  }
  metadata = {
    # OS Login を有効化
    enable-oslogin = "true"
  }

  # 起動スクリプトで必要なツールと GKE プラグインをインストール
  metadata_startup_script = <<EOF
#!/bin/bash
sudo apt update
sudo apt install kubectl unzip make
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
sudo apt install google-cloud-sdk-gke-gcloud-auth-plugin

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
rm awscliv2.zip

gcloud config set compute/region ${var.gcp_common.region}
gcloud config set compute/zone ${var.gcp_common.zone}
EOF

  scheduling {
    # 料金を抑えるためにプリエンプティブルにしておく
    preemptible = true
    # プリエンプティブルの場合は下のオプションが必須
    automatic_restart = false
  }
}

resource "google_compute_firewall" "ssh" {
  project = var.gcp_common.project
  name    = "${var.common.prefix}-${var.common.env}-vpc-ssh-allow"
  network = google_compute_network.vpc_network.name
  target_tags = ["ssh"]
  direction   = "INGRESS"
  allow {
    protocol = "tcp"
    ports    = ["22"]
  }
  source_ranges = [
    var.gcp_bastion.ssh_sourcerange
  ]
}

modules/gcp/variables.tf

変数は variables.tf に定義します。

modules/gcp/variables.tf

variable "common" {
  type = object ({
    prefix  = string
    env     = string
  })
  description = "リソース共通の設定値"
}

variable "gcp_common" {
  type = object ({
    project = string
    region  = string
    zone    = string
    email   = string
  })
  description = "GCPリソース共通の設定値"
}

variable "gcp_vpc" {
  type = object ({
    subnet_cidr = string
    secondary_ip_service_ranges = string
    secondary_ip_pod_ranges = string
  })
  description = "GCP VPC の設定値"
}

variable "gcp_gke" {
  type = object ({
    master_cidr  = string
    preemptible  = bool
    machine_type = string
    cluster_version = string
  })
  description = "GKE の設定値"
}

variable "gcp_bastion" {
  type = object ({
    machine_type    = string
    ssh_sourcerange = string
  })
  description = "GCP 踏み台 VM の設定値"
}

variable "aws_vpc" {
  type = object ({
    vpc_cidr = string
    subnet_availability_zones = list(string)
    public_subnet_cidr = list(string)
    private_subnet_cidr = list(string)
  })
  description = "AWS VPC の設定値"
}

modules/gcp/outputs.tf

terraform output で表示したいパラメータを outputs.tf に定義します。

modules/gcp/outputs.tf

output "vpc_network_id" {
  value = google_compute_network.vpc_network.id
}
output "cluster_name" {
  value = google_container_cluster.primary.name
}
output "cluster_location" {
  value = google_container_cluster.primary.location
}
output "cluster_project" {
  value = google_container_cluster.primary.project
}

AWS リソースの定義

GCP と同様に、AWS リソースの定義を modules/aws 配下に作成します。
VPN の設定内容は公式ドキュメントを参考にしました。

modules/aws/main.tf

自身のアカウントIDを取得するために aws_caller_identity を定義しています。

modules/aws/main.tf

data "aws_caller_identity" "current" {}

# Most recent Amazon Linux 2 AMI
data "aws_ami" "amazon_linux_2" {
  most_recent = true
  owners      = ["amazon"]
  filter {
    name   = "name"
    values = ["amzn2-ami-hvm*"]
  }
  filter {
    name   = "architecture"
    values = ["x86_64"]
  }
}

modules/aws/vpc.tf

各種ゲートウェイのフラグを true にして自動作成していますが、別リソースとして定義する事も可能です。
また、amazon_side_asn は仮想プライベートゲートウェイに設定するASNの値となります。

modules/aws/vpc.tf

# EKS クラスタ用 VPC
module "eks_vpc" {
  source = "terraform-aws-modules/vpc/aws"
  name   = "${var.common.prefix}-${var.common.env}-eks-vpc"
  cidr   = var.aws_vpc.vpc_cidr
  azs                  = var.aws_vpc.subnet_availability_zones
  public_subnets       = var.aws_vpc.public_subnet_cidr
  private_subnets      = var.aws_vpc.private_subnet_cidr
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
  enable_dns_support   = true
  enable_vpn_gateway   = true
  amazon_side_asn      = var.aws_bgp_asn

  tags = {
    "kubernetes.io/cluster/${var.common.prefix}-${var.common.env}-cluster" = "shared"
  }
  public_subnet_tags = {
    "kubernetes.io/cluster/${var.common.prefix}-${var.common.env}-cluster" = "shared"
    "kubernetes.io/role/elb" = 1
  }
  private_subnet_tags = {
    "kubernetes.io/cluster/${var.common.prefix}-${var.common.env}-cluster" = "shared"
    "kubernetes.io/role/internal-elb" = 1
  }
}

resource "aws_security_group" "default_sg" {
  vpc_id = module.eks_vpc.vpc_id
  name   = "${var.common.prefix}-${var.common.env}-allow-all-from-gcp"
  ingress {
    description = "All access from GCP"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = [
      var.gcp_vpc.subnet_cidr,
      var.gcp_vpc.secondary_ip_service_ranges,
      var.gcp_vpc.secondary_ip_pod_ranges
    ]
  }
  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
}

modules/aws/eks.tf

限定公開のEKSクラスタを作成します。
クラスタにアクセス可能なIAMユーザーがTerraform実行時のユーザーのみのため、あとから編集できるように configmap も一緒に作成しています。

modules/aws/eks.tf

＃EKSクラスタのIAMロール
resource "aws_iam_role" "cluster" {
  name = "${var.common.prefix}-${var.common.env}-eks-cluster"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "cluster-AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = "${aws_iam_role.cluster.name}"
}

resource "aws_iam_role_policy_attachment" "cluster-AmazonEKSServicePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = "${aws_iam_role.cluster.name}"
}

module "eks" {
  source           = "terraform-aws-modules/eks/aws"
  version          = "~> 19.19.1"
  cluster_version  = var.aws_eks.cluster_version
  cluster_name     = "${var.common.prefix}-${var.common.env}-cluster"
  vpc_id           = module.eks_vpc.vpc_id
  subnet_ids       = module.eks_vpc.private_subnets
  enable_irsa      = true
  iam_role_arn     = aws_iam_role.cluster.arn
  create_iam_role  = false
  aws_auth_accounts = [data.aws_caller_identity.current.account_id]

  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = false
  cluster_enabled_log_types       = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
  eks_managed_node_groups = {
    "${var.common.prefix}-${var.common.env}-ng1" = {
      desired_size     = 2
      max_capacity     = 3
      min_capacity     = 1
      instance_types   = [var.aws_eks.instance_types]
    }
  }
  cluster_additional_security_group_ids = [
    aws_security_group.eks_sg.id
  ]
  node_security_group_additional_rules = {
    admission_webhook = {
      description = "Admission Webhook"
      protocol    = "tcp"
      from_port   = 0
      to_port     = 65535
      type        = "ingress"
      source_cluster_security_group = true
    }
    ingress_node_communications = {
      description = "Ingress Node to node"
      protocol    = "tcp"
      from_port   = 0
      to_port     = 65535
      type        = "ingress"
      self        = true
    }
    egress_node_communications = {
      description = "Egress Node to node"
      protocol    = "tcp"
      from_port   = 0
      to_port     = 65535
      type        = "egress"
      self        = true
    }
  }
}

# Create a security group
resource "aws_security_group" "eks_sg" {
  name   = "${var.common.prefix}-${var.common.env}-eks-sg"
  vpc_id = module.eks_vpc.vpc_id
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_security_group_rule" "eks_sg_ingress_rule_https" {
  security_group_id = aws_security_group.eks_sg.id
  type              = "ingress"
  protocol          = "tcp"
  from_port         = 443
  to_port           = 443
  cidr_blocks       = ["0.0.0.0/0"]
  ipv6_cidr_blocks  = ["::/0"]
}

resource "aws_security_group_rule" "eks_sg_ingress_rule_from_gcp" {
  security_group_id = aws_security_group.eks_sg.id
  type              = "ingress"
  protocol          = "-1"
  from_port         = 0
  to_port           = 0
  cidr_blocks       = [
    var.gcp_vpc.subnet_cidr,
    var.gcp_vpc.secondary_ip_service_ranges,
    var.gcp_vpc.secondary_ip_pod_ranges
  ]
}

# IAMユーザー追加用の configmap をローカルに出力しておく
resource "null_resource" "get_auth_configmap" {
  triggers = {
    always_run = "${timestamp()}"
  }
  provisioner "local-exec" {
    command = <<-EOF
echo "${module.eks.aws_auth_configmap_yaml}
  mapUsers: |
    - userarn: ${data.aws_iam_user.tmatsuno.arn}
      username: ${data.aws_iam_user.tmatsuno.user_name}
      groups:
        - system:masters" > aws_auth_configmap.yaml
EOF
  }
  depends_on = [module.eks]
}

modules/aws/multicloud_vpn.tf

AWSとGCPのVPN用リソースをまとめて作成します。
VPN 用リソースには一部 GCP のリソースが含まれますが、わけて管理するとややこしくなるので、ここに入れ込んでしまいました。

modules/aws/multicloud_vpn.tf

# GCP HA VPN Gateway
resource "google_compute_ha_vpn_gateway" "gcp_aws_gateway" {
  name    = "${var.common.prefix}-${var.common.env}-vpn-gateway"
  network = var.gcp_vpc_nw_id
  region  = var.gcp_common.region
  vpn_interfaces {
    id = 0
  }
  vpn_interfaces {
    id = 1
  }
  depends_on = [ var.gcp_vpc_nw_id ]
}

# Google Compute Router
resource "google_compute_router" "gcp_vpn_router" {
  name    = "${var.common.prefix}-${var.common.env}-gcp-vpn-router"
  network = var.gcp_vpc_nw_id
  region  = var.gcp_common.region

  bgp {
    asn               = var.vpn_common.gcp_bgp_asn
    advertise_mode    = "CUSTOM"
    advertised_groups = [ "ALL_SUBNETS" ]
  }
  depends_on = [ var.gcp_vpc_nw_id ]
}

# Configuring route propagation for AWS Virtual Private Gateway
resource "aws_vpn_gateway_route_propagation" "aws_vgw_rp_public" {
  vpn_gateway_id = module.eks_vpc.vgw_id
  route_table_id = module.eks_vpc.public_route_table_ids[0]
}

resource "aws_vpn_gateway_route_propagation" "aws_vgw_rp_private" {
  vpn_gateway_id = module.eks_vpc.vgw_id
  route_table_id = module.eks_vpc.private_route_table_ids[0]
}

# AWS Customer Gateway with Public IP of GCP Cloud Gateway
resource "aws_customer_gateway" "google1" {
  bgp_asn    = var.vpn_common.gcp_bgp_asn
  ip_address = google_compute_ha_vpn_gateway.gcp_aws_gateway.vpn_interfaces[0].ip_address
  type       = "ipsec.1"
}

resource "aws_customer_gateway" "google2" {
  bgp_asn    = var.vpn_common.gcp_bgp_asn
  ip_address = google_compute_ha_vpn_gateway.gcp_aws_gateway.vpn_interfaces[1].ip_address
  type       = "ipsec.1"
}

# AWS VPN Tunnel
resource "aws_vpn_connection" "aws_tunnel1" {
  vpn_gateway_id      = module.eks_vpc.vgw_id
  customer_gateway_id = aws_customer_gateway.google1.id
  type                = "ipsec.1"
  static_routes_only  = false
  depends_on          = [ module.eks_vpc, aws_customer_gateway.google1 ]
}

resource "aws_vpn_connection" "aws_tunnel2" {
  vpn_gateway_id      = module.eks_vpc.vgw_id
  customer_gateway_id = aws_customer_gateway.google2.id
  type                = "ipsec.1"
  static_routes_only  = false
  depends_on          = [ module.eks_vpc, aws_customer_gateway.google2 ]
}

# GCP Peer VPN Gateway
resource "google_compute_external_vpn_gateway" "peer_gateway" {
  name    = "${var.common.prefix}-${var.common.env}-peer-gateway"
  redundancy_type = "FOUR_IPS_REDUNDANCY"
  interface {
    id = 0
    ip_address = aws_vpn_connection.aws_tunnel1.tunnel1_address
  }
  interface {
    id = 1
    ip_address = aws_vpn_connection.aws_tunnel1.tunnel2_address
  }
  interface {
    id = 2
    ip_address = aws_vpn_connection.aws_tunnel2.tunnel1_address
  }
  interface {
    id = 3
    ip_address = aws_vpn_connection.aws_tunnel2.tunnel2_address
  }
}

# GCP VPN tunnel
resource "google_compute_vpn_tunnel" "tunnel1" {
  name          = "${var.common.prefix}-${var.common.env}-gcp-aws-vpn-tunnel-1"
  region        = var.gcp_common.region
  shared_secret = aws_vpn_connection.aws_tunnel1.tunnel1_preshared_key
  ike_version   = 1
  router        = google_compute_router.gcp_vpn_router.id
  vpn_gateway   = google_compute_ha_vpn_gateway.gcp_aws_gateway.id
  vpn_gateway_interface = 0
  peer_external_gateway = google_compute_external_vpn_gateway.peer_gateway.id
  peer_external_gateway_interface = 0
  depends_on = [
    google_compute_ha_vpn_gateway.gcp_aws_gateway,
    google_compute_external_vpn_gateway.peer_gateway,
    aws_vpn_connection.aws_tunnel1
  ]
}

resource "google_compute_vpn_tunnel" "tunnel2" {
  name          = "${var.common.prefix}-${var.common.env}-gcp-aws-vpn-tunnel-2"
  region        = var.gcp_common.region
  shared_secret = aws_vpn_connection.aws_tunnel1.tunnel2_preshared_key
  ike_version   = 1
  router        = google_compute_router.gcp_vpn_router.id
  vpn_gateway   = google_compute_ha_vpn_gateway.gcp_aws_gateway.id
  vpn_gateway_interface = 0
  peer_external_gateway = google_compute_external_vpn_gateway.peer_gateway.id
  peer_external_gateway_interface = 1
  depends_on = [
    google_compute_ha_vpn_gateway.gcp_aws_gateway,
    google_compute_external_vpn_gateway.peer_gateway,
    aws_vpn_connection.aws_tunnel1
  ]
}

resource "google_compute_vpn_tunnel" "tunnel3" {
  name          = "${var.common.prefix}-${var.common.env}-gcp-aws-vpn-tunnel-3"
  region        = var.gcp_common.region
  shared_secret = aws_vpn_connection.aws_tunnel2.tunnel1_preshared_key
  ike_version   = 1
  router        = google_compute_router.gcp_vpn_router.id
  vpn_gateway   = google_compute_ha_vpn_gateway.gcp_aws_gateway.id
  vpn_gateway_interface = 1
  peer_external_gateway = google_compute_external_vpn_gateway.peer_gateway.id
  peer_external_gateway_interface = 2
  depends_on = [
    google_compute_ha_vpn_gateway.gcp_aws_gateway,
    google_compute_external_vpn_gateway.peer_gateway,
    aws_vpn_connection.aws_tunnel2
  ]
}

resource "google_compute_vpn_tunnel" "tunnel4" {
  name          = "${var.common.prefix}-${var.common.env}-gcp-aws-vpn-tunnel-4"
  region        = var.gcp_common.region
  shared_secret = aws_vpn_connection.aws_tunnel2.tunnel2_preshared_key
  ike_version   = 1
  router        = google_compute_router.gcp_vpn_router.id
  vpn_gateway   = google_compute_ha_vpn_gateway.gcp_aws_gateway.id
  vpn_gateway_interface = 1
  peer_external_gateway = google_compute_external_vpn_gateway.peer_gateway.id
  peer_external_gateway_interface = 3
  depends_on = [
    google_compute_ha_vpn_gateway.gcp_aws_gateway,
    google_compute_external_vpn_gateway.peer_gateway,
    aws_vpn_connection.aws_tunnel2
  ]
}

# GCP Router for tunnel 1
resource "google_compute_router_peer" "tunnel1_bgp1" {
  name            = "${var.common.prefix}-${var.common.env}-vpn-tunnel1-bgp1"
  region          = var.gcp_common.region
  router          = google_compute_router.gcp_vpn_router.name
  peer_ip_address = aws_vpn_connection.aws_tunnel1.tunnel1_vgw_inside_address
  ip_address      = aws_vpn_connection.aws_tunnel1.tunnel1_cgw_inside_address
  peer_asn        = var.vpn_common.aws_bgp_asn
  interface       = google_compute_router_interface.router_interface1.name
}

resource "google_compute_router_peer" "tunnel1_bgp2" {
  name            = "${var.common.prefix}-${var.common.env}-vpn-tunnel1-bgp2"
  region          = var.gcp_common.region
  router          = google_compute_router.gcp_vpn_router.name
  peer_ip_address = aws_vpn_connection.aws_tunnel1.tunnel2_vgw_inside_address
  ip_address      = aws_vpn_connection.aws_tunnel1.tunnel2_cgw_inside_address
  peer_asn        = var.vpn_common.aws_bgp_asn
  interface       = google_compute_router_interface.router_interface2.name
}

# GCP Router for tunnel 2
resource "google_compute_router_peer" "tunnel2_bgp1" {
  name            = "${var.common.prefix}-${var.common.env}-vpn-tunnel2-bgp1"
  region          = var.gcp_common.region
  router          = google_compute_router.gcp_vpn_router.name
  peer_ip_address = aws_vpn_connection.aws_tunnel2.tunnel1_vgw_inside_address
  ip_address      = aws_vpn_connection.aws_tunnel2.tunnel1_cgw_inside_address
  peer_asn        = var.vpn_common.aws_bgp_asn
  interface       = google_compute_router_interface.router_interface3.name
}

resource "google_compute_router_peer" "tunnel2_bgp2" {
  name            = "${var.common.prefix}-${var.common.env}-vpn-tunnel2-bgp2"
  region          = var.gcp_common.region
  router          = google_compute_router.gcp_vpn_router.name
  peer_ip_address = aws_vpn_connection.aws_tunnel2.tunnel2_vgw_inside_address
  ip_address      = aws_vpn_connection.aws_tunnel2.tunnel2_cgw_inside_address
  peer_asn        = var.vpn_common.aws_bgp_asn
  interface       = google_compute_router_interface.router_interface4.name
}

# Computer Router Interface for tunnel 1
resource "google_compute_router_interface" "router_interface1" {
  name       = "${var.common.prefix}-${var.common.env}-vpn-router-interface1"
  region     = var.gcp_common.region
  router     = google_compute_router.gcp_vpn_router.name
  ip_range   = "${aws_vpn_connection.aws_tunnel1.tunnel1_cgw_inside_address}/30"
  vpn_tunnel = google_compute_vpn_tunnel.tunnel1.name
}

resource "google_compute_router_interface" "router_interface2" {
  name       = "${var.common.prefix}-${var.common.env}-vpn-router-interface2"
  region     = var.gcp_common.region
  router     = google_compute_router.gcp_vpn_router.name
  ip_range   = "${aws_vpn_connection.aws_tunnel1.tunnel2_cgw_inside_address}/30"
  vpn_tunnel = google_compute_vpn_tunnel.tunnel1.name
}

# Computer Router Interface for tunnel 2
resource "google_compute_router_interface" "router_interface3" {
  name       = "${var.common.prefix}-${var.common.env}-vpn-router-interface3"
  region     = var.gcp_common.region
  router     = google_compute_router.gcp_vpn_router.name
  ip_range   = "${aws_vpn_connection.aws_tunnel2.tunnel1_cgw_inside_address}/30"
  vpn_tunnel = google_compute_vpn_tunnel.tunnel2.name
}

resource "google_compute_router_interface" "router_interface4" {
  name       = "${var.common.prefix}-${var.common.env}-vpn-router-interface4"
  region     = var.gcp_common.region
  router     = google_compute_router.gcp_vpn_router.name
  ip_range   = "${aws_vpn_connection.aws_tunnel2.tunnel2_cgw_inside_address}/30"
  vpn_tunnel = google_compute_vpn_tunnel.tunnel2.name
}

modules/aws/loadbalancer.tf

AWS Load Balancer Controllerを Helm でインストールできるようにするため、必要になるリソースを事前に作成しておきます。

modules/aws/loadbalancer.tf

data "http" "albc_policy_json" {
  url = "https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.5.4/docs/install/iam_policy.json"
}

resource "aws_iam_policy" "albc" {
  name   = "${var.common.prefix}-${var.common.env}-aws-loadbalancer-controller-iam-policy"
  policy = data.http.albc_policy_json.response_body
}

module "albc_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
  version = "5.33.0"
  create_role                   = true
  role_name                     = "${var.common.prefix}-${var.common.env}-aws-load-balancer-controller-role"
  role_policy_arns              = [aws_iam_policy.albc.arn]
  provider_url                  = module.eks.cluster_oidc_issuer_url
  oidc_fully_qualified_subjects = ["system:serviceaccount:kube-system:aws-load-balancer-controller"]
}

# AWS load balancer cotroller 用 ServiceAccount
resource "null_resource" "get_alb_service_account" {
  triggers = {
    always_run = "${timestamp()}"
  }

  provisioner "local-exec" {
    command = <<-EOF
echo "apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: aws-load-balancer-controller
  name: aws-load-balancer-controller
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: ${module.albc_irsa.iam_role_arn}" > aws_alb_sa.yaml
EOF
  }
  depends_on = [module.eks, module.albc_irsa]
}

modules/aws/variables.tf

GCP側と同様に variables.tf を作成します。

modules/aws/variables.tf

variable "common" {
  type = object ({
    prefix  = string
    env     = string
  })
  description = "リソース共通の設定値"
}

variable "aws_common" {
  type = object ({
    region  = string
  })
  description = "AWS リソース共通の設定値"
}

variable "aws_vpc" {
  type = object ({
    vpc_cidr = string
    subnet_availability_zones = list(string)
    public_subnet_cidr = list(string)
    private_subnet_cidr = list(string)
  })
  description = "AWS EKS VPC の設定値"
}

variable "aws_eks" {
  type = object ({
    instance_types  = string
    cluster_version = string
  })
  description = "AWS EKS の設定値"
}

variable "gcp_common" {
  type = object ({
    project = string
    region  = string
    zone    = string
    email   = string  # GCP ServiceAccount のメールアドレス
  })
  description = "GCPリソース共通の設定値"
}

variable "gcp_vpc" {
  type = object ({
    subnet_cidr = string
    secondary_ip_service_ranges = string
    secondary_ip_pod_ranges = string
  })
  description = "GCP VPC の設定値"
}

variable "vpn_common" {
  type = object ({
    gcp_bgp_asn = number
    aws_bgp_asn = number
  })
  description = "VPN の設定値"
}

variable "gcp_vpc_nw_id" {
  type = string
  description = "GCP VPC のID"
}

modules/aws/outputs.tf

EKS の kubeconfig を取得する際に必要となるパラメータを定義しています。

modules/aws/outputs.tf

output "cluster_name" {
  value = try(module.eks.cluster_name, "")
}

output "vpc_id" {
  value = try(module.eks_vpc.vpc_id, "")
}

環境毎のモジュール定義

先ほど作成した各クラウドリソースを利用する側のモジュールを作成します。
terraform.tfvars を環境毎に用意することで、部品を再利用しつつパラメータを変更することが可能となります。

environments/test/main.tf

各モジュールに後述の terraform.tfvars で定義した変数を引き渡します。

environments/test/main.tf

module "gcp_modules" {
  source      = "../../modules/gcp"
  common      = var.common
  gcp_common  = var.gcp_common
  gcp_vpc     = var.gcp_vpc
  gcp_gke     = var.gcp_gke
  gcp_bastion = var.gcp_bastion
  aws_vpc     = var.aws_vpc
}

module "aws_modules" {
  source     = "../../modules/aws"
  common     = var.common
  aws_common = var.aws_common
  aws_vpc    = var.aws_vpc
  aws_eks    = var.aws_eks

  gcp_vpc_nw_id = module.gcp_modules.vpc_network_id
  gcp_common    = var.gcp_common
  gcp_vpc       = var.gcp_vpc

  vpn_common    = var.vpn_common
}

environments/test/terraform.tfvars

環境毎のパラメータを定義します。

environments/test/terraform.tfvars

common = {
	prefix  = "tmatsuno"
	env     = "test"
}

# ----- 以下、GCP設定値 -----
gcp_common = {
	project = "<GCP Project Name>"
	region  = "asia-northeast1"
	zone    = "asia-northeast1-c"
	email   = "<GCP Account Name>@<GCP Project Name>.iam.gserviceaccount.com"
}

gcp_vpc = {
	subnet_cidr                 = "172.16.0.0/15"
	secondary_ip_service_ranges = "172.18.0.0/18"
	secondary_ip_pod_ranges     = "172.19.0.0/18"
}

gcp_gke = {
	master_cidr     = "192.168.16.0/28"
	preemptible     = true
	machine_type    = "e2-medium"
	cluster_version = "1.27.4-gke.900"
}

gcp_bastion = {
	machine_type    = "e2-small"
	ssh_sourcerange = "35.235.240.0/20"
}

# ----- 以下、AWS設定値 -----
aws_common = {
	region  = "ap-northeast-1"
}

aws_vpc = {
	vpc_cidr = "172.20.0.0/16"
	subnet_availability_zones = [
		"ap-northeast-1a",
		"ap-northeast-1c"
	]
	public_subnet_cidr = [
		"172.20.0.0/21",
		"172.20.8.0/21"
	]
	private_subnet_cidr = [
		"172.20.16.0/21",
		"172.20.24.0/21"
	]
}

aws_eks = {
	instance_types  = "t3.medium"
	cluster_version = "1.28"
}

# ----- 以下、VPN設定値 -----
vpn_common = {
	gcp_bgp_asn = 64513
	aws_bgp_asn = 65001
}

environments/test/variables.tf

terraform.tfvars の変数と型を一致させる必要があります。

environments/test/variables.tf

variable "common" {
  type = object ({
    prefix  = string  # 固有のプレフィクス (任意の文字列)
    env     = string  # 環境名 ( dev、prod など)
  })
  description = "リソース共通の設定値"
}

#----- GCP parameters -----
variable "gcp_common" {
  type = object ({
    project = string
    region  = string
    zone    = string
    email   = string  # GCP ServiceAccount のメールアドレス
  })
  description = "GCPリソース共通の設定値"
}

variable "gcp_vpc" {
  type = object ({
    subnet_cidr = string
    secondary_ip_service_ranges = string
    secondary_ip_pod_ranges     = string
    proxy_only_subnet_cidr      = string
  })
  description = "GCP VPC の設定値"
}

variable "gcp_gke" {
  type = object ({
    master_cidr     = string
    preemptible     = bool
    machine_type    = string
    cluster_version = string
  })
  description = "GKE の設定値"
}

variable "gcp_bastion" {
  type = object ({
    machine_type    = string
    ssh_sourcerange = string
  })
  description = "GCP 踏み台 VM の設定値"
}

#----- AWS parameters -----
variable "aws_common" {
  type = object ({
    region  = string
  })
  description = "AWS リソース共通の設定値"
}

variable "aws_vpc" {
  type = object ({
    vpc_cidr = string
    subnet_availability_zones = list(string)
    public_subnet_cidr = list(string)
    private_subnet_cidr = list(string)
  })
  description = "AWS EKS用 VPC の設定値"
}

variable "aws_eks" {
  type = object ({
    instance_types  = string
    cluster_version = string
  })
  description = "AWS EKS の設定値"
}

variable "vpn_common" {
  type = object ({
    gcp_bgp_asn = number
    aws_bgp_asn = number
  })
  description = "VPN の設定値"
}

environments/test/backend.tf

先に作成した GCS バケットを指定することで、tfstate がオブジェクトストレージに保存されます。
複数ユーザーから Terraform が実行される場合は、同時に更新してしまわないように運用フローを考慮する必要があります。

environments/test/backend.tf

terraform {
  backend "gcs" {
    bucket = "tmatsuno-gke-test-tfstate"
    prefix = "tmatsuno-gke-test"
  }
}

environments/test/provider.tf

利用するプロバイダの初期設定を定義します。

environments/test/provider.tf

provider "google" {
  project = var.gcp_common.project
  region  = var.gcp_common.region
  zone    = var.gcp_common.zone
}

provider "google-beta" {
  project     = var.gcp_common.project
  region      = var.gcp_common.region
  zone        = var.gcp_common.zone
}

provider "aws" {
  region = var.aws_common.region
  default_tags {
    tags = {
      Environment = "${var.common.prefix}-${var.common.env}"
    }
  }
}

environments/test/versions.tf

各プロバイダのバージョンは versions.tf にまとめて定義します。

environments/test/versions.tf

terraform {
  required_version = "~> 1.6.0"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.4"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 5.4"
    }
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.24"
    }
    http = {
      source  = "hashicorp/http"
      version = "~> 3.4"
    }
  }
}

environments/test/outputs.tf

リソースの作成が完了した際に標準出力に表示させたい内容を outputs.tf に定義します。

environments/test/outputs.tf

# GKE クラスタの kubeconfig を取得
output "command_to_connect_cluster" {
  value = "gcloud container clusters get-credentials ${module.gcp_modules.cluster_name} --region ${module.gcp_modules.cluster_location} --project ${module.gcp_modules.cluster_project}"
}

# EKS クラスタの kubeconfig を取得
output "eks_generage_config" {
  value = "aws eks update-kubeconfig --region ${var.aws_common.region} --name ${module.aws_modules.cluster_name}"
}

# AWS load balancer controller のインストールコマンド
output "eks_install_loadbalancer_cmd" {
  value = <<-EOT

        helm repo add eks https://aws.github.io/eks-charts
        helm repo update eks
        helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
          -n kube-system \
          --set clusterName=${module.aws_modules.cluster_name} \
          --set serviceAccount.create=false \
          --set serviceAccount.name=aws-load-balancer-controller 
    EOT
}

Terraform の実行

コンテナにログインした状態で、Terraform を実行します。

$ cd /work/environments/test

$ terraform init
$ terraform plan
$ terraform apply -auto-approve

正常終了すると outputs.tf に定義した内容が表示されます。


...

Outputs:

command_to_connect_cluster = "gcloud container clusters get-credentials tmatsuno-test-cluster --region asia-northeast1 --project <GCP Project Name>"
eks_generage_config = "aws eks update-kubeconfig --region ap-northeast-1 --name tmatsuno-test-cluster"
eks_install_loadbalancer_cmd = <<EOT

helm repo add eks https://aws.github.io/eks-charts
helm repo update eks
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=tmatsuno-test-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

EOT

クラスタにアクセス

GCP の踏み台サーバから k8s クラスタにアクセスします。
GCP コンソールから VM インスタンス一覧の「SSH」ボタンを押下してください。

SSHコンソールがポップアップで表示されるので、コマンドを実行します。
最初に GCP の認証情報を設定します。

$ gcloud auth login

You are running on a Google Compute Engine virtual machine.
It is recommended that you use service accounts for authentication.

You can run:

  $ gcloud config set account `ACCOUNT`

to switch accounts if necessary.

Your credentials may be visible to others with access to this
virtual machine. Are you sure you want to authenticate with
your personal account?

Do you want to continue (Y/n)?

「Y」を押下すると、URLが表示されるのでコピペしてブラウザアクセスしてください。
トークンを取得したら、SSHコンソールに戻ってトークンを貼り付けます。
以下のようなメッセージが表示されたらOKです。

You are now logged in as [takahiro.matsuno@systemi.co.jp].
Your current project is [xxxxxx].  You can change this setting by running:
  $ gcloud config set project PROJECT_ID

次に AWS の認証情報を設定します。
AWS CLI は初期化スクリプトでインストール済みなので、aws configure を実行します。

$ aws configure
AWS Access Key ID [None]: xxxxx
AWS Secret Access Key [None]: xxxxx
Default region name [None]: ap-northeast-1
Default output format [None]: json

認証情報を設定したら kubeconfig を取得します。

# GKE
$ gcloud container clusters get-credentials tmatsuno-test-cluster --region asia-northeast1 --project <GCP Project Name>
Fetching cluster endpoint and auth data.
kubeconfig entry generated for tmatsuno-test-cluster

# EKS
$ aws eks update-kubeconfig --region ap-northeast-1 --name tmatsuno-test-cluster
Updated context arn:aws:eks:ap-northeast-1:xxxxxx:cluster/tmatsuno-test-cluster in /home/takahiro_matsuno_systemi_co_jp/.kube/config

context が AWS EKS に向いていることを確認したら、AWS load balancer controller をインストールします。

# CURRENT 列にアスタリスクが付与されている行が現在の context となる
$ kubectl config get-contexts
CURRENT   NAME                                                                    CLUSTER                                                                 AUTHINFO                                                                NAMESPACE
*         arn:aws:eks:ap-northeast-1:xxxxxx:cluster/tmatsuno-test-cluster   arn:aws:eks:ap-northeast-1:xxxxxx:cluster/tmatsuno-test-cluster   arn:aws:eks:ap-northeast-1:xxxxxx:cluster/tmatsuno-test-cluster   
          gke_xxxxxxx_asia-northeast1_tmatsuno-test-cluster   gke_xxxxxx_asia-northeast1_tmatsuno-test-cluster   gke_xxxxxx_asia-northeast1_tmatsuno-test-cluster  

# AWS が提供する Helm Chart をインストールします
$ helm repo add eks https://aws.github.io/eks-charts
$ helm repo update eks
$ helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=tmatsuno-test-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

Terraform 実行時のカレントディレクトリにマニフェストファイルが出力されているので、こちらもデプロイします。

#どちらも AWS EKS にデプロイするリソースです
$ kubectl apply -f aws_alb_sa.yaml
$ kubectl apply -f aws_auth_configmap.yaml

クラスタ間通信の疎通確認

EKS にnginx、GKE にhttpdのコンテナを起動します。

$ export AWS_CTX="arn:aws:eks:ap-northeast-1:xxxxxx:cluster/tmatsuno-test-cluster"
$ export GCP_CTX="gke_xxxxxx_asia-northeast1_tmatsuno-test-cluster"

$ kubectl apply -f nginx.yaml --context=${AWS_CTX}
$ kubectl apply -f httpd.yaml --context=${GCP_CTX}

マニフェストの内容は以下の通りです。

nginx.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: nginx

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: nginx
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
    - name: http
      port: 80
      targetPort: 80

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: nginx
  name: nginx
  annotations:
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: nginx
              port:
                number: 80

httpd.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: httpd

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd
  namespace: httpd
spec:
  selector:
    matchLabels:
      app: httpd
  replicas: 1
  template:
    metadata:
      labels:
        app: httpd
    spec:
      containers:
      - name: httpd
        image: httpd:latest
        ports:
        - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: httpd
  namespace: httpd
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
spec:
  type: LoadBalancer
  selector:
    app: httpd
  ports:
    - name: http
      port: 80
      targetPort: 80

しばらくすると L4LB のデプロイが完了します。
内部ネットワークに公開されるエンドポイントがEXTERNAL-IP 列に表示されるので、手元に控えておきます。

$ kubectl get ing -n nginx --context=${AWS_CTX}
NAME    CLASS   HOSTS   ADDRESS                                                                           PORTS   AGE
nginx   alb     *       internal-k8s-nginx-nginx-029f60f89d-1145409129.ap-northeast-1.elb.amazonaws.com   80      53s

$ kubectl get svc -n httpd --context=${GCP_CTX}
NAME    TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
httpd   LoadBalancer   172.18.48.138   172.16.0.17   80:32435/TCP   4m39s

curl を実行するコンテナを起動して、クラスタを跨いだ通信が可能か確認します。

EKS 上の curl コンテナから GKE の httpd にアクセス

$ kubectl run -it --rm curl --image=curlimages/curl --context="${AWS_CTX}" -- /bin/sh
If you don\'t see a command prompt, try pressing enter.
~ $ curl -v http://172.16.0.17:80
*   Trying 172.16.0.17:80...
* Connected to 172.16.0.17 (172.16.0.17) port 80
> GET / HTTP/1.1
> Host: 172.16.0.17
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Sun, 24 Dec 2023 13:56:37 GMT
< Server: Apache/2.4.58 (Unix)
< Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
< ETag: "2d-432a5e4a73a80"
< Accept-Ranges: bytes
< Content-Length: 45
< Content-Type: text/html
< 
<html><body><h1>It works!</h1></body></html>
* Connection #0 to host 172.16.0.17 left intact

GKE 上の curl コンテナから EKS の nginx にアクセス

$ kubectl run -it --rm curl --image=curlimages/curl --context="${GCP_CTX}" -- /bin/sh
If you don\'t see a command prompt, try pressing enter.

~ $ curl -v http://internal-k8s-nginx-nginx-029f60f89d-1145409129.ap-northeast-1.elb.amazonaws.com:80
* Host internal-k8s-nginx-nginx-029f60f89d-1145409129.ap-northeast-1.elb.amazonaws.com:80 was resolved.
* IPv6: (none)
* IPv4: 172.20.23.80, 172.20.26.202
*   Trying 172.20.23.80:80...
* Connected to internal-k8s-nginx-nginx-029f60f89d-1145409129.ap-northeast-1.elb.amazonaws.com (172.20.23.80) port 80
> GET / HTTP/1.1
> Host: internal-k8s-nginx-nginx-029f60f89d-1145409129.ap-northeast-1.elb.amazonaws.com
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Sun, 24 Dec 2023 13:57:17 GMT
< Content-Type: text/html
< Content-Length: 615
< Connection: keep-alive
< Server: nginx/1.25.3
< Last-Modified: Tue, 24 Oct 2023 13:46:47 GMT
< ETag: "6537cac7-267"
< Accept-Ranges: bytes
< 
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Connection #0 to host internal-k8s-nginx-nginx-029f60f89d-1145409129.ap-northeast-1.elb.amazonaws.com left intact

簡単ですが、クラスタを跨いだ通信ができることが確認できました。

補足

GCP でマルチクラスタを構成する場合はGKE EnterpriseやAnthosという選択肢もあります。

まとめ

クラウド上の k8s クラスタをVPNで接続して、Internal Load Balancer 経由でクラスタを跨いだ通信を検証してみました。
本当はIstioでマルチクラスタ間通信を試してみたかったのですが、時間が取れず断念。
またの機会に検証してみたいと思います。