More than 1 year has passed since last update.

ROSAクラスタの構築（自前のVPC内、且つ、STSモードで、Private Linkを添えて）

Last updated at 2021-10-20Posted at 2021-08-23

ROSAのカスタマイズ

ROSAは愚直にインストール手順を実行すると、以下のような仕様となる。

インストーラが作ったVPC内にクラスタが作られる
Red Hat SRE Team用のIAM Userが勝手に作られる
Red Hat SRE Teamからのアクセスはインターネット経由になる

ところが、色々アップデートが入り、以下のようなアレンジが効くようになった模様。

自前のVPCにクラスタを作れる
IAM Userは不要で、AssumeRoleでSTSから一時トークンもらってSRE活動をしてくれる（たぶん）
Red Hat SRE TeamからのアクセスはPrivate Link経由になる

とまぁ、規定のLanding Zoneでしか活動が認められていない民にとっては、尊い機能がリリースされたんですって。
ところが、インストール手順をなぞってみてもうまく行かなくて死んだので、~~ビアを呷りながら~~きれいな心でリトライするんだ！(^◯^)

事前準備

以下のセットアップが完了しているものとする。

AWS Account
- SCP管理下
AWS CLI
- Administration
ROSA CLI
- アクセストークンがセット済み
- rosa initが未実行（実行するとosdCcsAdminというIAM Userが作られてしまうため）

環境変数の設定

クラスターのセットアップに使う情報を環境変数に埋める。

export VERSION=4.8.5 \
        ROSA_CLUSTER_NAME=rosacluster \
        AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text` \
        REGION=ap-northeast-1 \
        AWS_PAGER=""

自前のVPCを準備する

VPCの作成

VPCを作って、

VPC_ID=`aws ec2 create-vpc --cidr-block 10.0.0.0/16 | jq -r .Vpc.VpcId`
echo $VPC_ID
vpc-07XXXXXXXXXXXX2e

クラスタの名前でタグ付けする。

aws ec2 create-tags --resources $VPC_ID --tags Key=Name,Value=$ROSA_CLUSTER_NAME

さらにVPCのオプションを一部修正（--enable-dns-hostnames）

aws ec2 modify-vpc-attribute --vpc-id $VPC_ID --enable-dns-hostnames

Subnetの作成

Public Subnetを作って、

PUBLIC_SUBNET=`aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.128.0/17 | jq -r .Subnet.SubnetId`
echo $PUBLIC_SUBNET
subnet-06XXXXXXXXXXXX4b

クラスタの名前でタグ付けする。

aws ec2 create-tags --resources $PUBLIC_SUBNET --tags Key=Name,Value=$ROSA_CLUSTER_NAME-public

Private Subnetを作って、

PRIVATE_SUBNET=`aws ec2 create-subnet --vpc-id $VPC_ID  --cidr-block 10.0.0.0/17 | jq -r .Subnet.SubnetId`
echo $PRIVATE_SUBNET
subnet-07XXXXXXXXXXXXdf

クラスタの名前でタグ付けする。

aws ec2 create-tags --resources $PRIVATE_SUBNET --tags Key=Name,Value=$ROSA_CLUSTER_NAME-private

Internet Gatewayの作成

アウトバウンドはやっぱりInternet GatewayとNAT Gatewayが要るみたい。
この辺の勘違いで前回はうまく行かなかったのかな。
確かに、PrivateLink architecture model上はPublic Subnetは図示されていないのだけど、以下の部分は図示しないけど、アウトバウンドは必要ですぞ＾＾に見えてきた。

I_GW=`aws ec2 create-internet-gateway | jq -r .InternetGateway.InternetGatewayId`
echo $I_GW
igw-02XXXXXXXXXXXXa9

VPCにアタッチして、

aws ec2 attach-internet-gateway --vpc-id $VPC_ID --internet-gateway-id $I_GW

クラスタの名前でタグ付けする。

aws ec2 create-tags --resources $I_GW --tags Key=Name,Value=$ROSA_CLUSTER_NAME

Route Tableの作成（Public Subnet用）

Route Tableを作って、

R_TABLE=`aws ec2 create-route-table --vpc-id $VPC_ID | jq -r .RouteTable.RouteTableId`
echo $R_TABLE
rtb-0aXXXXXXXXXXXX3b

Internet Gatewayへのルーティングを追加して、

aws ec2 create-route --route-table-id $R_TABLE --destination-cidr-block 0.0.0.0/0 --gateway-id $I_GW
{
    "Return": true
}

Route TableをPublic SubnetにAssociateして、

aws ec2 associate-route-table --subnet-id $PUBLIC_SUBNET --route-table-id $R_TABLE
{
    "AssociationId": "rtbassoc-0eXXXXXXXXXXXX1e",
    "AssociationState": {
        "State": "associated"
    }
}

クラスタの名前でタグ付けする。

aws ec2 create-tags --resources $R_TABLE --tags Key=Name,Value=$ROSA_CLUSTER_NAME

NAT Gatewayの作成

EIPをかっさらってきて、

EIP=`aws ec2 allocate-address --domain vpc | jq -r .AllocationId`
echo $EIP
eipalloc-03XXXXXXXXXXXX0e

NAT Gatewayを作成して、

NAT_GW=`aws ec2 create-nat-gateway --subnet-id $PUBLIC_SUBNET --allocation-id $EIP | jq -r .NatGateway.NatGatewayId`
echo $NAT_GW
nat-09XXXXXXXXXXXXae

クラスタの名前でタグ付けする。

aws ec2 create-tags --resources $EIP --resources $NAT_GW --tags Key=Name,Value=$ROSA_CLUSTER_NAME

Route Tableの作成（Private Subnet用）

Route Tableを作って、

R_TABLE_NAT=`aws ec2 create-route-table --vpc-id $VPC_ID | jq -r .RouteTable.RouteTableId`
echo $R_TABLE_NAT
rtb-0aXXXXXXXXXXXXab

NAT Gatewayへのルーティングを追加して、

aws ec2 create-route --route-table-id $R_TABLE_NAT --destination-cidr-block 0.0.0.0/0 --gateway-id $NAT_GW
{
    "Return": true
}

Route TableをPrivate SubnetにAssociateして、

aws ec2 associate-route-table --subnet-id $PRIVATE_SUBNET --route-table-id $R_TABLE_NAT
{
    "AssociationId": "rtbassoc-07XXXXXXXXXXXX45",
    "AssociationState": {
        "State": "associated"
    }
}

諸々、タグ付けする。

aws ec2 create-tags --resources $R_TABLE_NAT $EIP --tags Key=Name,Value=$ROSA_CLUSTER_NAME-private

ROSAクラスタを作成する

PolicyとRoleの作成

STSモードでROSAを構築すると、IAM Userやそれに付随するPolicyを必要とせずにROSAをデプロイできる。
その代わりに、RoleとPolicyを使用して、クラスタのインストールと、Red HatのSRE Teamによる運用のアクセスを行う。

rosa create account-rolesを用いて、必要なPolicyとRoleを一挙に作成する。（なにこれ、超便利...）

rosa create account-roles --mode auto --version "${VERSION%.*}" -y
I: Creating roles using 'arn:aws:iam::XXXXXXXXXXXX:user/1ksen-rosa-poc'
I: Created role 'ManagedOpenShift-Installer-Role' with ARN 'arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-Installer-Role'
I: Created role 'ManagedOpenShift-ControlPlane-Role' with ARN 'arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-ControlPlane-Role'
I: Created role 'ManagedOpenShift-Worker-Role' with ARN 'arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-Worker-Role'
I: Created role 'ManagedOpenShift-Support-Role' with ARN 'arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-Support-Role'
I: Created policy with ARN 'arn:aws:iam::XXXXXXXXXXXX:policy/ManagedOpenShift-openshift-image-registry-installer-cloud-creden'
I: Created policy with ARN 'arn:aws:iam::XXXXXXXXXXXX:policy/ManagedOpenShift-openshift-ingress-operator-cloud-credentials'
I: Created policy with ARN 'arn:aws:iam::XXXXXXXXXXXX:policy/ManagedOpenShift-openshift-cluster-csi-drivers-ebs-cloud-credent'
I: Created policy with ARN 'arn:aws:iam::XXXXXXXXXXXX:policy/ManagedOpenShift-openshift-machine-api-aws-cloud-credentials'
I: Created policy with ARN 'arn:aws:iam::XXXXXXXXXXXX:policy/ManagedOpenShift-openshift-cloud-credential-operator-cloud-crede'
I: To create a cluster with these roles, run the following command:
rosa create cluster \
        --role-arn arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-Installer-Role \
        --master-iam-role arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-ControlPlane-Role \
        --worker-iam-role arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-Worker-Role \
        --support-role-arn arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-Support-Role

クラスタのインストールを開始

rosa create cluster -y --cluster-name ${ROSA_CLUSTER_NAME} \
   --region ${REGION} --version ${VERSION} \
   --subnet-ids=$PRIVATE_SUBNET \
   --private-link --machine-cidr=10.0.0.0/16 \
   --support-role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/ManagedOpenShift-Support-Role \
   --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/ManagedOpenShift-Installer-Role \
   --master-iam-role arn:aws:iam::${AWS_ACCOUNT_ID}:role/ManagedOpenShift-ControlPlane-Role \
   --worker-iam-role arn:aws:iam::${AWS_ACCOUNT_ID}:role/ManagedOpenShift-Worker-Role
W: You are choosing to use AWS PrivateLink for your cluster. Once the cluster is created, this option cannot be changed.
I: Creating cluster 'rosacluster'
I: To view a list of clusters and their status, run 'rosa list clusters'
I: Cluster 'rosacluster' has been created.
I: Once the cluster is installed you will need to add an Identity Provider before you can login into the cluster. See 'rosa create idp --help' for more information.
I: To determine when your cluster is Ready, run 'rosa describe cluster -c rosacluster'.
I: To watch your cluster installation logs, run 'rosa logs install -c rosacluster --watch'.
Name:                       rosacluster
ID:                         1mXXXXXXXXXXXXXXXXXXXXXXXXbi
External ID:                
OpenShift Version:          
Channel Group:              stable
DNS:                        rosacluster.XXXXXXXXXXXX.p1.openshiftapps.com
AWS Account:                XXXXXXXXXXXX
API URL:                    
Console URL:                
Region:                     ap-northeast-1
Multi-AZ:                   false
Nodes:
 - Master:                  3
 - Infra:                   2
 - Compute:                 2
Network:
 - Service CIDR:            172.30.0.0/16
 - Machine CIDR:            10.0.0.0/16
 - Pod CIDR:                10.128.0.0/14
 - Host Prefix:             /23
STS Role ARN:               arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-Installer-Role
Support Role ARN:           arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-Support-Role
Instance IAM Roles:
 - Master:                  arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-ControlPlane-Role
 - Worker:                  arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-Worker-Role
Operator IAM Roles:
 - arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-machine-api-aws-cloud-credentials
 - arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-cloud-credential-operator-cloud-credential
 - arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-image-registry-installer-cloud-credentials
 - arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-ingress-operator-cloud-credentials
 - arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-cluster-csi-drivers-ebs-cloud-credentials
State:                      pending (Preparing account)
Private:                    Yes
Created:                    Aug 23 2021 14:17:11 UTC
Details Page:               https://console.redhat.com/openshift/details/s/1xXXXXXXXXXXXXXXXXXXXXXXXXEV
OIDC Endpoint URL:          https://rh-oidc.s3.us-east-1.amazonaws.com/1mXXXXXXXXXXXXXXXXXXXXXXXXbi

OIDC Providerの設定

先ず、Pending状態になっていることを確認する。

while ! \
rosa describe cluster -c $ROSA_CLUSTER_NAME | grep "Waiting for OIDC"; \
do echo -n .; sleep 1; done
State:                      pending (Waiting for OIDC configuration)

Operator Roleの作成をする。（後で確認）

rosa create operator-roles -c $ROSA_CLUSTER_NAME --mode auto --yes
I: Creating roles using 'arn:aws:iam::XXXXXXXXXXXX:user/1ksen-rosa-poc'
I: Created role 'rosacluster-openshift-ingress-operator-cloud-credentials' with ARN 'arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-ingress-operator-cloud-credentials'
I: Created role 'rosacluster-openshift-cluster-csi-drivers-ebs-cloud-credentials' with ARN 'arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-cluster-csi-drivers-ebs-cloud-credentials'
I: Created role 'rosacluster-openshift-machine-api-aws-cloud-credentials' with ARN 'arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-machine-api-aws-cloud-credentials'
I: Created role 'rosacluster-openshift-cloud-credential-operator-cloud-credential' with ARN 'arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-cloud-credential-operator-cloud-credential'
I: Created role 'rosacluster-openshift-image-registry-installer-cloud-credentials' with ARN 'arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-image-registry-installer-cloud-credentials'

OIDC Providerの設定をする。（後で確認）

rosa create oidc-provider -c $ROSA_CLUSTER_NAME --mode auto --yes
I: Creating OIDC provider using 'arn:aws:iam::XXXXXXXXXXXX:user/1ksen-rosa-poc'
I: Created OIDC provider with ARN 'arn:aws:iam::XXXXXXXXXXXX:oidc-provider/rh-oidc.s3.us-east-1.amazonaws.com/1mXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXbi'

watchコマンドでStateがInstallingになっていることを確認する。

watch "rosa describe cluster -c $ROSA_CLUSTER_NAME"

インストールの進捗ログは以下で確認できる。

rosa logs install -c $ROSA_CLUSTER_NAME --watch --tail 10

無事に終わると以下のようなログが出る。

\ I: Cluster 'rosacluster' is now ready
time="2021-08-23T15:15:10Z" level=debug msg="Cluster is initialized"
time="2021-08-23T15:15:10Z" level=info msg="Waiting up to 10m0s for the openshift-console route to be created..."
time="2021-08-23T15:15:10Z" level=debug msg="Route found in openshift-console namespace: console"
time="2021-08-23T15:15:10Z" level=debug msg="OpenShift console route is admitted"
time="2021-08-23T15:15:10Z" level=info msg="Install complete!"
time="2021-08-23T15:15:10Z" level=info msg="To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/output/auth/kubeconfig'"
time="2021-08-23T15:15:10Z" level=info msg="Access the OpenShift web-console here: https://console-openshift-console.apps.rosacluster.XXXXXXXXXXXX.p1.openshiftapps.com"
REDACTED LINE OF OUTPUT
time="2021-08-23T15:15:10Z" level=debug msg="Time elapsed per stage:"
time="2021-08-23T15:15:10Z" level=debug msg="    Infrastructure: 3m56s"
time="2021-08-23T15:15:10Z" level=debug msg="Bootstrap Complete: 11m28s"
time="2021-08-23T15:15:10Z" level=debug msg="               API: 5m2s"
time="2021-08-23T15:15:10Z" level=debug msg=" Bootstrap Destroy: 1m9s"
time="2021-08-23T15:15:10Z" level=debug msg=" Cluster Operators: 29m33s"
time="2021-08-23T15:15:10Z" level=info msg="Time elapsed: 46m10s"
time="2021-08-23T15:15:11Z" level=info msg="command completed successfully" installID=bp7tpfz6
time="2021-08-23T15:15:11Z" level=info msg="saving installer output" installID=bp7tpfz6

インストール完了後は、インバウンド向けのエンドポイントがないため、踏み台使うなり、VPCe追加するなりでルートを作ってログインすればいい。（ここではやらない）

散策

以下は後で眺める用途。

IAM Policy

以下のドキュメントはRed Hatのドキュメントにあるので、あとで読む。

ManagedOpenShift-openshift-cloud-credential-operator-cloud-crede
ManagedOpenShift-openshift-cluster-csi-drivers-ebs-cloud-credent
ManagedOpenShift-openshift-image-registry-installer-cloud-creden
ManagedOpenShift-openshift-ingress-operator-cloud-credentials
ManagedOpenShift-openshift-machine-api-aws-cloud-credentials

IAM Role

rosa*

aws iam list-roles | jq '.Roles[] | select( .RoleName | contains("rosa"))'
{
  "Path": "/",
  "RoleName": "rosacluster-openshift-cloud-credential-operator-cloud-credential",
  "RoleId": "XXXXXXXXXXXX",
  "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-cloud-credential-operator-cloud-credential",
  "CreateDate": "2021-08-23T14:25:52+00:00",
  "AssumeRolePolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/rh-oidc.s3.us-east-1.amazonaws.com/XXXXXXXXXXXX"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "rh-oidc.s3.us-east-1.amazonaws.com/XXXXXXXXXXXX": "system:serviceaccount:openshift-cloud-credential-operator:cloud-credential-operator"
          }
        }
      }
    ]
  },
  "MaxSessionDuration": 3600
}
{
  "Path": "/",
  "RoleName": "rosacluster-openshift-cluster-csi-drivers-ebs-cloud-credentials",
  "RoleId": "XXXXXXXXXXXX",
  "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-cluster-csi-drivers-ebs-cloud-credentials",
  "CreateDate": "2021-08-23T14:24:17+00:00",
  "AssumeRolePolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/rh-oidc.s3.us-east-1.amazonaws.com/XXXXXXXXXXXX"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "rh-oidc.s3.us-east-1.amazonaws.com/XXXXXXXXXXXX": [
              "system:serviceaccount:openshift-cluster-csi-drivers:aws-ebs-csi-driver-operator",
              "system:serviceaccount:openshift-cluster-csi-drivers:aws-ebs-csi-driver-controller-sa"
            ]
          }
        }
      }
    ]
  },
  "MaxSessionDuration": 3600
}
{
  "Path": "/",
  "RoleName": "rosacluster-openshift-image-registry-installer-cloud-credentials",
  "RoleId": "XXXXXXXXXXXX",
  "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-image-registry-installer-cloud-credentials",
  "CreateDate": "2021-08-23T14:22:04+00:00",
  "AssumeRolePolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/rh-oidc.s3.us-east-1.amazonaws.com/XXXXXXXXXXXX"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "rh-oidc.s3.us-east-1.amazonaws.com/XXXXXXXXXXXX": [
              "system:serviceaccount:openshift-image-registry:cluster-image-registry-operator",
              "system:serviceaccount:openshift-image-registry:registry"
            ]
          }
        }
      }
    ]
  },
  "MaxSessionDuration": 3600
}
{
  "Path": "/",
  "RoleName": "rosacluster-openshift-ingress-operator-cloud-credentials",
  "RoleId": "XXXXXXXXXXXX",
  "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-ingress-operator-cloud-credentials",
  "CreateDate": "2021-08-23T14:24:07+00:00",
  "AssumeRolePolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/rh-oidc.s3.us-east-1.amazonaws.com/XXXXXXXXXXXX"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "rh-oidc.s3.us-east-1.amazonaws.com/XXXXXXXXXXXX": "system:serviceaccount:openshift-ingress-operator:ingress-operator"
          }
        }
      }
    ]
  },
  "MaxSessionDuration": 3600
}
{
  "Path": "/",
  "RoleName": "rosacluster-openshift-machine-api-aws-cloud-credentials",
  "RoleId": "XXXXXXXXXXXX",
  "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/rosacluster-openshift-machine-api-aws-cloud-credentials",
  "CreateDate": "2021-08-23T14:25:48+00:00",
  "AssumeRolePolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/rh-oidc.s3.us-east-1.amazonaws.com/XXXXXXXXXXXX"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "rh-oidc.s3.us-east-1.amazonaws.com/XXXXXXXXXXXX": "system:serviceaccount:openshift-machine-api:machine-api-controllers"
          }
        }
      }
    ]
  },
  "MaxSessionDuration": 3600
}

ManagedOpenShift*

]# aws iam list-roles | jq '.Roles[] | select( .RoleName | contains("ManagedOpenShift"))'
{
  "Path": "/",
  "RoleName": "ManagedOpenShift-ControlPlane-Role",
  "RoleId": "XXXXXXXXXXXX",
  "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-ControlPlane-Role",
  "CreateDate": "2021-08-23T14:14:24+00:00",
  "AssumeRolePolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Service": "ec2.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
      }
    ]
  },
  "MaxSessionDuration": 3600
}
{
  "Path": "/",
  "RoleName": "ManagedOpenShift-Installer-Role",
  "RoleId": "XXXXXXXXXXXX",
  "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-Installer-Role",
  "CreateDate": "2021-08-23T14:14:24+00:00",
  "AssumeRolePolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "AWS": "arn:aws:iam::710019948333:role/RH-Managed-OpenShift-Installer"
        },
        "Action": "sts:AssumeRole"
      }
    ]
  },
  "MaxSessionDuration": 3600
}
{
  "Path": "/",
  "RoleName": "ManagedOpenShift-Support-Role",
  "RoleId": "XXXXXXXXXXXX",
  "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-Support-Role",
  "CreateDate": "2021-08-23T14:14:26+00:00",
  "AssumeRolePolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "AWS": "arn:aws:iam::710019948333:role/RH-Technical-Support-Access"
        },
        "Action": "sts:AssumeRole"
      }
    ]
  },
  "MaxSessionDuration": 3600
}
{
  "Path": "/",
  "RoleName": "ManagedOpenShift-Worker-Role",
  "RoleId": "XXXXXXXXXXXX",
  "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/ManagedOpenShift-Worker-Role",
  "CreateDate": "2021-08-23T14:14:25+00:00",
  "AssumeRolePolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Service": "ec2.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
      }
    ]
  },
  "MaxSessionDuration": 3600
}

OpenID Provider

get-open-id-connect-provider.json

# aws iam get-open-id-connect-provider --open-id-connect-provider-arn arn:aws:iam::XXXXXXXXXXXX:oidc-provider/rh-oidc.s3.us-east-1.amazonaws.com/XXXXXXXXXXXX
{
    "Url": "rh-oidc.s3.us-east-1.amazonaws.com/XXXXXXXXXXXX",
    "ClientIDList": [
        "openshift",
        "sts.amazonaws.com"
    ],
    "ThumbprintList": [
        "XXXXXXXXXXXX"
    ],
    "CreateDate": "2021-08-23T14:27:02.020000+00:00",
    "Tags": []
}

Private Link

private-link-access-for.json

{
            "ServiceName": "com.amazonaws.vpce.ap-northeast-1.vpce-svc-XXXXXXXXXXXX",
            "ServiceId": "vpce-svc-XXXXXXXXXXXX",
            "ServiceType": [
                {
                    "ServiceType": "Interface"
                }
            ],
            "AvailabilityZones": [
                "ap-northeast-1d"
            ],
            "Owner": "XXXXXXXXXXXX",
            "BaseEndpointDnsNames": [
                "vpce-svc-XXXXXXXXXXXX.ap-northeast-1.vpce.amazonaws.com"
            ],
            "VpcEndpointPolicySupported": false,
            "AcceptanceRequired": false,
            "ManagesVpcEndpoints": false,
            "Tags": [
                {
                    "Key": "hive.openshift.io/private-link-access-for",
                    "Value": "rosacluster-XXXXXXXXXXXX"
                },
                {
                    "Key": "Name",
                    "Value": "rosacluster-XXXXXXXXXXXX-vpc-endpoint-service"
                }
            ]
}

# aws ec2 describe-vpc-endpoint-service-permissions --service-id vpce-svc-XXXXXXXXXXXX
{
    "AllowedPrincipals": [
        {
            "PrincipalType": "User",
            "Principal": "arn:aws:iam::710019948333:user/hive-privatelink-production"
        }
    ]
}

]# aws ec2 describe-vpc-endpoint-service-configurations
{
    "ServiceConfigurations": [
        {
            "ServiceType": [
                {
                    "ServiceType": "Interface"
                }
            ],
            "ServiceId": "vpce-svc-XXXXXXXXXXXX",
            "ServiceName": "com.amazonaws.vpce.ap-northeast-1.vpce-svc-XXXXXXXXXXXX",
            "ServiceState": "Available",
            "AvailabilityZones": [
                "ap-northeast-1d"
            ],
            "AcceptanceRequired": false,
            "ManagesVpcEndpoints": false,
            "NetworkLoadBalancerArns": [
                "arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXXXX:loadbalancer/net/rosacluster-XXXXXXXXXXXX-int/XXXXXXXXXXXX"
            ],
            "BaseEndpointDnsNames": [
                "vpce-svc-XXXXXXXXXXXX.ap-northeast-1.vpce.amazonaws.com"
            ],
            "PrivateDnsNameConfiguration": {},
            "Tags": [
                {
                    "Key": "hive.openshift.io/private-link-access-for",
                    "Value": "rosacluster-2gdb9"
                },
                {
                    "Key": "Name",
                    "Value": "rosacluster-2gdb9-vpc-endpoint-service"
                }
            ]
        }
    ]
}

ELB

{
            "LoadBalancerName": "XXXXXXXXXXXX",
            "DNSName": "internal-XXXXXXXXXXXX-XXXXXXXXXXXX.ap-northeast-1.elb.amazonaws.com",
            "CanonicalHostedZoneNameID": "XXXXXXXXXXXX",
            "ListenerDescriptions": [
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 443,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 31395
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 80,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 30884
                    },
                    "PolicyNames": []
                }
            ],
            "Policies": {
                "AppCookieStickinessPolicies": [],
                "LBCookieStickinessPolicies": [],
                "OtherPolicies": [
                    "k8s-proxyprotocol-enabled"
                ]
            },
            "BackendServerDescriptions": [
                {
                    "InstancePort": 30884,
                    "PolicyNames": [
                        "k8s-proxyprotocol-enabled"
                    ]
                },
                {
                    "InstancePort": 31395,
                    "PolicyNames": [
                        "k8s-proxyprotocol-enabled"
                    ]
                }
            ],
            "AvailabilityZones": [
                "ap-northeast-1d"
            ],
            "Subnets": [
                "subnet-XXXXXXXXXXXX"
            ],
            "VPCId": "vpc-XXXXXXXXXXXX",
            "Instances": [
                {
                    "InstanceId": "i-XXXXXXXXXXXX"
                },
                {
                    "InstanceId": "i-XXXXXXXXXXXX"
                },
                {
                    "InstanceId": "i-XXXXXXXXXXXX"
                },
                {
                    "InstanceId": "i-XXXXXXXXXXXX"
                },
                {
                    "InstanceId": "i-XXXXXXXXXXXX"
                },
                {
                    "InstanceId": "i-XXXXXXXXXXXX"
                },
                {
                    "InstanceId": "i-XXXXXXXXXXXX"
                }
            ],
            "HealthCheck": {
                "Target": "HTTP:30338/healthz",
                "Interval": 5,
                "Timeout": 4,
                "UnhealthyThreshold": 2,
                "HealthyThreshold": 2
            },
            "SourceSecurityGroup": {
                "OwnerAlias": "XXXXXXXXXXXX",
                "GroupName": "k8s-elb-XXXXXXXXXXXX"
            },
            "SecurityGroups": [
                "sg-XXXXXXXXXXXX"
            ],
            "CreatedTime": "2021-08-23T14:46:00.200000+00:00",
            "Scheme": "internal"
}

ELBv2

# aws elbv2 describe-listeners --load-balancer-arn arn:aws:elasticloadbalancing:ap
-northeast-1:XXXXXXXXXXXX:loadbalancer/net/rosacluster-XXXXXXXXXXXX-int/XXXXXXXXXXXX
{
    "Listeners": [
        {
            "ListenerArn": "arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXXXX:listener/net/rosacluster-XXXXXXXXXXXX-int/XXXXXXXXXXXX/XXXXXXXXXXXX",
            "LoadBalancerArn": "arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXXXX:loadbalancer/net/rosacluster-XXXXXXXXXXXX-int/XXXXXXXXXXXX",
            "Port": 6443,
            "Protocol": "TCP",
            "DefaultActions": [
                {
                    "Type": "forward",
                    "TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXXXX:targetgroup/rosacluster-XXXXXXXXXXXX-aint/XXXXXXXXXXXX",
                    "Order": 1,
                    "ForwardConfig": {
                        "TargetGroups": [
                            {
                                "TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXXXX:targetgroup/rosacluster-XXXXXXXXXXXX-aint/XXXXXXXXXXXX"
                            }
                        ]
                    }
                }
            ]
        },
        {
            "ListenerArn": "arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXXXX:listener/net/rosacluster-XXXXXXXXXXXX-int/XXXXXXXXXXXX/XXXXXXXXXXXX",
            "LoadBalancerArn": "arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXXXX:loadbalancer/net/rosacluster-XXXXXXXXXXXX-int/XXXXXXXXXXXX",
            "Port": 22623,
            "Protocol": "TCP",
            "DefaultActions": [
                {
                    "Type": "forward",
                    "TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXXXX:targetgroup/rosacluster-XXXXXXXXXXXX-sint/XXXXXXXXXXXX",
                    "Order": 1,
                    "ForwardConfig": {
                        "TargetGroups": [
                            {
                                "TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXXXX:targetgroup/rosacluster-XXXXXXXXXXXX-sint/XXXXXXXXXXXX"
                            }
                        ]
                    }
                }
            ]
        }
    ]
}

片付け

クラスタを消す。
ロールを消す。
OIDC Providerを消す
VPC周辺のリソースを消す
AWS Accountを消す

メモ

Internet GatewayやPublic SubnetはMUSTなのか。
- 組織のフォワードプロキシがあって、フロントにNLBが立っていれば、VPCeを作って、そちらにルーティングすればいけるんちゃうか？
Private LinkはPrivate Subnetに作られたNLBにLinkされてた。
- 22623ってSSH用途なんかな。

You get articles that match your needs
You can efficiently read back useful information
You can use dark theme

What you can do with signing up