Amazon EKS on Fargate - NLB (AWS Load Balancer Controller)

ポイント

• AWS Load Balancer Controller を利用する。AWS Load Balancer Controller によって Kubernetes クラスター向けの AWS ELB が管理される。

• Service で type: LoadBalancer を指定することで NLB が作成される。

kind: Service
spec:
  type: LoadBalancer

• AWS Fargate では、IPターゲットのみ使用可能。IPターゲットを使用するロードバランサーを作成するには、annotationsに以下の通り追加する。

kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

• デフォルトで NLB は internal で作成されるため以下の通り明示的に annotations に追加することは不要。

kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal

Demo

cluster

node には Fargate を利用するので、以下の通り fargateProfiles を準備。

clusterconfig-fargate.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: cluster-fargate
  region: ap-northeast-1
  version: "1.27"

fargateProfiles:
  - name: fp-default
    selectors:
      - namespace: default
      - namespace: kube-system

# cluster を作成
eksctl create cluster -f clusterconfig-fargate.yaml

# cluster が作成されたことを確認
eksctl get cluster -o yaml
    - Name: cluster-fargate
      Owned: "True"
      Region: ap-northeast-1

# fargateprofile が作成されていることを確認
eksctl get fargateprofile --cluster cluster-fargate -o yaml
    - name: fp-default
      podExecutionRoleARN: arn:aws:iam::12345678912:role/eksctl-cluster-fargate-clus-FargatePodExecutionRole-yjBKxAuTqZWa
      selectors:
      - namespace: default
      - namespace: kube-system
      status: ACTIVE
      subnets:
      - subnet-0131dadd998fb6ad3
      - subnet-0a233d27609fac647
      - subnet-00c7860bd84297c6d

IAM と ServiceAccount

ここでは、
AWS API を呼び出すことを許可する、AWS Load Balancer Controller 用の IAMポリシーを作成、
kube-system namespace に aws-load-balancer-controller という名前の Kubernetes ServiceAccount を作成、
Kubernetes ServiceAccount に IAMロールをアノテーション
します。
このあたりについてはまた別の記事で詳しく書こうと思います。

# iam_policy.json ダウンロード
curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.5.4/docs/install/iam_policy.json

# ダウンロードした　iam_policy.json で IAMポリシーを作成
aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json

# IAMロールと Serviceaccount を作成
eksctl create iamserviceaccount \
  --cluster=cluster-fargate \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn=arn:aws:iam::123456789012:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve

aws-load-balancer-controller

Fargate でコントローラーをデプロイする場合は、Helm を使用します。

# eks-charts リポジトリを追加
helm repo add eks https://aws.github.io/eks-charts
    "eks" has been added to your repositories

# ローカルリポジトリを更新
helm repo update eks
    Hang tight while we grab the latest from your chart repositories...
    ...Successfully got an update from the "eks" chart repository
    Update Complete. ⎈Happy Helming!⎈

# ローカルリポジトリを確認
helm repo list
    NAME            URL                                       
    eks             https://aws.github.io/eks-charts 

# aws-load-balancer-controller をインストール
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=fargate-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=ap-northeast-1 \
  --set vpcId=<vpc-id>

    NAME: aws-load-balancer-controller
    LAST DEPLOYED: Sat Nov  4 15:56:32 2023
    NAMESPACE: kube-system
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    AWS Load Balancer controller installed!

# aws-load-balancer-controller がインストールされていることを確認
kubectl get deployment -n kube-system aws-load-balancer-controller
    NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
    aws-load-balancer-controller   2/2     2            2           3m6s

Deployment

nlb-deployment.yaml

apiVersion: apps/v1
kind: Deployment

metadata:
  name: nlb-deployment
  namespace: default

spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: public.ecr.aws/nginx/nginx:latest
          ports:
            - name: tcp
              containerPort: 80

# nlb-deployment.yaml を apply
kubectl apply -f nlb-deployment.yaml
    deployment.apps/nlb-deployment created

# Deployment,Replicaset,pod を確認
kubectl get deploy,rs,pod -o wide
    NAME                             READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                              SELECTOR
    deployment.apps/nlb-deployment   2/2     2            2           3m48s   nginx        public.ecr.aws/nginx/nginx:latest   app=nginx

    NAME                                        DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES                              SELECTOR
    replicaset.apps/nlb-deployment-6c4dd645c8   2         2         2       3m48s   nginx        public.ecr.aws/nginx/nginx:latest   app=nginx,pod-template-hash=6c4dd645c8

    NAME                                  READY   STATUS    RESTARTS   AGE     IP                NODE                                                         NOMINATED NODE   READINESS GATES
    pod/nlb-deployment-6c4dd645c8-7b6vm   1/1     Running   0          3m48s   192.168.100.149   fargate-ip-192-168-100-149.ap-northeast-1.compute.internal   <none>           <none>
    pod/nlb-deployment-6c4dd645c8-lwr7f   1/1     Running   0          3m48s   192.168.148.11    fargate-ip-192-168-148-11.ap-northeast-1.compute.internal    <none>           <none>

Service

nlb-service.yaml

apiVersion: v1
kind: Service

metadata:
  name: nlb-service
  namespace: default
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: LoadBalancer
  selector:
    app: nginx

# nlb-service.yaml を apply
kubectl apply -f nlb-service.yaml 
    service/nlb-service created

# service が作成されていることを確認
kubectl get svc -o wide
    NAME          TYPE           CLUSTER-IP       EXTERNAL-IP                                                                         PORT(S)        AGE   SELECTOR
    nlb-service   LoadBalancer   10.100.167.138   k8s-default-nlbservi-a2f31fc77c-fd84375ea7195e12.elb.ap-northeast-1.amazonaws.com   80:31110/TCP   98s   app=nginx

# service の詳細を確認
kubectl describe svc nlb-service 
    Name:                     nlb-service
    Namespace:                default
    Labels:                   <none>
    Annotations:              service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
                              service.beta.kubernetes.io/aws-load-balancer-type: external
    Selector:                 app=nginx
    Type:                     LoadBalancer
    IP Family Policy:         SingleStack
    IP Families:              IPv4
    IP:                       10.100.167.138
    IPs:                      10.100.167.138
    LoadBalancer Ingress:     k8s-default-nlbservi-a2f31fc77c-fd84375ea7195e12.elb.ap-northeast-1.amazonaws.com
    Port:                     <unset>  80/TCP
    TargetPort:               80/TCP
    NodePort:                 <unset>  31110/TCP
    Endpoints:                192.168.100.149:80,192.168.148.11:80
    Session Affinity:         None
    External Traffic Policy:  Cluster
    Events:
      Type    Reason                  Age    From     Message
      ----    ------                  ----   ----     -------
      Normal  SuccessfullyReconciled  2m40s  service  Successfully reconcile

動作確認

内部にEC2インスタンスをたて、curl を実行

curl k8s-default-nlbservi-a2f31fc77c-XXXXXXXXXXXXXXXX.elb.ap-northeast-1.amazonaws.com
    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    <style>
    html { color-scheme: light dark; }
    body { width: 35em; margin: 0 auto;
    font-family: Tahoma, Verdana, Arial, sans-serif; }
    </style>
    </head>
    <body>
    <h1>Welcome to nginx!</h1>
    <p>If you see this page, the nginx web server is successfully installed and
    working. Further configuration is required.</p>
    
    <p>For online documentation and support please refer to
    <a href="http://nginx.org/">nginx.org</a>.<br/>
    Commercial support is available at
    <a href="http://nginx.com/">nginx.com</a>.</p>
    
    <p><em>Thank you for using nginx.</em></p>
    </body>
    </html>

参考