What's？

少し前に、AWS FireLensについて調べてみたのですが、今度は使ってみようかな、ということで。

AWS FireLensってなんだ？

お題

こんなお題で試してみます。

AWS Fargateクラスタを構築する
タスクには、nginxをリバースプロキシとして、Pythonで作ったアプリケーションを背後に配置
- ログドライバーとして、awsfirelensを使用する
サイドカーとして、AWS for Fluent Bitを配置して、nginxとアプリケーションのコンテナログを受け取る
受け取った他のコンテナログおよび、AWS for Fluent Bit自身のログは、Amazon CloudWatch Logsに送る
環境は、Terraformで構築する

環境

今回の環境は、こちらです。

$ terraform version
Terraform v0.13.3
+ provider registry.terraform.io/hashicorp/aws v3.6.0

AWSのクレデンシャルは、環境変数で設定するものとします。

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_DEFAULT_REGION=ap-northeast-1

Amazon ECRを作成する

最初に、自前で作成するコンテナイメージを格納するための、Amazon ECRリポジトリを作成しましょう。

nginx用と、アプリケーション用の2つを用意します。

main.tf

terraform {
  required_version = "0.13.3"

  required_providers {
    aws = "3.6.0"
  }
}

provider "aws" {
}

resource "aws_ecr_repository" "nginx" {
  name = "nginx"
}

resource "aws_ecr_repository" "app" {
  name = "app"
}

作成。

$ terraform apply

Dockerイメージを作成して、Amazon ECRにPushする

続いて、Dockerイメージを作成します。

最初は、nginx。

リバースプロキシとして構築するので、そのように設定ファイルを構成。

default.conf

server {
    listen       80;
    listen  [::]:80;
    server_name  localhost;

    location / {
        proxy_pass   http://localhost:8000;
    }
}

これを含めるようにDockerfileを定義して

Dockerfile

FROM nginx:1.19.2

COPY default.conf /etc/nginx/conf.d/default.conf

ビルド。

$ docker image build -t charon/nginx:latest .

続いて、アプリケーション側。Flask-RESTfulを使って作成することにしました。

ソースコードは、こちら。

index.py

from datetime import datetime
import logging

from flask import Flask
from flask_restful import Resource, Api

app = Flask(__name__)
api = Api(app)

app.logger.setLevel(logging.INFO)

class HelloWorld(Resource):
    def get(self):
        app.logger.info('access => Hello World')
        return {'message': 'Hello Flask!!', 'time': f'{datetime.now()}'}

api.add_resource(HelloWorld, '/')

if __name__ == '__main__':
    app.run(debug = True)

Dockerfileを用意します。WSGIサーバーとしては、Gunicornを使うことにしました。

Dockerfile

FROM python:3.8.5-slim-buster

COPY index.py index.py

RUN pip install flask-restful==0.3.8 gunicorn==20.0.4

EXPOSE 8000

ENTRYPOINT ["gunicorn", "index:app"]
CMD ["--threads", "10"]

ビルド。

$ docker image build -t charon/app:latest .

あとは、Amazon ECRにログインして

$ aws ecr get-login-password --region ap-northeast-1 | docker login --username AWS --password-stdin [AWSアカウントID].dkr.ecr.ap-northeast-1.amazonaws.com

作成したイメージに、Amazon ECR用のタグをつけてPushします。

$ docker image tag charon/nginx:latest [AWSアカウントID].dkr.ecr.ap-northeast-1.amazonaws.com/nginx:latest
$ docker image push [AWSアカウントID].dkr.ecr.ap-northeast-1.amazonaws.com/nginx:latest
$ docker image tag charon/app:latest [AWSアカウントID].dkr.ecr.ap-northeast-1.amazonaws.com/app:latest
$ docker image push [AWSアカウントID].dkr.ecr.ap-northeast-1.amazonaws.com/app:latest

AWS Fargateクラスタを構築する

それでは、AWS Fargateクラスタを構築しましょう。

定義はTerraformで書いていきます。VPCやALB、セキュリティグループ等も作成しますが、そちらのコードはまとめて最後に載せます。

main.tf

locals {
  vpc_id = module.vpc.vpc_id

  private_subnets = module.vpc.private_subnets
  app_with_nginx_service_security_groups = [
  module.app_with_nginx_service_sg.this_security_group_id]
  load_balancer_target_group_arn = module.load_balancer.target_group_arns[0]
}

IAMロールの作成。次の2つが必要です。

Amazon ECSに対するAssume Role。

data "aws_iam_policy_document" "assume_role" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["ecs-tasks.amazonaws.com"]
    }
  }
}

タスク実行用のIAMロール。

resource "aws_iam_role" "ecs_task_execution_role" {
  name               = "MyEcsTaskExecutionRole"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

resource "aws_iam_role_policy_attachment" "amazon_ecs_task_execution_role_policy" {
  role       = aws_iam_role.ecs_task_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

タスク用のIAMロール。

data "aws_iam_policy_document" "ecs_task_role_policy_document" {
  statement {
    effect = "Allow"

    actions = [
      "logs:DescribeLogStreams",
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]

    resources = ["*"]
  }
}

resource "aws_iam_policy" "ecs_task_role_policy" {
  name   = "MyEcsTaskPolicy"
  policy = data.aws_iam_policy_document.ecs_task_role_policy_document.json
}

resource "aws_iam_role" "ecs_task_role" {
  name               = "MyEcsTaskRole"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

resource "aws_iam_role_policy_attachment" "ecs_task_role_policy_attachment" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = aws_iam_policy.ecs_task_role_policy.arn
}

タスクに割り当てる権限として、今回は以下が必要になります。サイドカーとして配置する、AWS for Fluent BitコンテナがAmazon CloudWatch Logsにログを送信するに権限が必要だからです。

Fluent Bit Plugin for CloudWatch Logs / Permissions

data "aws_iam_policy_document" "ecs_task_role_policy_document" {
  statement {
    effect = "Allow"

    actions = [
      "logs:DescribeLogStreams",
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]

    resources = ["*"]
  }
}

これが設定できていないと、Fluent BitコンテナがAmazon CloudWatch Logsにログが送れず、こんな感じのエラーになります。

2020-09-17T07:47:10.566000+00:00 firelens/log_router/a02c7e22-4270-4587-8a7f-c98339246a5c time="2020-09-17T07:47:10Z" level=error msg="NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, .\nEC2RoleRequestError: no EC2 instance role found\ncaused by: RequestError: send request failed\ncaused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: invalid argument"

どのような権限が必要になるかは、ログの送信設定によって変わるので、利用するプラグインやリソースのドキュメントを見て設定してください。

カスタムログルーティング / 必要な IAM アクセス許可

Amazon CloudWatch Logsのロググループの作成。今回は、AWS FireLensログドライバーによって収集して送信するログも、AWS for Fluent Bit自身のログも、すべてAmazon CloudWatch Logsに送ります。

前もって作成しておきましょう。

resource "aws_cloudwatch_log_group" "nginx_container_log_group" {
  name = "/fargate/app/nginx"
}

resource "aws_cloudwatch_log_group" "app_container_log_group" {
  name = "/fargate/app/app"
}

resource "aws_cloudwatch_log_group" "firelens_container_log_group" {
  name = "/fargate/app/firelens"
}

AWS Fargetクラスタ定義。

resource "aws_ecs_cluster" "app_with_nginx" {
  name = "app-with-nginx-cluster"
}

resource "aws_ecs_task_definition" "app_with_nginx" {
  family                   = "app-with-nginx-task-definition"
  cpu                      = "1024"
  memory                   = "2048"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn            = aws_iam_role.ecs_task_role.arn

  container_definitions = <<JSON
  [
    {
      "name": "nginx",
      "image": "[AWSアカウントID].dkr.ecr.ap-northeast-1.amazonaws.com/nginx:latest",
      "essential": true,
      "portMappings": [
        {
          "protocol": "tcp",
          "containerPort": 80
        }
      ],
      "cpu": 256,
      "memory": 512,
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch",
          "region": "ap-northeast-1",
          "log_group_name": "/fargate/app/nginx",
          "log_stream_prefix": "nginx-",
          "auto_create_group": "false"
        }
      }
    },
    {
      "name": "app",
      "image": "[AWSアカウントID].dkr.ecr.ap-northeast-1.amazonaws.com/app:latest",
      "essential": true,
      "cpu": 256,
      "memory": 512,
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch",
          "region": "ap-northeast-1",
          "log_group_name": "/fargate/app/app",
          "log_stream_prefix": "app-",
          "auto_create_group": "false"
        }
      }
    },
    {
      "name": "log_router",
      "image": "906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:latest",
      "essential": true,
      "cpu": 256,
      "memory": 512,
      "firelensConfiguration": {
        "type": "fluentbit"
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/fargate/app/firelens",
          "awslogs-region": "ap-northeast-1",
          "awslogs-stream-prefix": "firelens"
        }
      }
    }
  ]
  JSON
}

resource "aws_ecs_service" "app_with_nginx" {
  name             = "app-with-nginx-service"
  cluster          = aws_ecs_cluster.app_with_nginx.arn
  task_definition  = aws_ecs_task_definition.app_with_nginx.arn
  desired_count    = 3
  launch_type      = "FARGATE"
  platform_version = "1.4.0"

  deployment_minimum_healthy_percent = 50

  network_configuration {
    assign_public_ip = false
    security_groups  = local.app_with_nginx_service_security_groups
    subnets          = local.private_subnets
  }

  load_balancer {
    target_group_arn = local.load_balancer_target_group_arn
    container_name   = "nginx"
    container_port   = 80
  }
}

ポイントは、タスク定義に先程作成したタスク実行用のIAMロール、タスク用のIAMロールを割り当てているところと

resource "aws_ecs_task_definition" "app_with_nginx" {
  family                   = "app-with-nginx-task-definition"
  cpu                      = "1024"
  memory                   = "2048"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn            = aws_iam_role.ecs_task_role.arn

コンテナ定義ですね。

  container_definitions = <<JSON
  [
    {
      "name": "nginx",
      "image": "[AWSアカウントID].dkr.ecr.ap-northeast-1.amazonaws.com/nginx:latest",
      "essential": true,
      "portMappings": [
        {
          "protocol": "tcp",
          "containerPort": 80
        }
      ],
      "cpu": 256,
      "memory": 512,
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch",
          "region": "ap-northeast-1",
          "log_group_name": "/fargate/app/nginx",
          "log_stream_prefix": "nginx-",
          "auto_create_group": "false"
        }
      }
    },
    {
      "name": "app",
      "image": "[AWSアカウントID].dkr.ecr.ap-northeast-1.amazonaws.com/app:latest",
      "essential": true,
      "cpu": 256,
      "memory": 512,
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch",
          "region": "ap-northeast-1",
          "log_group_name": "/fargate/app/app",
          "log_stream_prefix": "app-",
          "auto_create_group": "false"
        }
      }
    },
    {
      "name": "log_router",
      "image": "906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:latest",
      "essential": true,
      "cpu": 256,
      "memory": 512,
      "firelensConfiguration": {
        "type": "fluentbit"
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/fargate/app/firelens",
          "awslogs-region": "ap-northeast-1",
          "awslogs-stream-prefix": "firelens"
        }
      }
    }
  ]
  JSON

AWS for Fluent Bitのコンテナは、サイドカーとして仕込みます。

    {
      "name": "log_router",
      "image": "906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:latest",
      "essential": true,
      "cpu": 256,
      "memory": 512,
      "firelensConfiguration": {
        "type": "fluentbit"
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/fargate/app/firelens",
          "awslogs-region": "ap-northeast-1",
          "awslogs-stream-prefix": "firelens"
        }
      }
    }

AWS内で使うので、ドキュメントにしたがって、AWS for Fluent BitのイメージはAmazon ECRから取得しています。

AWS for Fluent Bitイメージの使用

typeはFluent Bitであることを示し、

FirelensConfiguration

      "firelensConfiguration": {
        "type": "fluentbit"
      },

AWS for Fluent Bitコンテナ自身のログは、Amazon CloudWatch Logsにawslogsログドライバーを使って構成します。

      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/fargate/app/firelens",
          "awslogs-region": "ap-northeast-1",
          "awslogs-stream-prefix": "firelens"
        }
      }

他の2つのコンテナは、logDriverをawsfirelensに設定します。

    {
      "name": "nginx",
      "image": "[AWSアカウントID].dkr.ecr.ap-northeast-1.amazonaws.com/nginx:latest",
      "essential": true,
      "portMappings": [
        {
          "protocol": "tcp",
          "containerPort": 80
        }
      ],
      "cpu": 256,
      "memory": 512,
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch",
          "region": "ap-northeast-1",
          "log_group_name": "/fargate/app/nginx",
          "log_stream_prefix": "nginx-",
          "auto_create_group": "false"
        }
      }
    },
    {
      "name": "app",
      "image": "[AWSアカウントID].dkr.ecr.ap-northeast-1.amazonaws.com/app:latest",
      "essential": true,
      "cpu": 256,
      "memory": 512,
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch",
          "region": "ap-northeast-1",
          "log_group_name": "/fargate/app/app",
          "log_stream_prefix": "app-",
          "auto_create_group": "false"
        }
      }
    },

optionsで指定した内容は、Fluent Bit（もしくはFluentd）の設定ファイルに変換されます。

FireLens構成を使用するタスク定義の作成

      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch",
          "region": "ap-northeast-1",
          "log_group_name": "/fargate/app/nginx",
          "log_stream_prefix": "nginx-",
          "auto_create_group": "false"
        }
      }

今回はcloudwatchを指定しているので、Fluent Bit Plugin for CloudWatch Logs向けの設定が出力されることになります。

イメージ的には、こんな感じでしょうか？

[OUTPUT]
    Name   cloudwatch
    Match  nginx-firelens*
    region ap-northeast-1
    log_group_name /fargate/app/nginx
    log_stream_prefix nginx-
    auto_create_group false

今度、ちゃんと確認してみましょう…。

ちなみに、Amazon CloudWatch Logsにログを転送する例は、ドキュメントにサンプルがあったりします。

CloudWatch Logs へのログの転送

では、AWS Fargeteを構築します。

$ terraform apply

確認

構築が終わったら、動作確認してみます。

まずは、AWS Fargate自体の動作確認。

$ curl -i [ALBのDNS名]
HTTP/1.1 200 OK
Date: Thu, 17 Sep 2020 08:57:01 GMT
Content-Type: application/json
Content-Length: 67
Connection: keep-alive
Server: nginx/1.19.2

{"message": "Hello Flask!!", "time": "2020-09-17 08:57:01.008154"}

OKです。

nginxのログを見てみます。

$ aws logs tail --follow /fargate/app/nginx
2020-09-17T08:56:19+00:00 nginx-nginx-firelens-f5c7c43b-71d9-4e99-8609-28f7d4c116e3 {"container_id":"f5c7c43b-71d9-4e99-8609-28f7d4c116e3-2531612879","container_name":"nginx","ecs_cluster":"app-with-nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/f5c7c43b-71d9-4e99-8609-28f7d4c116e3","ecs_task_definition":"app-with-nginx-task-definition:5","log":"/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration","source":"stdout"}
2020-09-17T08:56:19+00:00 nginx-nginx-firelens-f5c7c43b-71d9-4e99-8609-28f7d4c116e3 {"container_id":"f5c7c43b-71d9-4e99-8609-28f7d4c116e3-2531612879","container_name":"nginx","ecs_cluster":"app-with-nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/f5c7c43b-71d9-4e99-8609-28f7d4c116e3","ecs_task_definition":"app-with-nginx-task-definition:5","log":"/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/","source":"stdout"}
2020-09-17T08:56:19+00:00 nginx-nginx-firelens-f5c7c43b-71d9-4e99-8609-28f7d4c116e3 {"container_id":"f5c7c43b-71d9-4e99-8609-28f7d4c116e3-2531612879","container_name":"nginx","ecs_cluster":"app-with-nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/f5c7c43b-71d9-4e99-8609-28f7d4c116e3","ecs_task_definition":"app-with-nginx-task-definition:5","log":"/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh","source":"stdout"}
2020-09-17T08:56:19+00:00 nginx-nginx-firelens-f5c7c43b-71d9-4e99-8609-28f7d4c116e3 {"container_id":"f5c7c43b-71d9-4e99-8609-28f7d4c116e3-2531612879","container_name":"nginx","ecs_cluster":"app-with-nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/f5c7c43b-71d9-4e99-8609-28f7d4c116e3","ecs_task_definition":"app-with-nginx-task-definition:5","log":"10-listen-on-ipv6-by-default.sh: error: IPv6 listen already enabled","source":"stdout"}
2020-09-17T08:56:19+00:00 nginx-nginx-firelens-f5c7c43b-71d9-4e99-8609-28f7d4c116e3 {"container_id":"f5c7c43b-71d9-4e99-8609-28f7d4c116e3-2531612879","container_name":"nginx","ecs_cluster":"app-with-nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/f5c7c43b-71d9-4e99-8609-28f7d4c116e3","ecs_task_definition":"app-with-nginx-task-definition:5","log":"/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh","source":"stdout"}
2020-09-17T08:56:19+00:00 nginx-nginx-firelens-f5c7c43b-71d9-4e99-8609-28f7d4c116e3 {"container_id":"f5c7c43b-71d9-4e99-8609-28f7d4c116e3-2531612879","container_name":"nginx","ecs_cluster":"app-with-nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/f5c7c43b-71d9-4e99-8609-28f7d4c116e3","ecs_task_definition":"app-with-nginx-task-definition:5","log":"/docker-entrypoint.sh: Configuration complete; ready for start up","source":"stdout"}

〜省略〜

ログストリーム名は、prefix＋タグ＋コンテナIDみたいですね。

Fluent Bit Plugin for CloudWatch Logs / Plugin Options

nginx-nginx-firelens-f5c7c43b-71d9-4e99-8609-28f7d4c116e3

アプリケーションのログ。

$ aws logs tail --follow /fargate/app/app
2020-09-17T08:56:19+00:00 app-app-firelens-f5c7c43b-71d9-4e99-8609-28f7d4c116e3 {"container_id":"f5c7c43b-71d9-4e99-8609-28f7d4c116e3-527074092","container_name":"app","ecs_cluster":"app-with-nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/f5c7c43b-71d9-4e99-8609-28f7d4c116e3","ecs_task_definition":"app-with-nginx-task-definition:5","log":"[2020-09-17 08:56:19 +0000] [1] [INFO] Starting gunicorn 20.0.4","source":"stderr"}
2020-09-17T08:56:19+00:00 app-app-firelens-f5c7c43b-71d9-4e99-8609-28f7d4c116e3 {"container_id":"f5c7c43b-71d9-4e99-8609-28f7d4c116e3-527074092","container_name":"app","ecs_cluster":"app-with-nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/f5c7c43b-71d9-4e99-8609-28f7d4c116e3","ecs_task_definition":"app-with-nginx-task-definition:5","log":"[2020-09-17 08:56:19 +0000] [1] [INFO] Listening at: http://127.0.0.1:8000 (1)","source":"stderr"}
2020-09-17T08:56:19+00:00 app-app-firelens-f5c7c43b-71d9-4e99-8609-28f7d4c116e3 {"container_id":"f5c7c43b-71d9-4e99-8609-28f7d4c116e3-527074092","container_name":"app","ecs_cluster":"app-with-nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/f5c7c43b-71d9-4e99-8609-28f7d4c116e3","ecs_task_definition":"app-with-nginx-task-definition:5","log":"[2020-09-17 08:56:19 +0000] [1] [INFO] Using worker: threads","source":"stderr"}
2020-09-17T08:56:19+00:00 app-app-firelens-f5c7c43b-71d9-4e99-8609-28f7d4c116e3 {"container_id":"f5c7c43b-71d9-4e99-8609-28f7d4c116e3-527074092","container_name":"app","ecs_cluster":"app-with-nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/f5c7c43b-71d9-4e99-8609-28f7d4c116e3","ecs_task_definition":"app-with-nginx-task-definition:5","log":"[2020-09-17 08:56:19 +0000] [9] [INFO] Booting worker with pid: 9","source":"stderr"}
2020-09-17T08:56:21+00:00 app-app-firelens-12fe7680-7b28-4e4f-a6c8-9d946bb0d49e {"container_id":"12fe7680-7b28-4e4f-a6c8-9d946bb0d49e-527074092","container_name":"app","ecs_cluster":"app-with-nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/12fe7680-7b28-4e4f-a6c8-9d946bb0d49e","ecs_task_definition":"app-with-nginx-task-definition:5","log":"[2020-09-17 08:56:21 +0000] [1] [INFO] Starting gunicorn 20.0.4","source":"stderr"}

〜省略〜

AWS for Fluent Bitのログ。こちらは、awslogsロギングドライバーの利用になります。

$ aws logs tail --follow /fargate/app/firelens

起動時のログ。

2020-09-17T08:56:19.100000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 AWS for Fluent Bit Container Image Version 2.7.0
2020-09-17T08:56:19.148000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 Fluent Bit v1.5.6
2020-09-17T08:56:19.148000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 * Copyright (C) 2019-2020 The Fluent Bit Authors
2020-09-17T08:56:19.148000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 * Copyright (C) 2015-2018 Treasure Data
2020-09-17T08:56:19.148000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
2020-09-17T08:56:19.148000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 * https://fluentbit.io
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 0] plugin parameter log_group = '/fargate/app/nginx'"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 0] plugin parameter log_stream_prefix = 'nginx-'"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 0] plugin parameter log_stream_name = ''"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 0] plugin parameter region = 'ap-northeast-1'"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 0] plugin parameter log_key = ''"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 0] plugin parameter role_arn = ''"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 0] plugin parameter new_log_group_tags = ''"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 0] plugin parameter log_retention_days = '0'"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 0] plugin parameter endpoint = ''"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 0] plugin parameter sts_endpoint = ''"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 0] plugin parameter credentials_endpoint = "
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 0] plugin parameter log_format = ''"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 1] plugin parameter log_group = '/fargate/app/app'"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 1] plugin parameter log_stream_prefix = 'app-'"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 1] plugin parameter log_stream_name = ''"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 1] plugin parameter region = 'ap-northeast-1'"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 1] plugin parameter log_key = ''"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 1] plugin parameter role_arn = ''"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 1] plugin parameter new_log_group_tags = ''"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 1] plugin parameter log_retention_days = '0'"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 1] plugin parameter endpoint = ''"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 1] plugin parameter sts_endpoint = ''"
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 1] plugin parameter credentials_endpoint = "
2020-09-17T08:56:19.149000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 time="2020-09-17T08:56:19Z" level=info msg="[cloudwatch 1] plugin parameter log_format = ''"
2020-09-17T08:56:19.150000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 [2020/09/17 08:56:19] [ info] [engine] started (pid=1)
2020-09-17T08:56:19.150000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 [2020/09/17 08:56:19] [ info] [storage] version=1.0.5, initializing...
2020-09-17T08:56:19.150000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 [2020/09/17 08:56:19] [ info] [storage] in-memory
2020-09-17T08:56:19.150000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 [2020/09/17 08:56:19] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
2020-09-17T08:56:19.150000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 [2020/09/17 08:56:19] [ info] [input:tcp:tcp.0] listening on 127.0.0.1:8877
2020-09-17T08:56:19.150000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 [2020/09/17 08:56:19] [ info] [input:forward:forward.1] listening on unix:///var/run/fluent.sock
2020-09-17T08:56:19.150000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 [2020/09/17 08:56:19] [ info] [input:forward:forward.2] listening on 127.0.0.1:24224
2020-09-17T08:56:19.150000+00:00 firelens/log_router/f5c7c43b-71d9-4e99-8609-28f7d4c116e3 [2020/09/17 08:56:19] [ info] [sp] stream processor started

確かに、各コンテナのlogConfigurationの内容がプラグインの設定として渡されているようですね。

msg="[cloudwatch 0] plugin parameter log_group = '/fargate/app/nginx'"
msg="[cloudwatch 0] plugin parameter log_stream_prefix = 'nginx-'"
msg="[cloudwatch 0] plugin parameter log_stream_name = ''"
msg="[cloudwatch 0] plugin parameter region = 'ap-northeast-1'"
msg="[cloudwatch 0] plugin parameter log_key = ''"
msg="[cloudwatch 0] plugin parameter role_arn = ''"
msg="[cloudwatch 0] plugin parameter new_log_group_tags = ''"
msg="[cloudwatch 0] plugin parameter log_retention_days = '0'"
msg="[cloudwatch 0] plugin parameter endpoint = ''"
msg="[cloudwatch 0] plugin parameter sts_endpoint = ''"
msg="[cloudwatch 0] plugin parameter credentials_endpoint = "
msg="[cloudwatch 0] plugin parameter log_format = ''"
msg="[cloudwatch 1] plugin parameter log_group = '/fargate/app/app'"
msg="[cloudwatch 1] plugin parameter log_stream_prefix = 'app-'"
msg="[cloudwatch 1] plugin parameter log_stream_name = ''"
msg="[cloudwatch 1] plugin parameter region = 'ap-northeast-1'"
msg="[cloudwatch 1] plugin parameter log_key = ''"
msg="[cloudwatch 1] plugin parameter role_arn = ''"
msg="[cloudwatch 1] plugin parameter new_log_group_tags = ''"
msg="[cloudwatch 1] plugin parameter log_retention_days = '0'"
msg="[cloudwatch 1] plugin parameter endpoint = ''"
msg="[cloudwatch 1] plugin parameter sts_endpoint = ''"
msg="[cloudwatch 1] plugin parameter credentials_endpoint = "
msg="[cloudwatch 1] plugin parameter log_format = ''"

動作確認はできたのと、ある程度動きはわかったので、今回はこれでOKとしましょう！

VPC〜ALBまで（〜AWS Fargateも）

最後に、省略していたVPCからALBまでの定義を含めた、全体のTerraform定義を載せておきます。

main.tf

terraform {
  required_version = "0.13.3"

  required_providers {
    aws = "3.6.0"
  }
}

provider "aws" {
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "2.51.0"

  name = "my-vpc"
  cidr = "10.0.0.0/16"

  enable_dns_hostnames = true
  enable_dns_support   = true

  azs             = ["ap-northeast-1a", "ap-northeast-1c"]
  public_subnets  = ["10.0.10.0/24", "10.0.20.0/24"]
  private_subnets = ["10.0.30.0/24", "10.0.40.0/24"]

  map_public_ip_on_launch = true

  enable_nat_gateway     = true
  single_nat_gateway     = false
  one_nat_gateway_per_az = true
}

module "load_balancer_sg" {
  source  = "terraform-aws-modules/security-group/aws//modules/http-80"
  version = "3.16.0"

  name   = "load-balancer-sg"
  vpc_id = module.vpc.vpc_id

  ingress_cidr_blocks = ["0.0.0.0/0"]
}

module "app_with_nginx_service_sg" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "3.16.0"

  name   = "app-with-nginx-service-sg"
  vpc_id = module.vpc.vpc_id

  ingress_with_cidr_blocks = [
    {
      from_port   = 80
      to_port     = 80
      protocol    = "tcp"
      description = "app-with-nginx-service inbound ports"
      cidr_blocks = "10.0.10.0/24"
    },
    {
      from_port   = 80
      to_port     = 80
      protocol    = "tcp"
      description = "app-with-nginx-service inbound ports"
      cidr_blocks = "10.0.20.0/24"
    }
  ]

  egress_with_cidr_blocks = [
    {
      from_port   = 0
      to_port     = 0
      protocol    = "-1"
      description = "app-with-nginx-service outbound ports"
      cidr_blocks = "0.0.0.0/0"
    }
  ]
}

module "load_balancer" {
  source  = "terraform-aws-modules/alb/aws"
  version = "5.9.0"

  name = "app-with-nginx"

  vpc_id             = module.vpc.vpc_id
  load_balancer_type = "application"
  internal           = false

  subnets         = module.vpc.public_subnets
  security_groups = [module.load_balancer_sg.this_security_group_id]

  target_groups = [
    {
      backend_protocol = "HTTP"
      backend_port     = 80
      target_type      = "ip"

      health_check = {
        interval = 20
      }
    }
  ]

  http_tcp_listeners = [
    {
      port     = 80
      protocol = "HTTP"
    }
  ]
}

locals {
  vpc_id = module.vpc.vpc_id

  private_subnets = module.vpc.private_subnets
  app_with_nginx_service_security_groups = [
  module.app_with_nginx_service_sg.this_security_group_id]
  load_balancer_target_group_arn = module.load_balancer.target_group_arns[0]
}

data "aws_iam_policy_document" "assume_role" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["ecs-tasks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "ecs_task_execution_role" {
  name               = "MyEcsTaskExecutionRole"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

resource "aws_iam_role_policy_attachment" "amazon_ecs_task_execution_role_policy" {
  role       = aws_iam_role.ecs_task_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

data "aws_iam_policy_document" "ecs_task_role_policy_document" {
  statement {
    effect = "Allow"

    actions = [
      "logs:DescribeLogStreams",
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]

    resources = ["*"]
  }
}

resource "aws_iam_policy" "ecs_task_role_policy" {
  name   = "MyEcsTaskPolicy"
  policy = data.aws_iam_policy_document.ecs_task_role_policy_document.json
}

resource "aws_iam_role" "ecs_task_role" {
  name               = "MyEcsTaskRole"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

resource "aws_iam_role_policy_attachment" "ecs_task_role_policy_attachment" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = aws_iam_policy.ecs_task_role_policy.arn
}

resource "aws_cloudwatch_log_group" "nginx_container_log_group" {
  name = "/fargate/app/nginx"
}

resource "aws_cloudwatch_log_group" "app_container_log_group" {
  name = "/fargate/app/app"
}

resource "aws_cloudwatch_log_group" "firelens_container_log_group" {
  name = "/fargate/app/firelens"
}

resource "aws_ecs_cluster" "app_with_nginx" {
  name = "app-with-nginx-cluster"
}

resource "aws_ecs_task_definition" "app_with_nginx" {
  family                   = "app-with-nginx-task-definition"
  cpu                      = "1024"
  memory                   = "2048"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn            = aws_iam_role.ecs_task_role.arn

  container_definitions = <<JSON
  [
    {
      "name": "nginx",
      "image": "[AWSアカウントID].dkr.ecr.ap-northeast-1.amazonaws.com/nginx:latest",
      "essential": true,
      "portMappings": [
        {
          "protocol": "tcp",
          "containerPort": 80
        }
      ],
      "cpu": 256,
      "memory": 512,
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch",
          "region": "ap-northeast-1",
          "log_group_name": "/fargate/app/nginx",
          "log_stream_prefix": "nginx-",
          "auto_create_group": "false"
        }
      }
    },
    {
      "name": "app",
      "image": "[AWSアカウントID].dkr.ecr.ap-northeast-1.amazonaws.com/app:latest",
      "essential": true,
      "cpu": 256,
      "memory": 512,
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch",
          "region": "ap-northeast-1",
          "log_group_name": "/fargate/app/app",
          "log_stream_prefix": "app-",
          "auto_create_group": "false"
        }
      }
    },
    {
      "name": "log_router",
      "image": "906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:latest",
      "essential": true,
      "cpu": 256,
      "memory": 512,
      "firelensConfiguration": {
        "type": "fluentbit"
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/fargate/app/firelens",
          "awslogs-region": "ap-northeast-1",
          "awslogs-stream-prefix": "firelens"
        }
      }
    }
  ]
  JSON
}

resource "aws_ecs_service" "app_with_nginx" {
  name             = "app-with-nginx-service"
  cluster          = aws_ecs_cluster.app_with_nginx.arn
  task_definition  = aws_ecs_task_definition.app_with_nginx.arn
  desired_count    = 3
  launch_type      = "FARGATE"
  platform_version = "1.4.0"

  deployment_minimum_healthy_percent = 50

  network_configuration {
    assign_public_ip = false
    security_groups  = local.app_with_nginx_service_security_groups
    subnets          = local.private_subnets
  }

  load_balancer {
    target_group_arn = local.load_balancer_target_group_arn
    container_name   = "nginx"
    container_port   = 80
  }
}