AWS FireLensを使ってログ出力した時に、ログに付与されるAmazon ECSメタデータを確認する

What's？

Amazon ECSで利用できるログドライバー、AWS FireLensを使用するとAmazon ECSメタデータを付与できるそうです。

こちらの内容を、ちょっと確認してみたいなと思いまして。

AWS FireLensを使った時に付与されるメタデータ

AWS FireLensを使った時に付与されるメタデータは、以下のドキュメントに記載があります。

FireLens 設定を使用するタスク定義の作成 / Amazon ECS メタデータの使用

タスク定義においてAWS FireLensとして使うコンテナのenable-ecs-log-metadataをtrueにすると、以下のフィールドをログに付与するとあります。

• ecs_cluster – タスクが所属するクラスターの名前。
• ecs_task_arn – コンテナが所属するタスクの完全な ARN。
• ecs_task_definition – タスクが使用しているタスク定義名とリビジョン。

これが、Amazon ECSのメタデータです、と。

今回は、こちらの内容を実際に見てみたいと思います。

お題

なにはともあれAmazon ECSが必要なので、今回はnginxコンテナを使ったAWS Fargateクラスタを構築したいと思います。

AWS Fargateのタスク定義は、以下の2種類を試すことにします。

• nginxコンテナのみをタスク定義に含め、ログはawslogsログドライバーを使用してAmazon CloudWatch Logsに出力
• nginxコンテナおよびAWS for Fluent Bitコンテナを使い、nginxのログはfirelensログドライバーを使用してAmazon CloudWatch Logsに出力

環境は、すべてTerraformで構築するものとします。

環境

今回の環境は、こちらです。

$ terraform version
Terraform v0.14.7
+ provider registry.terraform.io/hashicorp/aws v3.29.0

また、AWSのクレデンシャルは環境変数で設定済みとします。

$ export AWS_ACCESS_KEY_ID=...
$ export AWS_SECRET_ACCESS_KEY=...
$ export AWS_DEFAULT_REGION=ap-northeast-1

AWS Fargateクラスタのリソース定義を行う

それでは、AWS Fargateクラスタを構築していきましょう。

Terraformでデプロイするわけですが、今回は単一のmain.tfに全部書いていきます。

まずは、TerraformやProviderのバージョン指定。

terraform {
  required_version = "0.14.7"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.29.0"
    }
  }
}

provider "aws" {
}

VPCやALB等も必要ですが、先にAWS Fargateに関する部分だけを載せて、その他のものは最後にまとめて載せたいと思います。

とはいえ、AWS Fargateを作るにもこれらのリソースに依存する部分はあるので、そのあたりは先にローカル変数にまとめておきます。

locals {
  vpc_id = module.vpc.vpc_id

  private_subnets                = module.vpc.private_subnets
  nginx_service_security_groups  = [module.nginx_service_sg.this_security_group_id]
  load_balancer_target_group_arn = module.load_balancer.target_group_arns[0]

  nginx_simple_container_definition = # nginxのみのタスク定義

  nginx_with_fluentbit_container_definitions = # nginx＋Fluent Bitのコンテナ定義
}

コメントアウトしている部分が、タスク定義で使用するコンテナの定義です。今回は、この2つ定義を切り替えてデプロイしていきたいと思います。

タスク定義に至るまでの部分を載せていきましょう。

タスク用ロールおよびタスク実行用ロール。

data "aws_iam_policy_document" "ecs_assume_role" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["ecs-tasks.amazonaws.com"]
    }
  }
}

data "aws_iam_policy" "ecs_task_execution_role_policy" {
  arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

resource "aws_iam_role" "ecs_task_execution_role" {
  name               = "MyEcsTaskExecutionRole"
  assume_role_policy = data.aws_iam_policy_document.ecs_assume_role.json
}

resource "aws_iam_role_policy_attachment" "ecs_task_execution_role_policy_attachment" {
  role       = aws_iam_role.ecs_task_execution_role.name
  policy_arn = data.aws_iam_policy.ecs_task_execution_role_policy.arn
}

data "aws_iam_policy_document" "ecs_task_role_policy_document" {
  statement {
    effect = "Allow"

    actions = [
      "logs:DescribeLogStreams",
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]

    resources = ["*"]
  }
}

resource "aws_iam_policy" "ecs_task_role_policy" {
  name   = "MyEcsTaskPolicy"
  policy = data.aws_iam_policy_document.ecs_task_role_policy_document.json
}

resource "aws_iam_role" "ecs_task_role" {
  name               = "MyEcsTaskRole"
  assume_role_policy = data.aws_iam_policy_document.ecs_assume_role.json
}

resource "aws_iam_role_policy_attachment" "ecs_task_role_policy_attachment" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = aws_iam_policy.ecs_task_role_policy.arn
}

タスク用ロールには、AWS for Fluent BitがAmazon CloudWatch Logsにログ出力可能なように、あらかじめAmazon CloudWatch Logs関連の権限を与えておきます。

Amazon CloudWatch Logsロググループ。各コンテナのログ出力先ですね。

AWS Fargateのクラスタ定義。

resource "aws_cloudwatch_log_group" "nginx" {
  name = "/fargate/containers/nginx"
}

resource "aws_cloudwatch_log_group" "fluentbit" {
  name = "/fargate/containers/fluentbit"
}

resource "aws_ecs_cluster" "nginx" {
  name = "nginx-cluster"
}

resource "aws_ecs_task_definition" "nginx" {
  family       = "nginx-task-definition"
  cpu          = "512"
  memory       = "1024"
  network_mode = "awsvpc"
  requires_compatibilities = [
  "FARGATE"]
  execution_role_arn = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn      = aws_iam_role.ecs_task_role.arn

  container_definitions = local.nginx_simple_container_definition
  # container_definitions = local.nginx_with_fluentbit_container_definitions
}

resource "aws_ecs_service" "nginx" {
  name             = "nginx-service"
  cluster          = aws_ecs_cluster.nginx.arn
  task_definition  = aws_ecs_task_definition.nginx.arn
  desired_count    = 3
  launch_type      = "FARGATE"
  platform_version = "1.4.0"

  deployment_minimum_healthy_percent = 50

  network_configuration {
    assign_public_ip = false
    security_groups  = local.nginx_service_security_groups
    subnets          = local.private_subnets
  }

  load_balancer {
    target_group_arn = local.load_balancer_target_group_arn
    container_name   = "nginx"
    container_port   = 80
  }
}

ここからは、container_definitionsの部分を切り替えていきます。

resource "aws_ecs_task_definition" "nginx" {
  family       = "nginx-task-definition"
  cpu          = "512"
  memory       = "1024"
  network_mode = "awsvpc"
  requires_compatibilities = [
  "FARGATE"]
  execution_role_arn = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn      = aws_iam_role.ecs_task_role.arn

  container_definitions = local.nginx_simple_container_definition
  # container_definitions = local.nginx_with_fluentbit_container_definitions
}

AWS FireLensなしで構築してみる

最初は、AWS FireLensなしで構築してみましょう。

タスク定義で使うコンテナの定義は、こちらになります。

  nginx_simple_container_definition = <<JSON
    [
      {
        "name": "nginx",
        "image": "nginx:1.19.7",
        "essential": true,
        "portMappings": [
          {
            "protocol": "tcp",
            "containerPort": 80
          }
        ],
        "logConfiguration": {
          "logDriver": "awslogs",
          "options": {
            "awslogs-group": "${aws_cloudwatch_log_group.nginx.name}",
            "awslogs-region": "ap-northeast-1",
            "awslogs-stream-prefix": "nginx-log-stream"
          }
        }
      }
    ]
    JSON

至ってシンプルです。ログ出力先はAmazon CloudWatch Logsで、先ほどリソース定義したロググループを指定しています。

では、このコンテナ定義をタスク定義で指定して

resource "aws_ecs_task_definition" "nginx" {
  family       = "nginx-task-definition"
  cpu          = "512"
  memory       = "1024"
  network_mode = "awsvpc"
  requires_compatibilities = [
  "FARGATE"]
  execution_role_arn = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn      = aws_iam_role.ecs_task_role.arn

  container_definitions = local.nginx_simple_container_definition
  # container_definitions = local.nginx_with_fluentbit_container_definitions
}

apply。

$ terraform apply

ロググループをtailして待ち、

$ aws logs tail --follow /fargate/containers/nginx

クラスタが構築されてコンテナが起動したら、curlでアクセスしてみます

$ curl [ALBのDNS名]
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

出力されるログは、こんな感じです。

※アクセス元のIPアドレスは変えています

2021-02-22T12:47:34.424000+00:00 nginx-log-stream/nginx/df733e0802104d3890ebaad461c8d7cf 10.0.10.209 - - [22/Feb/2021:12:47:34 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "aaa.bbb.ccc.ddd"
2021-02-22T12:47:35.754000+00:00 nginx-log-stream/nginx/df733e0802104d3890ebaad461c8d7cf 10.0.10.209 - - [22/Feb/2021:12:47:35 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "aaa.bbb.ccc.ddd"
2021-02-22T12:47:37.315000+00:00 nginx-log-stream/nginx/df733e0802104d3890ebaad461c8d7cf 10.0.10.209 - - [22/Feb/2021:12:47:37 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "aaa.bbb.ccc.ddd"
2021-02-22T12:47:41.261000+00:00 nginx-log-stream/nginx/163a9d4c115f4951b5cac39ef16fe60e 10.0.10.209 - - [22/Feb/2021:12:47:41 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "aaa.bbb.ccc.ddd"
2021-02-22T12:47:43.301000+00:00 nginx-log-stream/nginx/df733e0802104d3890ebaad461c8d7cf 10.0.10.209 - - [22/Feb/2021:12:47:43 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "aaa.bbb.ccc.ddd"
2021-02-22T12:47:45.343000+00:00 nginx-log-stream/nginx/df733e0802104d3890ebaad461c8d7cf 10.0.10.209 - - [22/Feb/2021:12:47:45 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "aaa.bbb.ccc.ddd"
2021-02-22T12:47:47.377000+00:00 nginx-log-stream/nginx/163a9d4c115f4951b5cac39ef16fe60e 10.0.10.209 - - [22/Feb/2021:12:47:47 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "aaa.bbb.ccc.ddd"
2021-02-22T12:47:49.408000+00:00 nginx-log-stream/nginx/df733e0802104d3890ebaad461c8d7cf 10.0.10.209 - - [22/Feb/2021:12:47:49 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "aaa.bbb.ccc.ddd"
2021-02-22T12:47:51.435000+00:00 nginx-log-stream/nginx/7b8175f48e1244f1a41912f4af179129 10.0.10.209 - - [22/Feb/2021:12:47:51 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "aaa.bbb.ccc.ddd"
2021-02-22T12:47:53.196000+00:00 nginx-log-stream/nginx/df733e0802104d3890ebaad461c8d7cf 10.0.10.209 - - [22/Feb/2021:12:47:53 +0000] "GET / HTTP/1.1" 200 612 "-" "ELB-HealthChecker/2.0" "-"
2021-02-22T12:47:53.196000+00:00 nginx-log-stream/nginx/7b8175f48e1244f1a41912f4af179129 10.0.10.209 - - [22/Feb/2021:12:47:53 +0000] "GET / HTTP/1.1" 200 612 "-" "ELB-HealthChecker/2.0" "-"
2021-02-22T12:47:53.468000+00:00 nginx-log-stream/nginx/163a9d4c115f4951b5cac39ef16fe60e 10.0.10.209 - - [22/Feb/2021:12:47:53 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "aaa.bbb.ccc.ddd"
2021-02-22T12:47:55.500000+00:00 nginx-log-stream/nginx/163a9d4c115f4951b5cac39ef16fe60e 10.0.10.209 - - [22/Feb/2021:12:47:55 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "aaa.bbb.ccc.ddd"
2021-02-22T12:47:57.541000+00:00 nginx-log-stream/nginx/7b8175f48e1244f1a41912f4af179129 10.0.10.209 - - [22/Feb/2021:12:47:57 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "aaa.bbb.ccc.ddd"

ログとして記録されている値だけを抜粋すると、こんな感じですね。

 10.0.10.209 - - [22/Feb/2021:12:47:34 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "aaa.bbb.ccc.ddd"

ログの形式は確認できたので、リソースを破棄します。

$ terraform destroy

AWS FireLens経由でログ出力する

続いて、AWS FireLens経由でログ出力しましょう。

AWS Fargateのタスク定義は、以下のようになります。

resource "aws_ecs_task_definition" "nginx" {
  family       = "nginx-task-definition"
  cpu          = "512"
  memory       = "1024"
  network_mode = "awsvpc"
  requires_compatibilities = [
  "FARGATE"]
  execution_role_arn = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn      = aws_iam_role.ecs_task_role.arn

  # container_definitions = local.nginx_simple_container_definition
  container_definitions = local.nginx_with_fluentbit_container_definitions
}

参照する実際のcontainer_definitionsの定義（コンテナ定義）ですが、Amazon ECSメタデータを付与する／しない設定ができるので、その両方で確認してみましょう。

Amazon ECSメタデータ付与あり

Amazon ECSのメタデータを付与する場合のコンテナ定義です。AWS for Fluent Bitと、nginxの2つのコンテナ定義があります。

  nginx_with_fluentbit_container_definitions = <<JSON
  [
    {
      "name": "log_router",
      "image": "906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.10.1",
      "essential": true,
      "firelensConfiguration": {
        "type": "fluentbit",
        "options":{
           "enable-ecs-log-metadata": "true"
        }
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "${aws_cloudwatch_log_group.fluentbit.name}",
          "awslogs-region": "ap-northeast-1",
          "awslogs-stream-prefix": "fluentbit-log-stream"
        }
      }
    },
    {
      "name": "nginx",
      "image": "nginx:1.19.7",
      "essential": true,
      "portMappings": [
        {
          "protocol": "tcp",
          "containerPort": 80
        }
      ],
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch",
          "region": "ap-northeast-1",
          "log_group_name": "${aws_cloudwatch_log_group.nginx.name}",
          "log_stream_prefix": "nginx-log-stream-",
          "auto_create_group": "false"
        }
      }
    }
  ]
    JSON

firelensConfigurationのオプション設定として、enable-ecs-log-metadataをtrueに設定しています。これでAWS FireLens経由で出力するログにAmazon ECSのメタデータを付与することができるわけですが、これはデフォルトの設定です。

      "firelensConfiguration": {
        "type": "fluentbit",
        "options":{
           "enable-ecs-log-metadata": "true"
        }
      },

なので、以下のようにしても同義ですね。

      "firelensConfiguration": {
        "type": "fluentbit"
      },

この状態でapplyして

$ terraform apply

先ほどと同様、ロググループをtailしつつリソースの構築後にcurlでアクセスしてみます。

$ aws logs tail --follow /fargate/containers/nginx

得られたログはこちらです。

2021-02-22T13:01:17.712000+00:00 nginx-log-stream-nginx-firelens-d90128b0816e4a38b34888f721bf60fb {"container_id":"d90128b0816e4a38b34888f721bf60fb-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/d90128b0816e4a38b34888f721bf60fb","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:17 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:01:19.759000+00:00 nginx-log-stream-nginx-firelens-d90128b0816e4a38b34888f721bf60fb {"container_id":"d90128b0816e4a38b34888f721bf60fb-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/d90128b0816e4a38b34888f721bf60fb","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:19 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:01:20.292000+00:00 nginx-log-stream-nginx-firelens-d90128b0816e4a38b34888f721bf60fb {"container_id":"d90128b0816e4a38b34888f721bf60fb-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/d90128b0816e4a38b34888f721bf60fb","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:20 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"ELB-HealthChecker/2.0\" \"-\"","source":"stdout"}
2021-02-22T13:01:21.790000+00:00 nginx-log-stream-nginx-firelens-e07924a4952c4e36844603a88819b4ce {"container_id":"e07924a4952c4e36844603a88819b4ce-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/e07924a4952c4e36844603a88819b4ce","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:21 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:01:23.825000+00:00 nginx-log-stream-nginx-firelens-d90128b0816e4a38b34888f721bf60fb {"container_id":"d90128b0816e4a38b34888f721bf60fb-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/d90128b0816e4a38b34888f721bf60fb","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:23 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:01:25.862000+00:00 nginx-log-stream-nginx-firelens-d90128b0816e4a38b34888f721bf60fb {"container_id":"d90128b0816e4a38b34888f721bf60fb-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/d90128b0816e4a38b34888f721bf60fb","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:25 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:01:27.899000+00:00 nginx-log-stream-nginx-firelens-d90128b0816e4a38b34888f721bf60fb {"container_id":"d90128b0816e4a38b34888f721bf60fb-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/d90128b0816e4a38b34888f721bf60fb","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:27 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:01:29.937000+00:00 nginx-log-stream-nginx-firelens-e07924a4952c4e36844603a88819b4ce {"container_id":"e07924a4952c4e36844603a88819b4ce-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/e07924a4952c4e36844603a88819b4ce","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:29 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:01:31.975000+00:00 nginx-log-stream-nginx-firelens-e07924a4952c4e36844603a88819b4ce {"container_id":"e07924a4952c4e36844603a88819b4ce-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/e07924a4952c4e36844603a88819b4ce","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:31 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:01:34.013000+00:00 nginx-log-stream-nginx-firelens-d6bcefeb9a014f17bc444b18f2f0d487 {"container_id":"d6bcefeb9a014f17bc444b18f2f0d487-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/d6bcefeb9a014f17bc444b18f2f0d487","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:34 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:01:36.049000+00:00 nginx-log-stream-nginx-firelens-e07924a4952c4e36844603a88819b4ce {"container_id":"e07924a4952c4e36844603a88819b4ce-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/e07924a4952c4e36844603a88819b4ce","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:36 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:01:38.093000+00:00 nginx-log-stream-nginx-firelens-d6bcefeb9a014f17bc444b18f2f0d487 {"container_id":"d6bcefeb9a014f17bc444b18f2f0d487-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/d6bcefeb9a014f17bc444b18f2f0d487","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:38 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:01:40.308000+00:00 nginx-log-stream-nginx-firelens-d6bcefeb9a014f17bc444b18f2f0d487 {"container_id":"d6bcefeb9a014f17bc444b18f2f0d487-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/d6bcefeb9a014f17bc444b18f2f0d487","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:40 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"ELB-HealthChecker/2.0\" \"-\"","source":"stdout"}
2021-02-22T13:01:40.313000+00:00 nginx-log-stream-nginx-firelens-d90128b0816e4a38b34888f721bf60fb {"container_id":"d90128b0816e4a38b34888f721bf60fb-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/d90128b0816e4a38b34888f721bf60fb","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:40 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"ELB-HealthChecker/2.0\" \"-\"","source":"stdout"}
2021-02-22T13:01:42.168000+00:00 nginx-log-stream-nginx-firelens-d6bcefeb9a014f17bc444b18f2f0d487 {"container_id":"d6bcefeb9a014f17bc444b18f2f0d487-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/d6bcefeb9a014f17bc444b18f2f0d487","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:42 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}

先ほどとは、ログの形式が変わりましたね。

1行抜き出して、整形してみましょう。

$ echo '{"container_id":"d90128b0816e4a38b34888f721bf60fb-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/d90128b0816e4a38b34888f721bf60fb","ecs_task_definition":"nginx-task-definition:8","log":"10.0.10.190 - - [22/Feb/2021:13:01:17 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}' | jq
{
  "container_id": "d90128b0816e4a38b34888f721bf60fb-2531612879",
  "container_name": "nginx",
  "ecs_cluster": "nginx-cluster",
  "ecs_task_arn": "arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/d90128b0816e4a38b34888f721bf60fb",
  "ecs_task_definition": "nginx-task-definition:8",
  "log": "10.0.10.190 - - [22/Feb/2021:13:01:17 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"",
  "source": "stdout"
}

ログがJSON形式になり、container_id、container_name、ecs_cluster、ecs_task_arn、ecs_task_definition、sourceの6つのフィールドが増えました。

本来のログは、logフィールドに格納されています。

おや？sourceはいいとしても、付与されるフィールドは3つだったのでは？Amazon ECSメタデータの付与もオフにして確認してみましょう。

※ちなみにAWS for Fluent Bit自身のログは、単にawslogsでAmazon CloudWatch Logsに出力しているだけなのでAWS FireLensと使わない場合と同じになります（よって割愛）

リソースを破棄します。

$ terraform destroy

Amazon ECSメタデータ付与なし

先ほどのコンテナ定義から、Amazon ECSのメタデータを付与しないように変更してみます。

  nginx_with_fluentbit_container_definitions = <<JSON
  [
    {
      "name": "log_router",
      "image": "906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.10.1",
      "essential": true,
      "firelensConfiguration": {
        "type": "fluentbit",
        "options":{
           "enable-ecs-log-metadata": "false"
        }
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "${aws_cloudwatch_log_group.fluentbit.name}",
          "awslogs-region": "ap-northeast-1",
          "awslogs-stream-prefix": "fluentbit-log-stream"
        }
      }
    },
    {
      "name": "nginx",
      "image": "nginx:1.19.7",
      "essential": true,
      "portMappings": [
        {
          "protocol": "tcp",
          "containerPort": 80
        }
      ],
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch",
          "region": "ap-northeast-1",
          "log_group_name": "${aws_cloudwatch_log_group.nginx.name}",
          "log_stream_prefix": "nginx-log-stream-",
          "auto_create_group": "false"
        }
      }
    }
  ]
    JSON

変更したのは、

      "firelensConfiguration": {
        "type": "fluentbit",
        "options":{
           "enable-ecs-log-metadata": "false"
        }
      },

この状態でapplyして

$ terraform apply

先ほどと同様、ロググループをtailしつつリソースの構築後にcurlでアクセスしてみます。

$ aws logs tail --follow /fargate/containers/nginx

得られたログはこちらです。

2021-02-22T13:39:14.048000+00:00 nginx-log-stream-nginx-firelens-37d9cebd8fbb4d57bd7fa646323e501a {"container_id":"37d9cebd8fbb4d57bd7fa646323e501a-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:14 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:39:16.090000+00:00 nginx-log-stream-nginx-firelens-37d9cebd8fbb4d57bd7fa646323e501a {"container_id":"37d9cebd8fbb4d57bd7fa646323e501a-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:16 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:39:18.137000+00:00 nginx-log-stream-nginx-firelens-37d9cebd8fbb4d57bd7fa646323e501a {"container_id":"37d9cebd8fbb4d57bd7fa646323e501a-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:18 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:39:20.287000+00:00 nginx-log-stream-nginx-firelens-37d9cebd8fbb4d57bd7fa646323e501a {"container_id":"37d9cebd8fbb4d57bd7fa646323e501a-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:20 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:39:22.330000+00:00 nginx-log-stream-nginx-firelens-37d9cebd8fbb4d57bd7fa646323e501a {"container_id":"37d9cebd8fbb4d57bd7fa646323e501a-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:22 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:39:24.369000+00:00 nginx-log-stream-nginx-firelens-9e9a09f1b8924ce3926c7c6653bd3ec1 {"container_id":"9e9a09f1b8924ce3926c7c6653bd3ec1-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:24 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:39:26.408000+00:00 nginx-log-stream-nginx-firelens-37d9cebd8fbb4d57bd7fa646323e501a {"container_id":"37d9cebd8fbb4d57bd7fa646323e501a-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:26 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:39:28.447000+00:00 nginx-log-stream-nginx-firelens-37d9cebd8fbb4d57bd7fa646323e501a {"container_id":"37d9cebd8fbb4d57bd7fa646323e501a-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:28 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:39:28.483000+00:00 nginx-log-stream-nginx-firelens-9e9a09f1b8924ce3926c7c6653bd3ec1 {"container_id":"9e9a09f1b8924ce3926c7c6653bd3ec1-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:28 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"ELB-HealthChecker/2.0\" \"-\"","source":"stdout"}
2021-02-22T13:39:28.485000+00:00 nginx-log-stream-nginx-firelens-37d9cebd8fbb4d57bd7fa646323e501a {"container_id":"37d9cebd8fbb4d57bd7fa646323e501a-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:28 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"ELB-HealthChecker/2.0\" \"-\"","source":"stdout"}
2021-02-22T13:39:28.485000+00:00 nginx-log-stream-nginx-firelens-e0f54e9a04cd4d59be03b4ae47c6c690 {"container_id":"e0f54e9a04cd4d59be03b4ae47c6c690-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:28 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"ELB-HealthChecker/2.0\" \"-\"","source":"stdout"}
2021-02-22T13:39:30.488000+00:00 nginx-log-stream-nginx-firelens-9e9a09f1b8924ce3926c7c6653bd3ec1 {"container_id":"9e9a09f1b8924ce3926c7c6653bd3ec1-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:30 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:39:32.530000+00:00 nginx-log-stream-nginx-firelens-37d9cebd8fbb4d57bd7fa646323e501a {"container_id":"37d9cebd8fbb4d57bd7fa646323e501a-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:32 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T13:39:34.569000+00:00 nginx-log-stream-nginx-firelens-9e9a09f1b8924ce3926c7c6653bd3ec1 {"container_id":"9e9a09f1b8924ce3926c7c6653bd3ec1-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:34 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}

1行抜き出して、先ほどと同じように整形してみます。

$ echo '{"container_id":"37d9cebd8fbb4d57bd7fa646323e501a-2531612879","container_name":"nginx","log":"10.0.10.227 - - [22/Feb/2021:13:39:14 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}' | jq
{
  "container_id": "37d9cebd8fbb4d57bd7fa646323e501a-2531612879",
  "container_name": "nginx",
  "log": "10.0.10.227 - - [22/Feb/2021:13:39:14 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"",
  "source": "stdout"
}

Amazon ECSのメタデータの付与をオフにしても、AWS FireLensを通すとcontainer_idとcontainer_nameは追加されるんですね。

まとめ

ここまでの結果をまとめると

• AWS FireLensを通してログ出力すると、JSON形式になり、少なくともcontainer_id、container_nameおよびログの出力元（sourceが追加される）
• enable-ecs-log-metadataをtrueにすると、ecs_cluster、ecs_task_arn、ecs_task_definitionの3つが追加される（これはデフォルトの動作）

ということですね。

単にAmazon ECSのログをAmazon CloudWatch Logsに保存しておくだけの場合は、どのタスクがどのロググループにログを保存しているかはリソース定義を見ればわかるのでそれほど意味がないかもしれません。

このログを、さらに他の場所に集約したりする場合には、これらのメタデータが有効になってくるのでしょうね。

オマケ

ここまで省略していたVPCやALB等を含む、全体のリソース定義を載せておきます。

main.tf

terraform {
  required_version = "0.14.7"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.29.0"
    }
  }
}

provider "aws" {
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "2.71.0"

  name = "my-vpc"
  cidr = "10.0.0.0/16"

  enable_dns_hostnames = true
  enable_dns_support   = true

  azs             = ["ap-northeast-1a", "ap-northeast-1c"]
  public_subnets  = ["10.0.10.0/24", "10.0.20.0/24"]
  private_subnets = ["10.0.30.0/24", "10.0.40.0/24"]

  map_public_ip_on_launch = false

  enable_nat_gateway     = true
  single_nat_gateway     = false
  one_nat_gateway_per_az = false
}

module "load_balancer_sg" {
  source  = "terraform-aws-modules/security-group/aws//modules/http-80"
  version = "3.18.0"

  name   = "load-balancer-sg"
  vpc_id = module.vpc.vpc_id

  ingress_cidr_blocks = ["0.0.0.0/0"]
}

module "nginx_service_sg" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "3.18.0"

  name   = "nginx-service-sg"
  vpc_id = module.vpc.vpc_id

  ingress_with_cidr_blocks = [
    {
      from_port   = 80
      to_port     = 80
      protocol    = "tcp"
      description = "nginx-service inbound ports"
      cidr_blocks = "10.0.10.0/24"
    },
    {
      from_port   = 80
      to_port     = 80
      protocol    = "tcp"
      description = "nginx-service inbound ports"
      cidr_blocks = "10.0.20.0/24"
    }
  ]

  egress_with_cidr_blocks = [
    {
      from_port   = 0
      to_port     = 0
      protocol    = "-1"
      description = "nginx-service outbound ports"
      cidr_blocks = "0.0.0.0/0"
    }
  ]
}

module "load_balancer" {
  source  = "terraform-aws-modules/alb/aws"
  version = "5.11.0"

  name = "nginx-alb"

  vpc_id             = module.vpc.vpc_id
  load_balancer_type = "application"
  internal           = false

  subnets         = module.vpc.public_subnets
  security_groups = [module.load_balancer_sg.this_security_group_id]

  target_groups = [
    {
      backend_protocol = "HTTP"
      backend_port     = 80
      target_type      = "ip"

      health_check = {
        interval = 20
      }
    }
  ]

  http_tcp_listeners = [
    {
      port     = 80
      protocol = "HTTP"
    }
  ]
}

locals {
  vpc_id = module.vpc.vpc_id

  private_subnets                = module.vpc.private_subnets
  nginx_service_security_groups  = [module.nginx_service_sg.this_security_group_id]
  load_balancer_target_group_arn = module.load_balancer.target_group_arns[0]

  nginx_simple_container_definition = <<JSON
    [
      {
        "name": "nginx",
        "image": "nginx:1.19.7",
        "essential": true,
        "portMappings": [
          {
            "protocol": "tcp",
            "containerPort": 80
          }
        ],
        "logConfiguration": {
          "logDriver": "awslogs",
          "options": {
            "awslogs-group": "${aws_cloudwatch_log_group.nginx.name}",
            "awslogs-region": "ap-northeast-1",
            "awslogs-stream-prefix": "nginx-log-stream"
          }
        }
      }
    ]
    JSON

  nginx_with_fluentbit_container_definitions = <<JSON
  [
    {
      "name": "log_router",
      "image": "906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.10.1",
      "essential": true,
      "firelensConfiguration": {
        "type": "fluentbit",
        "options":{
           "enable-ecs-log-metadata": "true"
        }
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "${aws_cloudwatch_log_group.fluentbit.name}",
          "awslogs-region": "ap-northeast-1",
          "awslogs-stream-prefix": "fluentbit-log-stream"
        }
      }
    },
    {
      "name": "nginx",
      "image": "nginx:1.19.7",
      "essential": true,
      "portMappings": [
        {
          "protocol": "tcp",
          "containerPort": 80
        }
      ],
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch",
          "region": "ap-northeast-1",
          "log_group_name": "${aws_cloudwatch_log_group.nginx.name}",
          "log_stream_prefix": "nginx-log-stream-",
          "auto_create_group": "false"
        }
      }
    }
  ]
    JSON
}

data "aws_iam_policy_document" "ecs_assume_role" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["ecs-tasks.amazonaws.com"]
    }
  }
}

data "aws_iam_policy" "ecs_task_execution_role_policy" {
  arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

resource "aws_iam_role" "ecs_task_execution_role" {
  name               = "MyEcsTaskExecutionRole"
  assume_role_policy = data.aws_iam_policy_document.ecs_assume_role.json
}

resource "aws_iam_role_policy_attachment" "ecs_task_execution_role_policy_attachment" {
  role       = aws_iam_role.ecs_task_execution_role.name
  policy_arn = data.aws_iam_policy.ecs_task_execution_role_policy.arn
}

data "aws_iam_policy_document" "ecs_task_role_policy_document" {
  statement {
    effect = "Allow"

    actions = [
      "logs:DescribeLogStreams",
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]

    resources = ["*"]
  }
}

resource "aws_iam_policy" "ecs_task_role_policy" {
  name   = "MyEcsTaskPolicy"
  policy = data.aws_iam_policy_document.ecs_task_role_policy_document.json
}

resource "aws_iam_role" "ecs_task_role" {
  name               = "MyEcsTaskRole"
  assume_role_policy = data.aws_iam_policy_document.ecs_assume_role.json
}

resource "aws_iam_role_policy_attachment" "ecs_task_role_policy_attachment" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = aws_iam_policy.ecs_task_role_policy.arn
}

resource "aws_cloudwatch_log_group" "nginx" {
  name = "/fargate/containers/nginx"
}

resource "aws_cloudwatch_log_group" "fluentbit" {
  name = "/fargate/containers/fluentbit"
}

resource "aws_ecs_cluster" "nginx" {
  name = "nginx-cluster"
}

resource "aws_ecs_task_definition" "nginx" {
  family       = "nginx-task-definition"
  cpu          = "512"
  memory       = "1024"
  network_mode = "awsvpc"
  requires_compatibilities = [
  "FARGATE"]
  execution_role_arn = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn      = aws_iam_role.ecs_task_role.arn

  # container_definitions = local.nginx_simple_container_definition
  container_definitions = local.nginx_with_fluentbit_container_definitions
}

resource "aws_ecs_service" "nginx" {
  name             = "nginx-service"
  cluster          = aws_ecs_cluster.nginx.arn
  task_definition  = aws_ecs_task_definition.nginx.arn
  desired_count    = 3
  launch_type      = "FARGATE"
  platform_version = "1.4.0"

  deployment_minimum_healthy_percent = 50

  network_configuration {
    assign_public_ip = false
    security_groups  = local.nginx_service_security_groups
    subnets          = local.private_subnets
  }

  load_balancer {
    target_group_arn = local.load_balancer_target_group_arn
    container_name   = "nginx"
    container_port   = 80
  }
}

output "alb_arn" {
  value = module.load_balancer.this_lb_arn
}

output "alb_dns_name" {
  value = module.load_balancer.this_lb_dns_name
}