業務でLinuxサーバに対するSSHログインを監視する処理を構築することになったため、「CloudWatch Logsに/var/log/secureログファイル転送→メトリクスフィルタで特定ログをフィルタリング→CloudWatch Alarmで拾ってメール通知」という流れをCDKで検証をしてまとめてみました。基本的にはBLEAテンプレートを参考にして作成しています。

構成図

サーバログイン監視に加えて、各リソース使用率も監視しています。
アラーム閾値は以下のように設定しています。

CPU : 15分内の2データポイントのCPUUtilization >= 80
Memory : 15分内の2データポイントのmem_used_percent >= 80
Disk : 1分内の1データポイントのdisk_used_percent >= 80

コンソール画面

ディレクトリ構造

本記事ではbin, lib配下のファイルを紹介します。

├── bin
│   └── aws_cloud_watch_sample.ts
├── lib
│   └── Stack
│       ├── aws_cloud_watch_sample-stack.ts
│       └── logs_sns_notification-stack.ts
│   └── Construct
│       ├── ec2.ts
│       ├── vpc.ts
│       ├── cloudwatch.ts
│       └── sns.ts
├── node_modules
├── test
│   └── aws_cloud_watch_sample.test.ts
├── cdk.json
├── jest.config.js
├── package-lock.json
├── package.json
├── README.md
└── tsconfig.json

CDK実装コード

bin

aws_cloud_watch_sample.ts

import * as cdk from 'aws-cdk-lib';
import { AwsCloudWatchSampleStack } from '../lib/Stack/aws_cloud_watch_sample-stack';
import { LogsSnsNotificationStack } from '../lib/Stack/logs_sns_notification-stack';

const app = new cdk.App();
const BaseStack = new AwsCloudWatchSampleStack(app, 'AwsCloudWatchSampleStack');
new LogsSnsNotificationStack(app, 'LogsSnsNotificationStack',{
    instance: BaseStack.createEc2(),
});

lib/Stack

aws_cloud_watch_sample-stack.ts : VPC, EC2, Endpointなど基本的な構成要素を定義
logs_sns_notification-stack.ts : メトリクスフィルタ, CloudWatch Alarm, SNS Topicなど監視・通知関連の要素を定義

LinuxでCloudWatch Agentのセットアップを行い、CloudWatch Logsにログを配信する部分は手動で行う必要があるので以下の手順で2つのStackをデプロイします。CloudWatch Agentの設定は以下の記事が非常に参考になります。

AwsCloudWatchSampleStackをデプロイ
CloudWatch Agentのセットアップ（SSM Parameter StoreでのConfig編集も含む）
LogsSnsNotificationStackをデプロイ

Agentの設定はお好きなように設定いただければと思います。以下に例として今回の設定値を記したConfigファイルを置いておきます。（Parameter Storeに格納）

{
	"agent": {
		"metrics_collection_interval": 60,
		"run_as_user": "root"
	},
	"logs": {
		"logs_collected": {
			"files": {
				"collect_list": [
					{
						"file_path": "/var/log/secure",
						"log_group_class": "STANDARD",
						"log_group_name": "secure",
						"log_stream_name": "{instance_id}",
						"retention_in_days": 7
					}
				]
			}
		}
	},
	"metrics": {
		"aggregation_dimensions": [
			[
				"InstanceId"
			]
		],
		"append_dimensions": {
			"InstanceId": "${aws:InstanceId}"
		},
		"metrics_collected": {
			"disk": {
				"measurement": [
					"used_percent"
				],
				"metrics_collection_interval": 60,
				"resources": [
					"*"
				]
			},
			"mem": {
				"measurement": [
					"mem_used_percent"
				],
				"metrics_collection_interval": 60
			}
		}
	}
}

aws_cloud_watch_sample-stack.ts

import { CfnOutput, RemovalPolicy, Stack, StackProps } from "aws-cdk-lib";
import { Construct } from 'constructs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import { VpcConstruct } from "../construct/vpc";
import { Ec2Construct } from "../construct/ec2";

export class AwsCloudWatchSampleStack extends Stack {
  private readonly ec2: ec2.Instance;

  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    // ----- VPC and Subnet -----
    const vpc_tokyo = new VpcConstruct(this, "VPC-Tokyo", {
      cidr: "10.10.0.0/16",
      vpcName: "VPC-Tokyo",
      az: "ap-northeast-1a",
    }).createVpc();

    // ----- KeyPair -----
    const cfnKeyPair = new ec2.CfnKeyPair(this, "CfnKeyPair", {
      keyName: "test-key-pair",
    });
    cfnKeyPair.applyRemovalPolicy(RemovalPolicy.DESTROY);

    new CfnOutput(this, 'GetSSHKeyCommand', {
      value: `aws ssm get-parameter --name /ec2/keypair/${cfnKeyPair.getAtt('KeyPairId')} --region ap-northeast-1 --with-decryption --query Parameter.Value --output text`,
    });

    // ----- EC2 instance -----
    const Server1 = new Ec2Construct(this, "Server1", {
      vpc: vpc_tokyo,
      ssmConnect: true,
      keyName: cfnKeyPair.keyName,
    });
    this.ec2 = Server1.createEc2();

    // -----Interface VPC endpoint-----
    const endpoint: Array<[ec2.InterfaceVpcEndpointAwsService, string]> =[
        [ec2.InterfaceVpcEndpointAwsService.SSM, "ssm"],
        [ec2.InterfaceVpcEndpointAwsService.SSM_MESSAGES, "ssm-messages"],
        [ec2.InterfaceVpcEndpointAwsService.EC2_MESSAGES, "ec2-messages"],
        [ec2.InterfaceVpcEndpointAwsService.CLOUDWATCH_LOGS, "logs"],
        [ec2.InterfaceVpcEndpointAwsService.CLOUDWATCH_MONITORING, "monitoring"],
        [ec2.InterfaceVpcEndpointAwsService.EC2, "ec2"],
      ]

    for (const data of endpoint) {
      new ec2.InterfaceVpcEndpoint(this, data[1], {
        vpc: vpc_tokyo,
        service: data[0],
        subnets: {
          availabilityZones: ["ap-northeast-1a"],
        }
      });
    }

    // -----Gateway VPC endpoint-----
    const S3endpoint = vpc_tokyo.addGatewayEndpoint("S3Endpoint", {
      service: ec2.GatewayVpcEndpointAwsService.S3,
    });

  }

  // not Ec2Construct but ec2.Instance
  public createEc2(): ec2.Instance {
    return this.ec2;
  }
}

logs_sns_notification-stack.ts

import { Stack, StackProps } from "aws-cdk-lib";
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import { Construct } from 'constructs';
import { SnsConstruct } from "../construct/sns";
import { CwConstruct } from "../construct/cloudwatch";

interface SubStackProps extends StackProps {
    instance: ec2.Instance;
}

export class LogsSnsNotificationStack extends Stack {

    constructor(scope: Construct, id: string, props: SubStackProps) {
        super(scope, id, props);

        // ----- SNS Topic -----
        const Topic_for_Alarm = new SnsConstruct(this, "Topic1", {
            notifyEmail: "xxxxxxxx@gmail.com"
        }).createTopic();

        // ----- CloudWatch Logs LogGroup + Metrics Filter + Alarm -----
        const Cwl1 = new CwConstruct(this, "Cwl1", {
            LinuxLogGroupName: "secure",
            AlarmTopic: Topic_for_Alarm,
            TargetInstance: props.instance,
        });

    }
}

lib/Construct

ec2.ts

今回は業務の関係でOSにはRHEL8を採用しています。RHEL8ではパッケージを取得する際に、初めにデフォルトで定められたリポジトリを確認し、その中からパッケージを見つけようとします。そこになければ指定したURLの方にパッケージを見つけに行くという仕様のようです。
今回のようなPrivateに閉じたネットワークからはリポジトリにアクセスできずにタイムアウトになってしまうため、「--disablerepo="*"」オプションによりそれらのリポジトリを確認せずに直接指定したURLを見に行くように設定してあげる必要があります。

import { Construct } from "constructs";
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as iam from "aws-cdk-lib/aws-iam";

export interface Ec2Props {
    vpc: ec2.Vpc;
    ssmConnect?: boolean;
    keyName?: string;
}

// https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ec2.Instance.html
export class Ec2Construct extends Construct {
    private readonly ec2: ec2.Instance;

    constructor(scope: Construct, id: string, props: Ec2Props) {
        super(scope, id);

        // Set ssmConnect to false if it's not provided in the props
        const ssmConnect = props.ssmConnect !== undefined ? props.ssmConnect : false;

        // Leave keyName undefined if it`s not provided in the props
        const keyName = props.keyName !== undefined ? props.keyName : undefined;

        // https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_iam.Role.html
        const ec2Role = new iam.Role(this, "Ec2Role for Cloudwatch", {
            assumedBy: new iam.ServicePrincipal("ec2.amazonaws.com"),
            description: "IAM role for EC2",
        });
        ec2Role.addManagedPolicy(iam.ManagedPolicy.fromAwsManagedPolicyName('CloudWatchAgentAdminPolicy'));

        // https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ec2.UserData.html
        const ec2UserData = ec2.UserData.forLinux({ shebang: '#!/bin/bash' });
        ec2UserData.addCommands(
            'cd /tmp',
            'sudo dnf --disablerepo="*" install -y https://s3.ap-northeast-1.amazonaws.com/amazon-ssm-ap-northeast-1/latest/linux_amd64/amazon-ssm-agent.rpm',
            'sudo systemctl enable amazon-ssm-agent',
            'sudo systemctl start amazon-ssm-agent',
            'sudo dnf --disablerepo="*" install -y https://s3.ap-northeast-1.amazonaws.com/amazoncloudwatch-agent-ap-northeast-1/redhat/amd64/latest/amazon-cloudwatch-agent.rpm',
        );

        // https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ec2.Instance.html
        const instance = new ec2.Instance(this, id, {
            vpc: props.vpc,
            instanceType: ec2.InstanceType.of(ec2.InstanceClass.M4, ec2.InstanceSize.LARGE),
            machineImage: new ec2.GenericLinuxImage({
                "ap-northeast-1":"ami-0d7425645234bd54d",
            }),
            role: ec2Role,
            ssmSessionPermissions: ssmConnect,
            keyName: keyName,
            userData: ec2UserData,
        });
        this.ec2 = instance;
    }

    // not Ec2Construct but ec2.Instance
    public createEc2(): ec2.Instance {
        return this.ec2;
    }
    
}

vpc.ts

import { Construct } from "constructs";
import * as ec2 from 'aws-cdk-lib/aws-ec2';

export interface VpcProps {
    cidr: string;
    vpcName: string;
    az: string;
}

// https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ec2.Vpc.html
export class VpcConstruct extends Construct {
    private readonly vpc: ec2.Vpc;

    constructor(scope: Construct, id: string, props: VpcProps) {
        super(scope, id);
        this.vpc = new ec2.Vpc(this, id, {
            ipAddresses: ec2.IpAddresses.cidr(props.cidr),
            vpcName: props.vpcName,
            availabilityZones: [props.az],
            subnetConfiguration: [
                {
                    name: 'Private',
                    cidrMask: 24,
                    subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
                }
            ],
        });
        // this.vpc.addFlowLog("FlowLog", {
        //     trafficType: ec2.FlowLogTrafficType.ALL
        // })
    }

    // Not VpcConstruct but ec2.Vpc
    public createVpc(): ec2.Vpc {
        return this.vpc;
    }
}

cloudwatch.ts

/var/log/secureをCloudWatch Logsに配信するようにAgentセットアップをおこなっており、secureという名のロググループが作成されます。今回はサーバログインを検知するため、フィルタパターンは"Accepted publickey"と設定しています。

import { Construct } from "constructs";
import * as cdk from 'aws-cdk-lib';
import * as logs from 'aws-cdk-lib/aws-logs';
import * as sns from "aws-cdk-lib/aws-sns";
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as cw from 'aws-cdk-lib/aws-cloudwatch';
import * as cwa from "aws-cdk-lib/aws-cloudwatch-actions";

export interface CloudwatchProps {
    LinuxLogGroupName: string;
    AlarmTopic: sns.ITopic;
    TargetInstance: ec2.Instance;
}

export class CwConstruct extends Construct {

    constructor(scope: Construct, id: string, props: CloudwatchProps) {
        super(scope, id);

        // https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_logs-readme.html
        const LinuxLogGroup = logs.LogGroup.fromLogGroupName(this, "Linux Log Group", props.LinuxLogGroupName);

        // https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_logs.MetricFilter.html
        const UserLoginMetricsFilter = new logs.MetricFilter(this, "ServerLoginMetricsFilter", {
            logGroup: LinuxLogGroup,
            filterPattern: logs.FilterPattern.anyTerm('Accepted publickey'), 
            filterName: "Metrics Filter to detect logging into server",
            metricNamespace: "CWLogs",
            metricName: "server-login",
            metricValue: "1",
        });

        // https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cloudwatch.Alarm.html
        new cw.Alarm(this, "ServerLoginAlarm", {
            metric: UserLoginMetricsFilter.metric({
                statistic: cw.Stats.SUM,
                period: cdk.Duration.seconds(60)
            }),
            comparisonOperator: cw.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
            threshold: 1,
            evaluationPeriods: 1,
            datapointsToAlarm: 1,
            treatMissingData: cw.TreatMissingData.NOT_BREACHING,
            alarmName: "ServerLoginAlarm",
        }).addAlarmAction(new cwa.SnsAction(props.AlarmTopic));

        // https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cloudwatch.Metric.html
        new cw.Alarm(this, "CPUUtilizationAlarm", {
            metric: new cw.Metric({
                namespace: "AWS/EC2",
                metricName: "CPUUtilization",
                dimensionsMap: {
                    "InstanceId": props.TargetInstance.instanceId
                },
                statistic: cw.Stats.AVERAGE,
                period: cdk.Duration.seconds(300),
            }),
            comparisonOperator: cw.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
            threshold: 80,
            evaluationPeriods: 3,
            datapointsToAlarm: 2,
            treatMissingData: cw.TreatMissingData.BREACHING,
            alarmName: "CPUUtilizationAlarm",
        }).addAlarmAction(new cwa.SnsAction(props.AlarmTopic));
        
        new cw.Alarm(this, "MemoryUsageAlarm", {
            metric: new cw.Metric({
                namespace: "CWAgent",
                metricName: "mem_used_percent",
                dimensionsMap: {
                    "InstanceId": props.TargetInstance.instanceId
                },
                statistic: cw.Stats.AVERAGE,
                period: cdk.Duration.seconds(300),
            }),
            comparisonOperator: cw.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
            threshold: 80,
            evaluationPeriods: 3,
            datapointsToAlarm: 2,
            treatMissingData: cw.TreatMissingData.BREACHING,
            alarmName: "MemoryUsageAlarm",
        }).addAlarmAction(new cwa.SnsAction(props.AlarmTopic));

        new cw.Alarm(this, "DiskUsageAlarm", {
            metric: new cw.Metric({
                namespace: "CWAgent",
                metricName: "disk_used_percent",
                dimensionsMap: {
                     "InstanceId": props.TargetInstance.instanceId,
                     "device": "xvda2",
                     "fstype": "xfs",
                     "path": "/"
                },
                statistic: cw.Stats.AVERAGE,
                period: cdk.Duration.seconds(60),
            }),
            comparisonOperator: cw.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
            threshold: 80,
            evaluationPeriods: 1,
            datapointsToAlarm: 1,
            treatMissingData: cw.TreatMissingData.BREACHING,
            alarmName: "DiskUsageAlarm",
        }).addAlarmAction(new cwa.SnsAction(props.AlarmTopic));
    }

}

sns.ts

import { Construct } from "constructs";
import * as sns from "aws-cdk-lib/aws-sns"

export interface SnsProps {
    notifyEmail: string;
}

export class SnsConstruct extends Construct{
    private readonly topic: sns.ITopic

    constructor(scope: Construct, id: string, props: SnsProps) {
        super(scope, id);

        // https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_sns-readme.html
        const topic = new sns.Topic(this, "Topic for Alarm");

        new sns.Subscription(this, "MonitorAlarmEmail", {
            endpoint: props.notifyEmail,
            protocol: sns.SubscriptionProtocol.EMAIL,
            topic: topic,
        });
        this.topic = topic;
    }

    // not SnsConstruct but sns.ITopic
    public createTopic(): sns.ITopic {
        return this.topic;
    }
}

最後に

BLEAテンプレートは本当に素晴らしいですが、構成図がないのが少しネックなんですよね...
時間があればBLEAの構成図を全て丁寧にまとめて記事にしたいと思います。