LoginSignup
6

More than 5 years have passed since last update.

CloudWatch LogsのデータをFirehoseを使ってS3に保存するCloudFormationのテンプレート

Posted at

こちらの、リンクを参考にしています。
CloudWatchLogsのログをFirehose経由でS3に出力してみた | Developers.IO

変更した点は、以下の通りです。

  • 既存のターゲットロググループを指定できる
  • 保存するS3のバケットおよびprefixを指定できる
  • 保存したS3のオブジェクトの有効期限をリスト選択できるようにした(デフォルトは365日)
  • Firehose実行ログの有効期限をリスト選択できるようにした(デフォルトは7日)

参考になれば幸いです。
Developers.IO最高です!

cfn-cwlogs-direct-firehose.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: Save directly to S3 from CloudWatch Logs using Firehose
Parameters:
  TargetLogGroup:
    Description: 'Input target log group'
    Type: String
    Default: ''
  TargetS3bucket:
    Description: 'Input S3 bucket name to save'
    Type: String
    Default: ''
  TargetS3bucketPrefix:
    Description: 'Input prefix in S3 bucket'
    Type: String
    Default: ''
  S3Expiredate:
    Description: 'Number of days to keep S3 file'
    Type: Number
    Default: 365
    AllowedValues: [1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653]
  CWlogsFirehoseExpiredate:
    Description: 'Number of days to keep Cloudwatch logs'
    Type: Number
    Default: 7
    AllowedValues: [1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653]
Resources:
  TargetS3bucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Delete
    Properties:
      BucketName: !Ref 'TargetS3bucket'
      LifecycleConfiguration:
        Rules:
          - Id: AutoDelete
            Status: Enabled
            ExpirationInDays: !Ref 'S3Expiredate'
  SubscriptionFilter:
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      DestinationArn: !GetAtt 'Deliverystream.Arn'
      FilterPattern: ''
      LogGroupName: !Ref 'TargetLogGroup'
      RoleArn: !GetAtt 'LogsRole.Arn'
  LogsRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: !Sub 'logs.${AWS::Region}.amazonaws.com'
            Action:
              - sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - firehose:*
                Resource: !GetAtt 'Deliverystream.Arn'
  Deliverystream:
    DependsOn:
      - DeliveryPolicy
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      ExtendedS3DestinationConfiguration:
        BucketARN: !GetAtt 'TargetS3bucket.Arn'
        BufferingHints:
          IntervalInSeconds: '60'
          SizeInMBs: '50'
        CompressionFormat: GZIP
        Prefix: !Join [ "/", [ !Ref TargetS3bucketPrefix, '' ] ]
        RoleARN: !GetAtt 'DeliveryRole.Arn'
        ProcessingConfiguration:
          Enabled: 'false'
  DeliveryRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: ''
            Effect: Allow
            Principal:
              Service: firehose.amazonaws.com
            Action: sts:AssumeRole
            Condition:
              StringEquals:
                sts:ExternalId: !Ref 'AWS::AccountId'
  DeliveryPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: firehose_delivery_policy
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - s3:AbortMultipartUpload
              - s3:GetBucketLocation
              - s3:GetObject
              - s3:ListBucket
              - s3:ListBucketMultipartUploads
              - s3:PutObject
            Resource:
              - !Sub 'arn:aws:s3:::${TargetS3bucket}'
              - !Sub 'arn:aws:s3:::${TargetS3bucket}*'
      Roles:
        - !Ref 'DeliveryRole'
  LogGroupFirehose:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub '/aws/firehose/${Deliverystream}'
      RetentionInDays: !Ref 'CWlogsFirehoseExpiredate'

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
6