Splunkへのログ連携のスタンダード、SQS-based S3
Splunkを開発している方であれば、Splunk Add-on for AWSを一度は使ったことがあるのではないでしょうか？
その中でも、S3バケットに蓄積されたログをSplunkに取り込む「SQS-based S3」はログ連携の代表的な方式です。
SQS-based S3を実現するには、S3・SQS・SNSを中心とした設定が必要になります。
▲SQS-based S3によるログ連携の模式図
CloudFormationでAWS環境の構築を自動化
Splunkのログ入力の設定ごとにS3イベント通知・SNS・SQSを設定していくのですが、これをAWSマネジメントコンソールのGUIで設定するのはすごく面倒ですよね。
特に、ログの種類が多い環境だと、「あれ、どこまで設定したっけ？」となって設定中のオペミスを誘発します。
GUIからの設定はぶっちゃけ単純作業なので、自動化したいと誰もが思ったはず。
そこで役に立つのがCloudFormation。
AWSサービスのデプロイを自動化してくれるだけでなく、作成したテンプレートのパラメータを変更するだけで再利用・横展開が可能な優れものです。パラメータは、デプロイ時に画面入力から設定することも可能なので、CloudFormationに詳しくない人でも簡単に扱えます。
ここまでの長い前置きを読んで、「CloudFormationで自動化できるのは知ってるけど、書き方が分かんねえから困ってるんだよ」と思った人もいるはず。
安心してください。
下にテンプレートありますよ👇
私はCloudFormationを初めて作りましたが、生成AIとAWS公式ドキュメントのおかげでなんとか作り上げることができました。みなさんもトライしてみてください。
今回のテンプレートでは、以下のリソース・設定をデプロイできます。
（実際にデプロイした際の様子は別記事に投稿予定です。）
- S3バケット・イベント通知
- SQSキュー・デッドレターキュー
- SNSトピック・サブスクリプション
- KMSキー（SNSメッセージの暗号化に利用）
※ポリシーの絞り込みが不十分な箇所もあると思います。本番環境へのデプロイの際は必ず見直してください。
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "S3 PUT -> SNS -> SQS pipeline for Splunk ingestion (minimal access policies)",
"Parameters": {
"BucketName": {
"Type": "String",
"Description": "Name for the S3 bucket storing logs"
},
"TopicName": {
"Type": "String",
"Default": "s3-event-topic",
"Description": "SNS topic name"
},
"QueueName": {
"Type": "String",
"Default": "s3-event-queue",
"Description": "SQS queue name"
},
"DLQueueName": {
"Type": "String",
"Default": "s3-event-dlqueue",
"Description": "SQS dead letter queue name"
},
"S3EventPrefix": {
"Type": "String",
"Default": "test/",
"Description": "Optional key prefix filter for S3 events (e.g., logs/). Leave empty for none."
},
"S3EventSuffix": {
"Type": "String",
"Default": ".log",
"Description": "Optional key suffix filter for S3 events (e.g., .log). Leave empty for none."
},
"KMSKeyAliasName": {
"Type": "String",
"Default": "alias/key-for-sns-msg",
"Description": "Name of KMS key to encrypt SNS messages"
}
},
"Conditions": {
"UsePrefixFilter": {
"Fn::Not": [
{
"Fn::Equals": [
{
"Ref": "S3EventPrefix"
},
""
]
}
]
},
"UseSuffixFilter": {
"Fn::Not": [
{
"Fn::Equals": [
{
"Ref": "S3EventSuffix"
},
""
]
}
]
}
},
"Resources": {
"SnsTopic": {
"Type": "AWS::SNS::Topic",
"Properties": {
"TopicName": {
"Ref": "TopicName"
},
"KmsMasterKeyId": {
"Ref": "KMSKey"
}
},
"DependsOn": [
"KMSKey"
]
},
"SnsTopicPolicy": {
"Type": "AWS::SNS::TopicInlinePolicy",
"Properties": {
"TopicArn": {
"Ref": "SnsTopic"
},
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "S3toSNSPublishPolicy",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": {
"Ref": "SnsTopic"
},
"Condition": {
"ArnLike": {
"aws:SourceArn": [
{
"Fn::Sub": "arn:aws:s3:::${BucketName}"
}
]
}
}
}
]
}
},
"DependsOn": [
"SnsTopic"
]
},
"SqsQueue": {
"Type": "AWS::SQS::Queue",
"Properties": {
"QueueName": {
"Ref": "QueueName"
},
"VisibilityTimeout": 300,
"RedrivePolicy": {
"deadLetterTargetArn": {
"Fn::GetAtt": [
"SqsDLQueue",
"Arn"
]
},
"maxReceiveCount": 5
}
}
},
"SnsToSqsSubscription": {
"Type": "AWS::SNS::Subscription",
"Properties": {
"Protocol": "sqs",
"TopicArn": {
"Ref": "SnsTopic"
},
"Endpoint": {
"Fn::GetAtt": [
"SqsQueue",
"Arn"
]
}
},
"DependsOn": [
"SqsQueuePolicy"
]
},
"SqsQueuePolicy": {
"Type": "AWS::SQS::QueuePolicy",
"Properties": {
"Queues": [
{
"Ref": "SqsQueue"
}
],
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowSNSDeliveryFromSpecificTopic",
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": "SQS:SendMessage",
"Resource": {
"Fn::GetAtt": [
"SqsQueue",
"Arn"
]
},
"Condition": {
"ArnEquals": {
"aws:SourceArn": {
"Ref": "SnsTopic"
}
}
}
},
{
"Sid": "AllowSplunkUserAccess",
"Effect": "Allow",
"Principal": {
"AWS": {
"Fn::Sub": "arn:${AWS::Partition}:iam::${AWS::AccountId}:user/splunk_user"
}
},
"Action": [
"sqs:GetQueueAttributes",
"sqs:ListQueues",
"sqs:ReceiveMessage",
"sqs:GetQueueUrl",
"sqs:SendMessage",
"sqs:DeleteMessage"
],
"Resource": {
"Fn::GetAtt": [
"SqsQueue",
"Arn"
]
}
}
]
}
},
"DependsOn": [
"SqsQueue"
]
},
"SqsDLQueue": {
"Type": "AWS::SQS::Queue",
"Properties": {
"QueueName": {
"Ref": "DLQueueName"
},
"VisibilityTimeout": 300
}
},
"SqsDLQueuePolicy": {
"Type": "AWS::SQS::QueuePolicy",
"Properties": {
"Queues": [
{
"Ref": "SqsDLQueue"
}
],
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowSNSDeliveryFromSpecificTopic",
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": "SQS:SendMessage",
"Resource": {
"Fn::GetAtt": [
"SqsDLQueue",
"Arn"
]
},
"Condition": {
"ArnEquals": {
"aws:SourceArn": {
"Ref": "SnsTopic"
}
}
}
}
]
}
}
},
"S3LogsBucket": {
"Type": "AWS::S3::Bucket",
"Properties": {
"BucketName": {
"Ref": "BucketName"
},
"NotificationConfiguration": {
"TopicConfigurations": [
{
"Event": "s3:ObjectCreated:*",
"Topic": {
"Ref": "SnsTopic"
},
"Filter": {
"S3Key": {
"Rules": [
{
"Fn::If": [
"UsePrefixFilter",
{
"Name": "prefix",
"Value": {
"Ref": "S3EventPrefix"
}
},
{
"Name": "prefix",
"Value": ""
}
]
},
{
"Fn::If": [
"UseSuffixFilter",
{
"Name": "suffix",
"Value": {
"Ref": "S3EventSuffix"
}
},
{
"Name": "suffix",
"Value": ""
}
]
}
]
}
}
}
]
}
},
"DependsOn": [
"SnsTopicPolicy"
]
},
"KMSKey": {
"Type": "AWS::KMS::Key",
"Properties": {
"BypassPolicyLockoutSafetyCheck": false,
"Description": "KMS Key for SNS",
"EnableKeyRotation": false,
"Enabled": true,
"KeyPolicy": {
"Version": "2012-10-17",
"Id": "key-consolepolicy-3",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": {
"Fn::Sub": "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
}
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow access for Key Administrators",
"Effect": "Allow",
"Principal": {
"AWS": [
{
"Fn::Sub": "arn:${AWS::Partition}:iam::${AWS::AccountId}:user/Admin"
}
]
},
"Action": [
"kms:Create*",
"kms:Describe*",
"kms:Enable*",
"kms:List*",
"kms:Put*",
"kms:Update*",
"kms:Revoke*",
"kms:Disable*",
"kms:Get*",
"kms:Delete*",
"kms:TagResource",
"kms:UntagResource",
"kms:ScheduleKeyDeletion",
"kms:CancelKeyDeletion",
"kms:RotateKeyOnDemand"
],
"Resource": "*"
},
{
"Sid": "Allow attachment of persistent resources",
"Effect": "Allow",
"Principal": {
"Service": [
"sns.amazonaws.com",
"s3.amazonaws.com"
]
},
"Action": "kms:*",
"Resource": "*"
}
]
}
}
},
"KMSKeyAlias": {
"Type": "AWS::KMS::Alias",
"Properties": {
"AliasName": {
"Ref": "KMSKeyAliasName"
},
"TargetKeyId": {
"Ref": "KMSKey"
}
}
}
},
"Outputs": {
"BucketArn": {
"Value": {
"Fn::Sub": "arn:aws:s3:::${S3LogsBucket}"
}
},
"TopicArn": {
"Value": {
"Ref": "SnsTopic"
}
},
"QueueUrl": {
"Value": {
"Ref": "SqsQueue"
}
},
"QueueArn": {
"Value": {
"Fn::GetAtt": [
"SqsQueue",
"Arn"
]
}
},
"DLQueueUrl": {
"Value": {
"Ref": "SqsDLQueue"
}
},
"DLQueueArn": {
"Value": {
"Fn::GetAtt": [
"SqsDLQueue",
"Arn"
]
}
},
"KMSKeyId": {
"Value": {
"Ref": "KMSKey"
}
}
}
}
参考
テンプレートの中身はAWSのドキュメントを参考にして解読してみてください。