AWS
aws-cli
Kinesis
Firehose

[JAWS-UG CLI] Amazon Kinesis Firehose re:入門 (1) 事前準備(Source(Kinesis Agent)およびDestination(S3)の作成)

More than 1 year has passed since last update.


この記事について

JAWS-UG CLI専門支部 #90 Kinesis Firehose 復習編で実施するハンズオン用の手順書です。


前提条件


必要な権限

作業にあたっては、以下の権限を有したIAMユーザもしくはIAMロールを利用してください。


  • 以下のサービスに対するフルコントロール権限


    • Kinesis Firehose

    • IAM

    • EC2

    • S3

    • CloudWatch Logs

    • STS

    • (Lambda)


      • データの変換を行う場合



    • (KMS)


      • データの暗号化を行う場合






0. 準備


0.1. リージョンを指定

オレゴンリージョンで実施します。(東京マダー?)


コマンド

export AWS_DEFAULT_REGION="us-west-2"



0.2. 資格情報を確認


コマンド

aws configure list


インスタンスプロファイルを設定したEC2インスタンスでアクセスキーを設定せずに実行した場合、以下のようになります。


結果

      Name                    Value             Type    Location

---- ----- ---- --------
profile <not set> None None
access_key ****************QSAA iam-role
secret_key ****************c1xY iam-role
region us-west-2 env AWS_DEFAULT_REGION


0.3. バージョン確認


コマンド

aws --version



結果

aws-cli/1.11.129 Python/2.7.12 Linux/4.9.38-16.33.amzn1.x86_64 botocore/1.5.92



0.4. バージョンアップ(必要に応じて)


コマンド

sudo pip install -U awscli



1. 管理対象の構築

CloudFormationを利用して、Source(Kinesis AgentをインストールしたEC2インスタンス)とDestination(S3バケット)を作成します。


1.1. KeyPairの作成

EC2インスタンス用にKeyPairを作成します。


KeyPairの名前を指定


コマンド

AWS_ID=$(aws sts get-caller-identity \

--query "Account" \
--output text) \
&& echo ${AWS_ID}


コマンド

KEY_PAIR_NAME="${AWS_ID}_firehose_jawsug_cli"

KEY_MATERIAL_FILE_NAME=${KEY_PAIR_NAME}.pem


同名KeyPairの不存在を確認


コマンド

aws ec2 describe-key-pairs \

--query "KeyPairs[?KeyName==\`${KEY_PAIR_NAME}\`]"


結果

[]



KeyPairの作成


コマンド

aws ec2 create-key-pair \

--key-name ${KEY_PAIR_NAME} \
--query "KeyMaterial" \
--output text \
> ~/.ssh/${KEY_MATERIAL_FILE_NAME} \
&& cat ~/.ssh/${KEY_MATERIAL_FILE_NAME}


KeyPairの存在を確認


コマンド

aws ec2 describe-key-pairs \

--query "KeyPairs[?KeyName==\`${KEY_PAIR_NAME}\`]"


結果

[

{
"KeyName": "XXXXXXXXXXXX_firehose_jawsug_cli",
"KeyFingerprint": "xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"
}
]


秘密鍵のPermissionを変更


コマンド

chmod 600 ~/.ssh/${KEY_MATERIAL_FILE_NAME}

ls -al ~/.ssh/${KEY_MATERIAL_FILE_NAME}


結果

-rw------- 1 ec2-user ec2-user 1671 Aug  5 18:33 /home/ec2-user/.ssh/788063364413_firehose_jawsug_cli.pem



1.2. CloudFormation テンプレートの生成


テンプレートの作成


コマンド

CF_TEMPLATE_FILE_NAME="firehose_jawsug_cli.yml"



コマンド

cat << EOF > ${CF_TEMPLATE_FILE_NAME}

AWSTemplateFormatVersion: "2010-09-09"
Description: JAWS-UG CLI Kinesis Firehose Hands-on

Parameters:
VPCNetworkAddress:
Type: String
Description: "Network Address on AWS"
MinLength: 9
MaxLength: 18
# AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
Default: "10.0.0.0/16"
PublicSubnetAddr:
Type: String
Description: "Network Address on AWS"
MinLength: 9
MaxLength: 18
# AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
Default: "10.0.0.0/24"
KeyPairName:
Type: AWS::EC2::KeyPair::KeyName

Resources:
S3Bucket:
Type: "AWS::S3::Bucket"
IAMRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: "service-role-firehose"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "firehose.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
IAMPolicy:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: "service-policy-firehose"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- "s3:AbortMultipartUpload"
- "s3:GetBucketLocation"
- "s3:GetObject"
- "s3:ListBucket"
- "s3:ListBucketMultipartUploads"
- "s3:PutObject"
Resource:
- !GetAtt S3Bucket.Arn
- Fn::Join:
- "/"
-
- !GetAtt S3Bucket.Arn
- "*"
-
Effect: "Allow"
Action:
- "logs:PutLogEvents"
Resource: "*"
Roles:
- Ref: IAMRole

VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Ref VPCNetworkAddress
Tags:
-
Key: "Name"
Value: "KinesisFirehoseClient"
IGW:
Type: AWS::EC2::InternetGateway
Properties:
Tags:
-
Key: "Name"
Value: "KinesisFirehoseClient"
AttachIGW:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId:
Ref: VPC
InternetGatewayId:
Ref: IGW
PublicSubnet:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone:
Fn::Select:
- 0
- Fn::GetAZs: ""
CidrBlock: !Ref PublicSubnetAddr
MapPublicIpOnLaunch: true
VpcId:
Ref: VPC
Tags:
-
Key: "Name"
Value: "Public"
PublicRT:
Type: AWS::EC2::RouteTable
Properties:
VpcId:
Ref: VPC
Tags:
-
Key: Name
Value: Public
PublicDefaultRoute:
Type: AWS::EC2::Route
Properties:
DestinationCidrBlock: "0.0.0.0/0"
GatewayId:
Ref: IGW
RouteTableId:
Ref: PublicRT
PublicSubnetARouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId:
Ref: PublicSubnet
RouteTableId:
Ref: PublicRT
SecurityGroup:
Type: "AWS::EC2::SecurityGroup"
Properties:
GroupDescription: WebServer
SecurityGroupEgress:
-
IpProtocol: "-1"
CidrIp: "0.0.0.0/0"
SecurityGroupIngress:
-
IpProtocol: "tcp"
FromPort: 80
ToPort: 80
CidrIp: "0.0.0.0/0"
-
IpProtocol: "tcp"
FromPort: 22
ToPort: 22
CidrIp: "0.0.0.0/0"
VpcId:
Ref: VPC
InstanceRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: "instance-role"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "ec2.amazonaws.com"
- "ssm.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM"
- "arn:aws:iam::aws:policy/AmazonKinesisFirehoseFullAccess"
InstanceProfile:
Type: "AWS::IAM::InstanceProfile"
Properties:
Path: "/"
Roles:
- !Ref InstanceRole
Instance:
Type: "AWS::EC2::Instance"
Properties:
KeyName:
Ref: KeyPairName
ImageId: ami-6df1e514
InstanceType: t2.micro
SecurityGroupIds:
- Ref: SecurityGroup
SubnetId:
Ref: PublicSubnet
IamInstanceProfile:
Ref: InstanceProfile
BlockDeviceMappings:
- DeviceName: "/dev/sdm"
Ebs:
VolumeType: "gp2"
DeleteOnTermination: "true"
VolumeSize: "8"
UserData:
Fn::Base64: |
#!/bin/bash
cd /tmp
sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
sudo yum install -y aws-kinesis-agent
sudo chkconfig aws-kinesis-agent on
sudo service aws-kinesis-agent start
sudo yum install -y httpd
sudo chkconfig httpd on
sudo service httpd start
sudo yum install npm --enablerepo=epel -y
sudo npm install -g jsonlint

Outputs:
S3BucketName:
Value:
Ref: S3Bucket
IAMRoleARN:
Value: !GetAtt IAMRole.Arn
PublicIP:
Value: !GetAtt Instance.PublicIp
EOF

cat ${CF_TEMPLATE_FILE_NAME}



CloudFormation テンプレートの検証


コマンド

aws cloudformation validate-template \

--template-body file://${CF_TEMPLATE_FILE_NAME}


結果

{

"CapabilitiesReason": "The following resource(s) require capabilities: [AWS::IAM::InstanceProfile, AWS::IAM::Role]",
"Description": "JAWS-UG CLI Kinesis Firehose Hands-on",
"Parameters": [
{
"NoEcho": false,
"ParameterKey": "KeyPairName"
},
{
"DefaultValue": "10.0.0.0/16",
"NoEcho": false,
"Description": "Network Address on AWS",
"ParameterKey": "VPCNetworkAddress"
},
{
"DefaultValue": "10.0.0.0/24",
"NoEcho": false,
"Description": "Network Address on AWS",
"ParameterKey": "PublicSubnetAddr"
}
],
"Capabilities": [
"CAPABILITY_NAMED_IAM"
]
}


1.3. CloudFormation Stackの作成


CloudFormation Stack名の指定


コマンド

CF_STACK_NAME="firehose-jawsug-cli"



同名CloudFormation Stackの不存在を確認


コマンド

aws cloudformation describe-stacks \

--query "Stacks[?StackName==\`${CF_STACK_NAME}\`]"


結果

[]



CloudFormation Stackの作成


コマンド

aws cloudformation create-stack \

--stack-name ${CF_STACK_NAME} \
--template-body file://${CF_TEMPLATE_FILE_NAME} \
--capabilities "CAPABILITY_NAMED_IAM" \
--parameters ParameterKey=KeyPairName,ParameterValue=${KEY_PAIR_NAME},UsePreviousValue=false


結果

{

"StackId": "arn:aws:cloudformation:us-west-2:XXXXXXXXXXXX:stack/firehose-jawsug-cli/8812e540-7a0e-11e7-aac3-50a68d01a68d"
}


CloudFormation Stackの作成完了を待機

5分程度で作成が完了すると思います。


コマンド

aws cloudformation wait stack-create-complete \

--stack-name ${CF_STACK_NAME}


結果

(返値無し)



CloudFormation Stackの存在を確認

"StackStatus"が"CREATE_COMPLETE"になっていることを確認します。


コマンド

aws cloudformation describe-stacks \

--stack-name ${CF_STACK_NAME}


結果

{

"Stacks": [
{
"StackId": "arn:aws:cloudformation:us-west-2:XXXXXXXXXXXX:stack/firehose-jawsug-cli/4043bde0-7a16-11e7-8701-50a686be73ba",
"Description": "JAWS-UG CLI Kinesis Firehose Hands-on",
"Parameters": [
{
"ParameterValue": "XXXXXXXXXXXX_firehose_jawsug_cli",
"ParameterKey": "KeyPairName"
},
{
"ParameterValue": "10.0.0.0/16",
"ParameterKey": "VPCNetworkAddress"
},
{
"ParameterValue": "10.0.0.0/24",
"ParameterKey": "PublicSubnetAddr"
}
],
"Tags": [],
"Outputs": [
{
"OutputKey": "PublicIP",
"OutputValue": "54.191.102.113"
},
{
"OutputKey": "IAMRoleARN",
"OutputValue": "arn:aws:iam::XXXXXXXXXXXX:role/service-role-firehose"
},
{
"OutputKey": "S3BucketName",
"OutputValue": "firehose-jawsug-cli-s3bucket-134czh3hcofqz"
}
],
"CreationTime": "2017-08-05T19:42:44.440Z",
"Capabilities": [
"CAPABILITY_NAMED_IAM"
],
"StackName": "firehose-jawsug-cli",
"NotificationARNs": [],
"StackStatus": "CREATE_COMPLETE",
"DisableRollback": false
}
]
}


1.4. パラメータの確認

以降の手順で必要になるパラメータを抽出します。


  • IAMロールARN

  • S3バケット名

  • パブリックIPアドレス


コマンド

OUTPUTKEY_ROLE_ARN="IAMRoleARN"

OUTPUTKEY_BUCKET_NAME="S3BucketName"
OUTPUTKEY_PUBLIC_IP_ADDRESS="PublicIP"


コマンド

ROLE_ARN=$(aws cloudformation describe-stacks \

--stack-name ${CF_STACK_NAME} \
--query "Stacks[].Outputs[?OutputKey==\`${OUTPUTKEY_ROLE_ARN}\`].OutputValue[]" \
--output text) \
&& echo ${ROLE_ARN}


結果

arn:aws:iam::XXXXXXXXXXXX:role/service-role-firehose



コマンド

BUCKET_NAME=$(aws cloudformation describe-stacks \

--stack-name ${CF_STACK_NAME} \
--query "Stacks[].Outputs[?OutputKey==\`${OUTPUTKEY_BUCKET_NAME}\`].OutputValue[]" \
--output text) \
&& echo ${BUCKET_NAME}


結果

firehose-jawsug-cli-s3bucket-134czh3hcofqz



コマンド

PUBLIC_IP_ADDRESS=$(aws cloudformation describe-stacks \

--stack-name ${CF_STACK_NAME} \
--query "Stacks[].Outputs[?OutputKey==\`${OUTPUTKEY_PUBLIC_IP_ADDRESS}\`].OutputValue[]" \
--output text) \
&& echo ${PUBLIC_IP_ADDRESS}


結果

54.191.***.***



動作確認

パブリックIPアドレスにブラウザでアクセスします。

以上