[JAWS-UG CLI] Amazon Kinesis Firehose re:入門 (1) 事前準備（Source（Kinesis Agent）およびDestination（S3）の作成）

この記事について

JAWS-UG CLI専門支部 #90 Kinesis Firehose 復習編で実施するハンズオン用の手順書です。

前提条件

必要な権限

作業にあたっては、以下の権限を有したIAMユーザもしくはIAMロールを利用してください。

• 以下のサービスに対するフルコントロール権限
  • Kinesis Firehose
  • IAM
  • EC2
  • S3
  • CloudWatch Logs
  • STS
  • (Lambda)
    • データの変換を行う場合
  • (KMS)
    • データの暗号化を行う場合

0. 準備

0.1. リージョンを指定

オレゴンリージョンで実施します。（東京マダー？）

コマンド

export AWS_DEFAULT_REGION="us-west-2"

0.2. 資格情報を確認

コマンド

aws configure list

インスタンスプロファイルを設定したEC2インスタンスでアクセスキーを設定せずに実行した場合、以下のようになります。

結果

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************QSAA         iam-role
secret_key     ****************c1xY         iam-role
    region                us-west-2              env    AWS_DEFAULT_REGION

0.3. バージョン確認

コマンド

aws --version

結果

aws-cli/1.11.129 Python/2.7.12 Linux/4.9.38-16.33.amzn1.x86_64 botocore/1.5.92

0.4. バージョンアップ（必要に応じて）

コマンド

sudo pip install -U awscli

1. 管理対象の構築

CloudFormationを利用して、Source（Kinesis AgentをインストールしたEC2インスタンス）とDestination（S3バケット）を作成します。

1.1. KeyPairの作成

EC2インスタンス用にKeyPairを作成します。

KeyPairの名前を指定

コマンド

AWS_ID=$(aws sts get-caller-identity \
    --query "Account" \
    --output text) \
    && echo ${AWS_ID}

コマンド

KEY_PAIR_NAME="${AWS_ID}_firehose_jawsug_cli"
KEY_MATERIAL_FILE_NAME=${KEY_PAIR_NAME}.pem

同名KeyPairの不存在を確認

コマンド

aws ec2 describe-key-pairs \
    --query "KeyPairs[?KeyName==\`${KEY_PAIR_NAME}\`]"

結果

[]

KeyPairの作成

コマンド

aws ec2 create-key-pair \
    --key-name ${KEY_PAIR_NAME} \
    --query "KeyMaterial" \
    --output text \
    > ~/.ssh/${KEY_MATERIAL_FILE_NAME} \
    && cat ~/.ssh/${KEY_MATERIAL_FILE_NAME}

KeyPairの存在を確認

コマンド

aws ec2 describe-key-pairs \
    --query "KeyPairs[?KeyName==\`${KEY_PAIR_NAME}\`]"

結果

[
    {
        "KeyName": "XXXXXXXXXXXX_firehose_jawsug_cli",
        "KeyFingerprint": "xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"
    }
]

秘密鍵のPermissionを変更

コマンド

chmod 600 ~/.ssh/${KEY_MATERIAL_FILE_NAME}
ls -al ~/.ssh/${KEY_MATERIAL_FILE_NAME}

結果

-rw------- 1 ec2-user ec2-user 1671 Aug  5 18:33 /home/ec2-user/.ssh/788063364413_firehose_jawsug_cli.pem

1.2. CloudFormation テンプレートの生成

テンプレートの作成

コマンド

CF_TEMPLATE_FILE_NAME="firehose_jawsug_cli.yml"

コマンド

cat << EOF > ${CF_TEMPLATE_FILE_NAME}
AWSTemplateFormatVersion: "2010-09-09"
Description: JAWS-UG CLI Kinesis Firehose Hands-on

Parameters: 
  VPCNetworkAddress: 
    Type: String
    Description: "Network Address on AWS"
    MinLength: 9
    MaxLength: 18
    # AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    Default: "10.0.0.0/16"
  PublicSubnetAddr: 
    Type: String
    Description: "Network Address on AWS"
    MinLength: 9
    MaxLength: 18
    # AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    Default: "10.0.0.0/24"
  KeyPairName:
    Type: AWS::EC2::KeyPair::KeyName


Resources:
  S3Bucket: 
    Type: "AWS::S3::Bucket"
  IAMRole: 
    Type: "AWS::IAM::Role"
    Properties: 
      RoleName: "service-role-firehose"
      AssumeRolePolicyDocument: 
        Version: "2012-10-17"
        Statement: 
          - 
            Effect: "Allow"
            Principal: 
              Service: 
                - "firehose.amazonaws.com"
            Action: 
              - "sts:AssumeRole"
      Path: "/"
  IAMPolicy:
    Type: "AWS::IAM::Policy"
    Properties: 
      PolicyName: "service-policy-firehose"
      PolicyDocument: 
        Version: "2012-10-17"
        Statement: 
          - 
            Effect: "Allow"
            Action: 
              - "s3:AbortMultipartUpload"
              - "s3:GetBucketLocation"
              - "s3:GetObject"
              - "s3:ListBucket"
              - "s3:ListBucketMultipartUploads"
              - "s3:PutObject"
            Resource:
              - !GetAtt S3Bucket.Arn
              - Fn::Join: 
                - "/"
                - 
                  - !GetAtt S3Bucket.Arn
                  - "*"
          - 
            Effect: "Allow"
            Action: 
              - "logs:PutLogEvents"
            Resource: "*"
      Roles:
        - Ref: IAMRole

  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VPCNetworkAddress
      Tags: 
        - 
          Key: "Name"
          Value: "KinesisFirehoseClient"
  IGW:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags: 
        - 
          Key: "Name"
          Value: "KinesisFirehoseClient"
  AttachIGW:
      Type: AWS::EC2::VPCGatewayAttachment
      Properties:
          VpcId:
              Ref: VPC
          InternetGatewayId:
              Ref: IGW
  PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties: 
      AvailabilityZone: 
        Fn::Select: 
          - 0
          - Fn::GetAZs: ""
      CidrBlock: !Ref PublicSubnetAddr
      MapPublicIpOnLaunch: true
      VpcId: 
        Ref: VPC
      Tags: 
        - 
          Key: "Name"
          Value: "Public"
  PublicRT:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId:
        Ref: VPC
      Tags: 
        - 
          Key: Name
          Value: Public
  PublicDefaultRoute:
    Type: AWS::EC2::Route
    Properties: 
      DestinationCidrBlock: "0.0.0.0/0"
      GatewayId: 
        Ref: IGW
      RouteTableId: 
        Ref: PublicRT
  PublicSubnetARouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId:
        Ref: PublicSubnet
      RouteTableId:
        Ref: PublicRT
  SecurityGroup:
    Type: "AWS::EC2::SecurityGroup"
    Properties: 
      GroupDescription: WebServer
      SecurityGroupEgress:
        - 
          IpProtocol: "-1"
          CidrIp: "0.0.0.0/0"
      SecurityGroupIngress:
        - 
          IpProtocol: "tcp"
          FromPort: 80
          ToPort: 80
          CidrIp: "0.0.0.0/0"        
        - 
          IpProtocol: "tcp"
          FromPort: 22
          ToPort: 22
          CidrIp: "0.0.0.0/0"
      VpcId:
        Ref: VPC    
  InstanceRole: 
    Type: "AWS::IAM::Role"
    Properties: 
      RoleName: "instance-role"
      AssumeRolePolicyDocument: 
        Version: "2012-10-17"
        Statement: 
          - 
            Effect: "Allow"
            Principal: 
              Service: 
                - "ec2.amazonaws.com"
                - "ssm.amazonaws.com"
            Action: 
              - "sts:AssumeRole"
      Path: "/"
      ManagedPolicyArns: 
        - "arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM"
        - "arn:aws:iam::aws:policy/AmazonKinesisFirehoseFullAccess"
  InstanceProfile: 
    Type: "AWS::IAM::InstanceProfile"
    Properties: 
      Path: "/"
      Roles: 
        - !Ref InstanceRole
  Instance:
    Type: "AWS::EC2::Instance"
    Properties:
      KeyName: 
        Ref: KeyPairName
      ImageId: ami-6df1e514
      InstanceType: t2.micro
      SecurityGroupIds: 
        - Ref: SecurityGroup
      SubnetId:
        Ref: PublicSubnet
      IamInstanceProfile:
        Ref: InstanceProfile
      BlockDeviceMappings: 
        - DeviceName: "/dev/sdm"
          Ebs: 
            VolumeType: "gp2"
            DeleteOnTermination: "true"
            VolumeSize: "8"
      UserData:
        Fn::Base64: |
          #!/bin/bash
          cd /tmp
          sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
          sudo yum install -y aws-kinesis-agent
          sudo chkconfig aws-kinesis-agent on
          sudo service aws-kinesis-agent start
          sudo yum install -y httpd
          sudo chkconfig httpd on
          sudo service httpd start
          sudo yum install npm --enablerepo=epel -y
          sudo npm install -g jsonlint

Outputs:
  S3BucketName:
    Value:
      Ref: S3Bucket
  IAMRoleARN:
    Value: !GetAtt IAMRole.Arn
  PublicIP:
    Value: !GetAtt Instance.PublicIp
EOF

cat ${CF_TEMPLATE_FILE_NAME}

CloudFormation テンプレートの検証

コマンド

aws cloudformation validate-template \
    --template-body file://${CF_TEMPLATE_FILE_NAME}

結果

{
    "CapabilitiesReason": "The following resource(s) require capabilities: [AWS::IAM::InstanceProfile, AWS::IAM::Role]",
    "Description": "JAWS-UG CLI Kinesis Firehose Hands-on",
    "Parameters": [
        {
            "NoEcho": false,
            "ParameterKey": "KeyPairName"
        },
        {
            "DefaultValue": "10.0.0.0/16",
            "NoEcho": false,
            "Description": "Network Address on AWS",
            "ParameterKey": "VPCNetworkAddress"
        },
        {
            "DefaultValue": "10.0.0.0/24",
            "NoEcho": false,
            "Description": "Network Address on AWS",
            "ParameterKey": "PublicSubnetAddr"
        }
    ],
    "Capabilities": [
        "CAPABILITY_NAMED_IAM"
    ]
}

1.3. CloudFormation Stackの作成

CloudFormation Stack名の指定

コマンド

CF_STACK_NAME="firehose-jawsug-cli"

同名CloudFormation Stackの不存在を確認

コマンド

aws cloudformation describe-stacks \
    --query "Stacks[?StackName==\`${CF_STACK_NAME}\`]"

結果

[]

CloudFormation Stackの作成

コマンド

aws cloudformation create-stack \
    --stack-name ${CF_STACK_NAME} \
    --template-body file://${CF_TEMPLATE_FILE_NAME} \
    --capabilities "CAPABILITY_NAMED_IAM" \
    --parameters ParameterKey=KeyPairName,ParameterValue=${KEY_PAIR_NAME},UsePreviousValue=false

結果

{
    "StackId": "arn:aws:cloudformation:us-west-2:XXXXXXXXXXXX:stack/firehose-jawsug-cli/8812e540-7a0e-11e7-aac3-50a68d01a68d"
}

CloudFormation Stackの作成完了を待機

5分程度で作成が完了すると思います。

コマンド

aws cloudformation wait stack-create-complete \
    --stack-name ${CF_STACK_NAME}

結果

（返値無し）

CloudFormation Stackの存在を確認

"StackStatus"が"CREATE_COMPLETE"になっていることを確認します。

コマンド

aws cloudformation describe-stacks \
    --stack-name ${CF_STACK_NAME}

結果

{
    "Stacks": [
        {
            "StackId": "arn:aws:cloudformation:us-west-2:XXXXXXXXXXXX:stack/firehose-jawsug-cli/4043bde0-7a16-11e7-8701-50a686be73ba",
            "Description": "JAWS-UG CLI Kinesis Firehose Hands-on",
            "Parameters": [
                {
                    "ParameterValue": "XXXXXXXXXXXX_firehose_jawsug_cli",
                    "ParameterKey": "KeyPairName"
                },
                {
                    "ParameterValue": "10.0.0.0/16",
                    "ParameterKey": "VPCNetworkAddress"
                },
                {
                    "ParameterValue": "10.0.0.0/24",
                    "ParameterKey": "PublicSubnetAddr"
                }
            ],
            "Tags": [],
            "Outputs": [
                {
                    "OutputKey": "PublicIP",
                    "OutputValue": "54.191.102.113"
                },
                {
                    "OutputKey": "IAMRoleARN",
                    "OutputValue": "arn:aws:iam::XXXXXXXXXXXX:role/service-role-firehose"
                },
                {
                    "OutputKey": "S3BucketName",
                    "OutputValue": "firehose-jawsug-cli-s3bucket-134czh3hcofqz"
                }
            ],
            "CreationTime": "2017-08-05T19:42:44.440Z",
            "Capabilities": [
                "CAPABILITY_NAMED_IAM"
            ],
            "StackName": "firehose-jawsug-cli",
            "NotificationARNs": [],
            "StackStatus": "CREATE_COMPLETE",
            "DisableRollback": false
        }
    ]
}

1.4. パラメータの確認

以降の手順で必要になるパラメータを抽出します。

• IAMロールARN
• S3バケット名
• パブリックIPアドレス

コマンド

OUTPUTKEY_ROLE_ARN="IAMRoleARN"
OUTPUTKEY_BUCKET_NAME="S3BucketName"
OUTPUTKEY_PUBLIC_IP_ADDRESS="PublicIP"

コマンド

ROLE_ARN=$(aws cloudformation describe-stacks \
    --stack-name ${CF_STACK_NAME} \
    --query "Stacks[].Outputs[?OutputKey==\`${OUTPUTKEY_ROLE_ARN}\`].OutputValue[]" \
    --output text) \
    && echo ${ROLE_ARN}

結果

arn:aws:iam::XXXXXXXXXXXX:role/service-role-firehose

コマンド

BUCKET_NAME=$(aws cloudformation describe-stacks \
    --stack-name ${CF_STACK_NAME} \
    --query "Stacks[].Outputs[?OutputKey==\`${OUTPUTKEY_BUCKET_NAME}\`].OutputValue[]" \
    --output text) \
    && echo ${BUCKET_NAME}

結果

firehose-jawsug-cli-s3bucket-134czh3hcofqz

コマンド

PUBLIC_IP_ADDRESS=$(aws cloudformation describe-stacks \
    --stack-name ${CF_STACK_NAME} \
    --query "Stacks[].Outputs[?OutputKey==\`${OUTPUTKEY_PUBLIC_IP_ADDRESS}\`].OutputValue[]" \
    --output text) \
    && echo ${PUBLIC_IP_ADDRESS}

結果

54.191.***.***

動作確認

パブリックIPアドレスにブラウザでアクセスします。

以上