Help us understand the problem. What is going on with this article?

[JAWS-UG CLI] Amazon Kinesis Firehose re:入門 (1) 事前準備(Source(Kinesis Agent)およびDestination(S3)の作成)

More than 3 years have passed since last update.

この記事について

JAWS-UG CLI専門支部 #90 Kinesis Firehose 復習編で実施するハンズオン用の手順書です。

前提条件

必要な権限

作業にあたっては、以下の権限を有したIAMユーザもしくはIAMロールを利用してください。

  • 以下のサービスに対するフルコントロール権限
    • Kinesis Firehose
    • IAM
    • EC2
    • S3
    • CloudWatch Logs
    • STS
    • (Lambda)
      • データの変換を行う場合
    • (KMS)
      • データの暗号化を行う場合

0. 準備

0.1. リージョンを指定

オレゴンリージョンで実施します。(東京マダー?)

コマンド
export AWS_DEFAULT_REGION="us-west-2"

0.2. 資格情報を確認

コマンド
aws configure list

インスタンスプロファイルを設定したEC2インスタンスでアクセスキーを設定せずに実行した場合、以下のようになります。

結果
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************QSAA         iam-role
secret_key     ****************c1xY         iam-role
    region                us-west-2              env    AWS_DEFAULT_REGION

0.3. バージョン確認

コマンド
aws --version
結果
aws-cli/1.11.129 Python/2.7.12 Linux/4.9.38-16.33.amzn1.x86_64 botocore/1.5.92

0.4. バージョンアップ(必要に応じて)

コマンド
sudo pip install -U awscli

1. 管理対象の構築

CloudFormationを利用して、Source(Kinesis AgentをインストールしたEC2インスタンス)とDestination(S3バケット)を作成します。

1.1. KeyPairの作成

EC2インスタンス用にKeyPairを作成します。

KeyPairの名前を指定

コマンド
AWS_ID=$(aws sts get-caller-identity \
    --query "Account" \
    --output text) \
    && echo ${AWS_ID}
コマンド
KEY_PAIR_NAME="${AWS_ID}_firehose_jawsug_cli"
KEY_MATERIAL_FILE_NAME=${KEY_PAIR_NAME}.pem

同名KeyPairの不存在を確認

コマンド
aws ec2 describe-key-pairs \
    --query "KeyPairs[?KeyName==\`${KEY_PAIR_NAME}\`]"
結果
[]

KeyPairの作成

コマンド
aws ec2 create-key-pair \
    --key-name ${KEY_PAIR_NAME} \
    --query "KeyMaterial" \
    --output text \
    > ~/.ssh/${KEY_MATERIAL_FILE_NAME} \
    && cat ~/.ssh/${KEY_MATERIAL_FILE_NAME}

KeyPairの存在を確認

コマンド
aws ec2 describe-key-pairs \
    --query "KeyPairs[?KeyName==\`${KEY_PAIR_NAME}\`]"
結果
[
    {
        "KeyName": "XXXXXXXXXXXX_firehose_jawsug_cli",
        "KeyFingerprint": "xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"
    }
]

秘密鍵のPermissionを変更

コマンド
chmod 600 ~/.ssh/${KEY_MATERIAL_FILE_NAME}
ls -al ~/.ssh/${KEY_MATERIAL_FILE_NAME}
結果
-rw------- 1 ec2-user ec2-user 1671 Aug  5 18:33 /home/ec2-user/.ssh/788063364413_firehose_jawsug_cli.pem

1.2. CloudFormation テンプレートの生成

テンプレートの作成

コマンド
CF_TEMPLATE_FILE_NAME="firehose_jawsug_cli.yml"
コマンド
cat << EOF > ${CF_TEMPLATE_FILE_NAME}
AWSTemplateFormatVersion: "2010-09-09"
Description: JAWS-UG CLI Kinesis Firehose Hands-on

Parameters: 
  VPCNetworkAddress: 
    Type: String
    Description: "Network Address on AWS"
    MinLength: 9
    MaxLength: 18
    # AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    Default: "10.0.0.0/16"
  PublicSubnetAddr: 
    Type: String
    Description: "Network Address on AWS"
    MinLength: 9
    MaxLength: 18
    # AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    Default: "10.0.0.0/24"
  KeyPairName:
    Type: AWS::EC2::KeyPair::KeyName


Resources:
  S3Bucket: 
    Type: "AWS::S3::Bucket"
  IAMRole: 
    Type: "AWS::IAM::Role"
    Properties: 
      RoleName: "service-role-firehose"
      AssumeRolePolicyDocument: 
        Version: "2012-10-17"
        Statement: 
          - 
            Effect: "Allow"
            Principal: 
              Service: 
                - "firehose.amazonaws.com"
            Action: 
              - "sts:AssumeRole"
      Path: "/"
  IAMPolicy:
    Type: "AWS::IAM::Policy"
    Properties: 
      PolicyName: "service-policy-firehose"
      PolicyDocument: 
        Version: "2012-10-17"
        Statement: 
          - 
            Effect: "Allow"
            Action: 
              - "s3:AbortMultipartUpload"
              - "s3:GetBucketLocation"
              - "s3:GetObject"
              - "s3:ListBucket"
              - "s3:ListBucketMultipartUploads"
              - "s3:PutObject"
            Resource:
              - !GetAtt S3Bucket.Arn
              - Fn::Join: 
                - "/"
                - 
                  - !GetAtt S3Bucket.Arn
                  - "*"
          - 
            Effect: "Allow"
            Action: 
              - "logs:PutLogEvents"
            Resource: "*"
      Roles:
        - Ref: IAMRole

  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VPCNetworkAddress
      Tags: 
        - 
          Key: "Name"
          Value: "KinesisFirehoseClient"
  IGW:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags: 
        - 
          Key: "Name"
          Value: "KinesisFirehoseClient"
  AttachIGW:
      Type: AWS::EC2::VPCGatewayAttachment
      Properties:
          VpcId:
              Ref: VPC
          InternetGatewayId:
              Ref: IGW
  PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties: 
      AvailabilityZone: 
        Fn::Select: 
          - 0
          - Fn::GetAZs: ""
      CidrBlock: !Ref PublicSubnetAddr
      MapPublicIpOnLaunch: true
      VpcId: 
        Ref: VPC
      Tags: 
        - 
          Key: "Name"
          Value: "Public"
  PublicRT:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId:
        Ref: VPC
      Tags: 
        - 
          Key: Name
          Value: Public
  PublicDefaultRoute:
    Type: AWS::EC2::Route
    Properties: 
      DestinationCidrBlock: "0.0.0.0/0"
      GatewayId: 
        Ref: IGW
      RouteTableId: 
        Ref: PublicRT
  PublicSubnetARouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId:
        Ref: PublicSubnet
      RouteTableId:
        Ref: PublicRT
  SecurityGroup:
    Type: "AWS::EC2::SecurityGroup"
    Properties: 
      GroupDescription: WebServer
      SecurityGroupEgress:
        - 
          IpProtocol: "-1"
          CidrIp: "0.0.0.0/0"
      SecurityGroupIngress:
        - 
          IpProtocol: "tcp"
          FromPort: 80
          ToPort: 80
          CidrIp: "0.0.0.0/0"        
        - 
          IpProtocol: "tcp"
          FromPort: 22
          ToPort: 22
          CidrIp: "0.0.0.0/0"
      VpcId:
        Ref: VPC    
  InstanceRole: 
    Type: "AWS::IAM::Role"
    Properties: 
      RoleName: "instance-role"
      AssumeRolePolicyDocument: 
        Version: "2012-10-17"
        Statement: 
          - 
            Effect: "Allow"
            Principal: 
              Service: 
                - "ec2.amazonaws.com"
                - "ssm.amazonaws.com"
            Action: 
              - "sts:AssumeRole"
      Path: "/"
      ManagedPolicyArns: 
        - "arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM"
        - "arn:aws:iam::aws:policy/AmazonKinesisFirehoseFullAccess"
  InstanceProfile: 
    Type: "AWS::IAM::InstanceProfile"
    Properties: 
      Path: "/"
      Roles: 
        - !Ref InstanceRole
  Instance:
    Type: "AWS::EC2::Instance"
    Properties:
      KeyName: 
        Ref: KeyPairName
      ImageId: ami-6df1e514
      InstanceType: t2.micro
      SecurityGroupIds: 
        - Ref: SecurityGroup
      SubnetId:
        Ref: PublicSubnet
      IamInstanceProfile:
        Ref: InstanceProfile
      BlockDeviceMappings: 
        - DeviceName: "/dev/sdm"
          Ebs: 
            VolumeType: "gp2"
            DeleteOnTermination: "true"
            VolumeSize: "8"
      UserData:
        Fn::Base64: |
          #!/bin/bash
          cd /tmp
          sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
          sudo yum install -y aws-kinesis-agent
          sudo chkconfig aws-kinesis-agent on
          sudo service aws-kinesis-agent start
          sudo yum install -y httpd
          sudo chkconfig httpd on
          sudo service httpd start
          sudo yum install npm --enablerepo=epel -y
          sudo npm install -g jsonlint

Outputs:
  S3BucketName:
    Value:
      Ref: S3Bucket
  IAMRoleARN:
    Value: !GetAtt IAMRole.Arn
  PublicIP:
    Value: !GetAtt Instance.PublicIp
EOF

cat ${CF_TEMPLATE_FILE_NAME}

CloudFormation テンプレートの検証

コマンド
aws cloudformation validate-template \
    --template-body file://${CF_TEMPLATE_FILE_NAME}
結果
{
    "CapabilitiesReason": "The following resource(s) require capabilities: [AWS::IAM::InstanceProfile, AWS::IAM::Role]",
    "Description": "JAWS-UG CLI Kinesis Firehose Hands-on",
    "Parameters": [
        {
            "NoEcho": false,
            "ParameterKey": "KeyPairName"
        },
        {
            "DefaultValue": "10.0.0.0/16",
            "NoEcho": false,
            "Description": "Network Address on AWS",
            "ParameterKey": "VPCNetworkAddress"
        },
        {
            "DefaultValue": "10.0.0.0/24",
            "NoEcho": false,
            "Description": "Network Address on AWS",
            "ParameterKey": "PublicSubnetAddr"
        }
    ],
    "Capabilities": [
        "CAPABILITY_NAMED_IAM"
    ]
}

1.3. CloudFormation Stackの作成

CloudFormation Stack名の指定

コマンド
CF_STACK_NAME="firehose-jawsug-cli"

同名CloudFormation Stackの不存在を確認

コマンド
aws cloudformation describe-stacks \
    --query "Stacks[?StackName==\`${CF_STACK_NAME}\`]"
結果
[]

CloudFormation Stackの作成

コマンド
aws cloudformation create-stack \
    --stack-name ${CF_STACK_NAME} \
    --template-body file://${CF_TEMPLATE_FILE_NAME} \
    --capabilities "CAPABILITY_NAMED_IAM" \
    --parameters ParameterKey=KeyPairName,ParameterValue=${KEY_PAIR_NAME},UsePreviousValue=false
結果
{
    "StackId": "arn:aws:cloudformation:us-west-2:XXXXXXXXXXXX:stack/firehose-jawsug-cli/8812e540-7a0e-11e7-aac3-50a68d01a68d"
}

CloudFormation Stackの作成完了を待機

5分程度で作成が完了すると思います。

コマンド
aws cloudformation wait stack-create-complete \
    --stack-name ${CF_STACK_NAME}
結果
(返値無し)

CloudFormation Stackの存在を確認

"StackStatus"が"CREATE_COMPLETE"になっていることを確認します。

コマンド
aws cloudformation describe-stacks \
    --stack-name ${CF_STACK_NAME}
結果
{
    "Stacks": [
        {
            "StackId": "arn:aws:cloudformation:us-west-2:XXXXXXXXXXXX:stack/firehose-jawsug-cli/4043bde0-7a16-11e7-8701-50a686be73ba",
            "Description": "JAWS-UG CLI Kinesis Firehose Hands-on",
            "Parameters": [
                {
                    "ParameterValue": "XXXXXXXXXXXX_firehose_jawsug_cli",
                    "ParameterKey": "KeyPairName"
                },
                {
                    "ParameterValue": "10.0.0.0/16",
                    "ParameterKey": "VPCNetworkAddress"
                },
                {
                    "ParameterValue": "10.0.0.0/24",
                    "ParameterKey": "PublicSubnetAddr"
                }
            ],
            "Tags": [],
            "Outputs": [
                {
                    "OutputKey": "PublicIP",
                    "OutputValue": "54.191.102.113"
                },
                {
                    "OutputKey": "IAMRoleARN",
                    "OutputValue": "arn:aws:iam::XXXXXXXXXXXX:role/service-role-firehose"
                },
                {
                    "OutputKey": "S3BucketName",
                    "OutputValue": "firehose-jawsug-cli-s3bucket-134czh3hcofqz"
                }
            ],
            "CreationTime": "2017-08-05T19:42:44.440Z",
            "Capabilities": [
                "CAPABILITY_NAMED_IAM"
            ],
            "StackName": "firehose-jawsug-cli",
            "NotificationARNs": [],
            "StackStatus": "CREATE_COMPLETE",
            "DisableRollback": false
        }
    ]
}

1.4. パラメータの確認

以降の手順で必要になるパラメータを抽出します。

  • IAMロールARN
  • S3バケット名
  • パブリックIPアドレス
コマンド
OUTPUTKEY_ROLE_ARN="IAMRoleARN"
OUTPUTKEY_BUCKET_NAME="S3BucketName"
OUTPUTKEY_PUBLIC_IP_ADDRESS="PublicIP"
コマンド
ROLE_ARN=$(aws cloudformation describe-stacks \
    --stack-name ${CF_STACK_NAME} \
    --query "Stacks[].Outputs[?OutputKey==\`${OUTPUTKEY_ROLE_ARN}\`].OutputValue[]" \
    --output text) \
    && echo ${ROLE_ARN}
結果
arn:aws:iam::XXXXXXXXXXXX:role/service-role-firehose
コマンド
BUCKET_NAME=$(aws cloudformation describe-stacks \
    --stack-name ${CF_STACK_NAME} \
    --query "Stacks[].Outputs[?OutputKey==\`${OUTPUTKEY_BUCKET_NAME}\`].OutputValue[]" \
    --output text) \
    && echo ${BUCKET_NAME}
結果
firehose-jawsug-cli-s3bucket-134czh3hcofqz
コマンド
PUBLIC_IP_ADDRESS=$(aws cloudformation describe-stacks \
    --stack-name ${CF_STACK_NAME} \
    --query "Stacks[].Outputs[?OutputKey==\`${OUTPUTKEY_PUBLIC_IP_ADDRESS}\`].OutputValue[]" \
    --output text) \
    && echo ${PUBLIC_IP_ADDRESS}
結果
54.191.***.***

動作確認

パブリックIPアドレスにブラウザでアクセスします。

以上

Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
Comments
No comments
Sign up for free and join this conversation.
If you already have a Qiita account
Why do not you register as a user and use Qiita more conveniently?
You need to log in to use this function. Qiita can be used more conveniently after logging in.
You seem to be reading articles frequently this month. Qiita can be used more conveniently after logging in.
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
ユーザーは見つかりませんでした