LoginSignup
0

More than 1 year has passed since last update.

DynamoDBデータのCSV出力方法 メモ

Posted at
  • DynamoDBのデータをCSV出力する方法についてメモする。

テストデータ準備

  • docker-compose.yml

    • DynamoDBローカルを用いてテスト環境を作る。

      version: '3'
      
      services:
        dynamodb-local:
          container_name: dynamodb-local
          image: amazon/dynamodb-local:latest
          user: root
          command: -jar DynamoDBLocal.jar -sharedDb -dbPath /data
          volumes:
            - dynamodb-data:/data
          ports:
            - 8000:8000
          networks:
            - dynamodb-network
      
        dynamodb-admin:
          container_name: dynamodb-admin
          image: aaronshaf/dynamodb-admin:latest
          environment:
            - DYNAMO_ENDPOINT=dynamodb-local:8000
          ports:
            - 8001:8001
          depends_on:
            - dynamodb-local
          networks:
            - dynamodb-network
      
      volumes:
        dynamodb-data:
      
      networks:
        dynamodb-network:
          driver: bridge
      
  • create_table.sh

    • テスト用テーブル作成用スクリプト

      # テーブルtest_data作成
      aws dynamodb create-table --cli-input-json file://data/test_data_schema.json  --endpoint-url http://localhost:8000 --profile dynamodb
      echo Creating Table "test_data" Succeeded 
      # TTL有効化
      aws dynamodb update-time-to-live --table-name test_data --time-to-live-specification "Enabled=true, AttributeName=ttl" --endpoint-url http://localhost:8000 --profile dynamodb
      
      # 初期データ登録
      
      # TTL設定(現在時刻+1時間)
      NOW=$(date +%s)
      TTL=$(date +%s -d "1 hour")
      echo NOW=${NOW}
      echo TTL=${TTL}
      # 初期データJSON
      tempfile=$(mktemp)
      cat  > ${tempfile} << EOS
      {
          "test_data": [
              {
                  "PutRequest": {
                      "Item": {
                          "email": {
                              "S": "test@example.com"
                          },
                          "ttl": {
                              "N": "${TTL}"
                          },
                          "id": {
                              "S": "abcde12345"
                          }
                      }
                  }
              },
              {
                  "PutRequest": {
                      "Item": {
                          "email": {
                              "S": "test2@example.com"
                          },
                          "ttl": {
                              "N": "${TTL}"
                          },
                          "id": {
                              "S": "fghij98765"
                          }
                      }
                  }
              }
          ]
      }
      EOS
      aws dynamodb batch-write-item --request-items file://${tempfile} --endpoint-url http://localhost:8000 --profile dynamodb
      echo Putting Initial Data Succeeded
      rm -f ${tempfile}
      
  • test_data_schema.json

    • テストテーブル定義

      {
          "AttributeDefinitions": [
            {
              "AttributeName": "id",
              "AttributeType": "S"
            },
            {
              "AttributeName": "email",
              "AttributeType": "S"
            }
          ],
          "TableName": "test_data",
          "KeySchema": [
            {
              "AttributeName": "id",
              "KeyType": "HASH"
            },
            {
              "AttributeName": "email",
              "KeyType": "RANGE"
            }
          ],
          "ProvisionedThroughput": {
              "ReadCapacityUnits": 5,
              "WriteCapacityUnits": 5
          }
        }
      

AWS CLI+jqコマンドによるCSV出力

  • テスト環境起動

    docker-compose up
    
  • テストテーブル作成+テストデータ投入

    sh create_table.sh
    
  • CSV出力

    aws dynamodb scan --table-name test_data --endpoint-url http://localhost:8000 --profile dynamodb\
    | jq -r '.Items[] | [.id.S, .email.S, .ttl.N] | @csv' > backup.csv
    
  • CSV出力結果backup.csv

    "abcde12345","test@example.com","1651981544"
    "fghij98765","test2@example.com","1651981544"
    

その他

  • コンソールからの「CSVへエクスポート」でもCSV出力可能。

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
0