search
LoginSignup
0

posted at

DynamoDBデータのCSV出力方法 メモ

  • DynamoDBのデータをCSV出力する方法についてメモする。

テストデータ準備

  • docker-compose.yml

    • DynamoDBローカルを用いてテスト環境を作る。

      version: '3'
      
      services:
        dynamodb-local:
          container_name: dynamodb-local
          image: amazon/dynamodb-local:latest
          user: root
          command: -jar DynamoDBLocal.jar -sharedDb -dbPath /data
          volumes:
            - dynamodb-data:/data
          ports:
            - 8000:8000
          networks:
            - dynamodb-network
      
        dynamodb-admin:
          container_name: dynamodb-admin
          image: aaronshaf/dynamodb-admin:latest
          environment:
            - DYNAMO_ENDPOINT=dynamodb-local:8000
          ports:
            - 8001:8001
          depends_on:
            - dynamodb-local
          networks:
            - dynamodb-network
      
      volumes:
        dynamodb-data:
      
      networks:
        dynamodb-network:
          driver: bridge
      
  • create_table.sh

    • テスト用テーブル作成用スクリプト

      # テーブルtest_data作成
      aws dynamodb create-table --cli-input-json file://data/test_data_schema.json  --endpoint-url http://localhost:8000 --profile dynamodb
      echo Creating Table "test_data" Succeeded 
      # TTL有効化
      aws dynamodb update-time-to-live --table-name test_data --time-to-live-specification "Enabled=true, AttributeName=ttl" --endpoint-url http://localhost:8000 --profile dynamodb
      
      # 初期データ登録
      
      # TTL設定(現在時刻+1時間)
      NOW=$(date +%s)
      TTL=$(date +%s -d "1 hour")
      echo NOW=${NOW}
      echo TTL=${TTL}
      # 初期データJSON
      tempfile=$(mktemp)
      cat  > ${tempfile} << EOS
      {
          "test_data": [
              {
                  "PutRequest": {
                      "Item": {
                          "email": {
                              "S": "test@example.com"
                          },
                          "ttl": {
                              "N": "${TTL}"
                          },
                          "id": {
                              "S": "abcde12345"
                          }
                      }
                  }
              },
              {
                  "PutRequest": {
                      "Item": {
                          "email": {
                              "S": "test2@example.com"
                          },
                          "ttl": {
                              "N": "${TTL}"
                          },
                          "id": {
                              "S": "fghij98765"
                          }
                      }
                  }
              }
          ]
      }
      EOS
      aws dynamodb batch-write-item --request-items file://${tempfile} --endpoint-url http://localhost:8000 --profile dynamodb
      echo Putting Initial Data Succeeded
      rm -f ${tempfile}
      
  • test_data_schema.json

    • テストテーブル定義

      {
          "AttributeDefinitions": [
            {
              "AttributeName": "id",
              "AttributeType": "S"
            },
            {
              "AttributeName": "email",
              "AttributeType": "S"
            }
          ],
          "TableName": "test_data",
          "KeySchema": [
            {
              "AttributeName": "id",
              "KeyType": "HASH"
            },
            {
              "AttributeName": "email",
              "KeyType": "RANGE"
            }
          ],
          "ProvisionedThroughput": {
              "ReadCapacityUnits": 5,
              "WriteCapacityUnits": 5
          }
        }
      

AWS CLI+jqコマンドによるCSV出力

  • テスト環境起動

    docker-compose up
    
  • テストテーブル作成+テストデータ投入

    sh create_table.sh
    
  • CSV出力

    aws dynamodb scan --table-name test_data --endpoint-url http://localhost:8000 --profile dynamodb\
    | jq -r '.Items[] | [.id.S, .email.S, .ttl.N] | @csv' > backup.csv
    
  • CSV出力結果backup.csv

    "abcde12345","test@example.com","1651981544"
    "fghij98765","test2@example.com","1651981544"
    

その他

  • コンソールからの「CSVへエクスポート」でもCSV出力可能。

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
What you can do with signing up
0