LoginSignup
3
4

More than 5 years have passed since last update.

S3上ファイル存在チェックスクリプト

Last updated at Posted at 2017-10-27

EmbulkでS3上のファイルを読み込む際、ファイルがなくてもエラーにならないので、事前にチェックすることにした
awsコマンドが使用できる前提
環境変数でリトライ回数・間隔を指定可能

関連:
https://qiita.com/pilot/items/3014910fee5dce0ce3e3

check_s3_file_exists.sh
#!/bin/bash

if [ $# -ne 3 ]; then
  echo "Usage: $0 bucket work_dir file" >&2
  exit 1
fi

bucket=$1
work_dir=$2
file=$3

myname=`basename $0 .sh`
resultfile=${work_dir}/${myname}_result_$$.txt

retry_count=${RETRY_COUNT:-5}
retry_unit_seconds=${RETRY_UNIT_SECONDS:-10}
#echo retry_count=${retry_count}, retry_unit_seconds=${retry_unit_seconds} # for debug

count=1
while :
do
  aws s3api list-objects --bucket ${bucket} > ${resultfile}
  status=$?
  if [ ${status} -eq 0 ]; then
    break
  fi
  if [ ${count} -eq ${retry_count} ]; then
    echo Retry over >&2
    rm -f ${resultfile}
    exit 1
  fi
  sleep_seconds=$((count * retry_unit_seconds))
  count=$((count + 1))
  echo Retry after ${sleep_seconds} seconds, next count is ${count} >&2
  sleep ${sleep_seconds}
done

grep '^ *"Key" *: *"'${file}'" *, *$' ${resultfile} > /dev/null
status=$?
rm -f ${resultfile}

if [ ${status} != "0" ]; then
  echo "File not found: "${file} >&2
  exit 1
fi

echo "File found: "${file}

exit 0
3
4
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
3
4