aws s3 syncするシェルスクリプトでワイルドカードでexcludeしたときのメモ

aws s3 syncをしようとしたときに、--excludeを大量に書きたくないので、excludeしたいファイルを別ファイルに書いて、シェルスクリプト内で読み込んでで実行するようにしたところ、*(アスタリスク)が自動展開されて困ったので対応のメモ

除外ファイル一覧

.s3_sync_exclude

Gemfile
Gemfile.lock
.git/*
.gitignore

最初に作ったシェルスクリプト

sync_dryrun.sh

#!/bin/bash
dir=$(cd $(dirname ${0}); pwd)
file=${0##*/}
command="aws s3 sync ${dir}/ s3://bucket_name/ --delete --dryrun --exclude .s3_sync_exclude --exclude ${file}"
while read exclude
do
  command="${command} --exclude ${exclude}"
done < ${dir}/.s3_sync_exclude
set -x
${command}

実行結果

% ./sync_dryrun.sh
+ aws s3 sync /path/to/dir/ s3://bucket_name/ --delete --dryrun --exclude .s3_sync_exclude --exclude sync_dryrun.sh --exclude Gemfile --exclude Gemfile.lock --exclude .git/HEAD .git/branches .git/config .git/description .git/hooks .git/info .git/objects .git/refs --exclude .gitignore

Unknown options: .git/branches, .git/config, .git/description, .git/hooks, .git/info, .git/objects, .git/refs
zsh: exit 255   ./sync_dryrun.sh

*(アスタリスク)が展開されて大変なことになった…

*(アスタリスク)が展開されないようにダブルクォートで囲ってみる

sync_dryrun.sh

#!/bin/bash
dir=$(cd $(dirname ${0}); pwd)
file=${0##*/}
command="aws s3 sync ${dir}/ s3://bucket_name/ --delete --dryrun --exclude .s3_sync_exclude --exclude ${file}"
while read exclude
do
  command="${command} --exclude \"${exclude}\""
done < ${dir}/.s3_sync_exclude
set -x
${command}

実行結果

% ./sync_dryrun.sh
+ aws s3 sync /path/to/dir/ s3://bucket_name/ --delete --dryrun --exclude .s3_sync_exclude --exclude sync_dryrun.sh --exclude '"Gemfile"' --exclude '"Gemfile.lock"' --exclude '".git/*"' --exclude '".gitignore"'
(dryrun) upload: .git/HEAD to s3://bucket_name/.git/HEAD
(dryrun) upload: .git/config to s3://bucket_name/.git/config
(dryrun) upload: .git/description to s3://bucket_name/.git/description
(dryrun) upload: .git/hooks/applypatch-msg.sample to s3://bucket_name/.git/hooks/applypatch-msg.sample
(dryrun) upload: .git/hooks/post-update.sample to s3://bucket_name/.git/hooks/post-update.sample
(dryrun) upload: .git/hooks/commit-msg.sample to s3://bucket_name/.git/hooks/commit-msg.sample
(dryrun) upload: .git/hooks/pre-commit.sample to s3://bucket_name/.git/hooks/pre-commit.sample
(dryrun) upload: .git/hooks/pre-applypatch.sample to s3://bucket_name/.git/hooks/pre-applypatch.sample
(dryrun) upload: .git/hooks/pre-push.sample to s3://bucket_name/.git/hooks/pre-push.sample
(dryrun) upload: .git/hooks/pre-rebase.sample to s3://bucket_name/.git/hooks/pre-rebase.sample
(dryrun) upload: .git/hooks/prepare-commit-msg.sample to s3://bucket_name/.git/hooks/prepare-commit-msg.sample
(dryrun) upload: .git/hooks/update.sample to s3://bucket_name/.git/hooks/update.sample
(dryrun) upload: .git/info/exclude to s3://bucket_name/.git/info/exclude
(dryrun) upload: ./Gemfile to s3://bucket_name/Gemfile
(dryrun) upload: ./Gemfile.lock to s3://bucket_name/Gemfile.lock

""で囲った文字列がさらに''で囲まれている
引数として""を含んだ文字列が渡されてexcludeしてもらえない

シングルクォートで囲んでみる

sync_dryrun.sh

#!/bin/bash
dir=$(cd $(dirname ${0}); pwd)
file=${0##*/}
command="aws s3 sync ${dir}/ s3://bucket_name/ --delete --dryrun --exclude .s3_sync_exclude --exclude ${file}"
while read exclude
do
  command="${command} --exclude '${exclude}'"
done < ${dir}/.s3_sync_exclude
set -x
${command}

実行結果

% ./sync_dryrun.sh
+ aws s3 sync /path/to/dir/ s3://bucket_name/ --delete --dryrun --exclude .s3_sync_exclude --exclude sync_dryrun.sh --exclude ''\''Gemfile'\''' --exclude ''\''Gemfile.lock'\''' --exclude ''\''.git/*'\''' --exclude ''\''.gitignore'\'''
(dryrun) upload: .git/HEAD to s3://bucket_name/.git/HEAD
(dryrun) upload: .git/config to s3://bucket_name/.git/config
(dryrun) upload: .git/description to s3://bucket_name/.git/description
(dryrun) upload: .git/hooks/applypatch-msg.sample to s3://bucket_name/.git/hooks/applypatch-msg.sample
(dryrun) upload: .git/hooks/post-update.sample to s3://bucket_name/.git/hooks/post-update.sample
(dryrun) upload: .git/hooks/commit-msg.sample to s3://bucket_name/.git/hooks/commit-msg.sample
(dryrun) upload: .git/hooks/pre-commit.sample to s3://bucket_name/.git/hooks/pre-commit.sample
(dryrun) upload: .git/hooks/pre-applypatch.sample to s3://bucket_name/.git/hooks/pre-applypatch.sample
(dryrun) upload: .git/hooks/pre-push.sample to s3://bucket_name/.git/hooks/pre-push.sample
(dryrun) upload: .git/hooks/pre-rebase.sample to s3://bucket_name/.git/hooks/pre-rebase.sample
(dryrun) upload: .git/hooks/prepare-commit-msg.sample to s3://bucket_name/.git/hooks/prepare-commit-msg.sample
(dryrun) upload: .git/hooks/update.sample to s3://bucket_name/.git/hooks/update.sample
(dryrun) upload: .git/info/exclude to s3://bucket_name/.git/info/exclude
(dryrun) upload: ./Gemfile to s3://bucket_name/Gemfile
(dryrun) upload: ./Gemfile.lock to s3://bucket_name/Gemfile.lock

なんだかわからないけど'が大量に追加された。excludeされるはずもなく。。。

ダブルクォートに戻してからevalしてみる (解決編)

sync_dryrun.sh

#!/bin/bash
dir=$(cd $(dirname ${0}); pwd)
file=${0##*/}
command="aws s3 sync ${dir}/ s3://bucket_name/ --delete --dryrun --exclude .s3_sync_exclude --exclude ${file}"
while read exclude
do
  command="${command} --exclude \"${exclude}\""
done < ${dir}/.s3_sync_exclude
set -x
eval "${command}"

実行結果

% ./sync_dryrun.sh
+ eval 'aws s3 sync /path/to/dir/ s3://bucket_name/ --delete --dryrun --exclude .s3_sync_exclude --exclude sync_dryrun.sh --exclude "Gemfile" --exclude "Gemfile.lock" --exclude ".git/*" --exclude ".gitignore"'
++ aws s3 sync /path/to/dir/ s3://bucket_name/ --delete --dryrun --exclude .s3_sync_exclude --exclude sync_dryrun.sh --exclude Gemfile --exclude Gemfile.lock --exclude '.git/*' --exclude .gitignore

""がなくなったり、*が含まれる文字列が''で囲まれたりしてるけど、正しい動きをしてくれた。