LoginSignup
8
8

More than 5 years have passed since last update.

@takeru@github

無限の容量をもつAmazon Drive

Last updated at Posted at 2016-11-15

無限の容量をもつAmazon Driveに大切なデータをバックアップしたい。
で、
ほんとにちゃんと出来てるのか確かめたい。

Amazon Drive

https://www.amazon.com/clouddrive
14959021_10154778480681869_865178105_o.png

rclone

brew install rclone
% rclone -V
rclone v1.34-DEV


Modified time and MD5SUMs
Amazon Drive doesn’t allow modification times to be changed via the API so these won’t be accurate or used for syncing.

It does store MD5SUMs so for a more accurate sync, you can use the --checksum flag.

AmazonDriveはファイルの更新時刻を保存できないからMD5でチェックすべし。

セットアップ

http://rclone.org/amazonclouddrive/
に従って設定。

~/.rclone.confに設定が保存される。

~/.rclone.conf
[amz]
type = amazon cloud drive
client_id =
client_secret =
token = {.....}

amazon.co.jpとamazon.comのログインがコンフリクトして挙動不審だったけど何回もやり直したらできた。謎。

コピー

rclone -v copy ~/Pictures/写真\ Library.photoslibrary/ amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/

このときはv1.33だった。
420GBで1週間かかった。
image

確認

rclone check --checksum ~/Pictures/写真\ Library.photoslibrary/ amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/ 2>&1 | tee check_`ts`.txt

2016/11/15 13:24:33 Local file system at /Users/takeru/Pictures/写真 Library.photoslibrary: Building file list
2016/11/15 13:24:33 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Building file list
2016/11/15 13:25:33
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      1m6.4s

......

2016/11/16 02:59:33
Transferred:      0 Bytes (0 Bytes/s)
Errors:              3933
Checks:            324417
Transferred:            0
Elapsed time:  13h35m6.4s
Checking:
 * Database/Faces/Detected/%tE6BEKPQZubOu%1nMo+tw.apdetected
 * Database/Places/B9zGz6YFSHinJ9eJI+6%bw.applace
 * Database/Versions/2011/05/14/20110514-125249/hopKwBNjR+W77Li7vfxUhQ/Master.apmaster
 * Database/Versions/2013/08/22/20130822-132448/wC6E6H9WR%iXgAqV0dDmEw/Version-0.apversion
 * Database/Versions/2013/11/12/20131112-135919/sFpvdWdgTi6n9zWkjvQYtg/Master.apmaster
 * Database/Versions/2015/05/05/20150505-171157/NDtysoLkSDmMs0SZ6xgdgw/Version-0.apversion
 * Masters/2014/09/17/20140917-135845/P1120427.JPG
 * resources/modelresources/117/72/uHJ%6BPWQWmg2BHFAhu81Q/P1250301.jpg

2016/11/16 03:00:08 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': 3933 differences found
2016/11/16 03:00:08 Failed to check: 3933 differences found

13.5時間かかった。

% grep "File not in amazon drive" check_20161115_132426.txt | wc -l
    3882
% grep "Sizes differ" check_20161115_132426.txt | wc -l
      40
% grep "MD5 differ" check_20161115_132426.txt | wc -l
       7
% grep "File not in Local file system" check_20161115_132426.txt | wc -l
       3
% grep "Failed to calculate src hash" check_20161115_132426.txt | wc -l
       1

3882+40+7+3+1 = 3933

copyとcheckの間に取り込みをしているので差がある。

リスト取得

checkじゃなくて手動で確認するにはlsとmd5sumを使えそう。

% rclone ls -c amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/
     2048 iPhotoAux.db
      341 ProjectDBVersion.plist
       12 Library6.iPhoto
       16 .ipspot_update
       22 Library.iPhoto
    20097 com.apple.iPhoto.plist
       51 iPhoto.ipspot
        1 iPhotoLock.data
     6144 iPhotoMain.db
       22 Library.data
   950272 Projects.db
   149641 Thumbnails/IMG_2258_1024.jpg
    65982 Thumbnails/IMG_2260.jpg
    36507 Thumbnails/IMG_2259.jpg
   340738 Thumbnails/IMG_2256_1024.jpg
    36298 Thumbnails/IMG_2255.jpg
   147391 Thumbnails/IMG_2259_1024.jpg
    68173 Thumbnails/IMG_2262.jpg
    67724 Thumbnails/IMG_2261.jpg
   242106 Thumbnails/IMG_2263_1024.jpg
    50023 Thumbnails/IMG_2263.jpg
    72239 Thumbnails/IMG_2264.jpg
   361674 Thumbnails/IMG_2261_1024.jpg
    73621 Thumbnails/iMovie Data
   399933 Thumbnails/IMG_2264_1024.jpg
   305229 Thumbnails/IMG_2257_1024.jpg
    36485 Thumbnails/IMG_2258.jpg
...

13:43:07 tkrimac2:~% rclone md5sum amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/
10f81bf64edc6c17e8c2c4bcfa1ff2c1  iPhotoAux.db
3d4a1ff0b4efbc59cbe1e05e6b1f7dfd  ProjectDBVersion.plist
9174a025431e8cd040f310f6e9383942  Library6.iPhoto
5bcd17a07bb9b4cd0a739da0501d03b0  .ipspot_update
8c52f4fe8975dbf90921baf1e049a276  Library.data
7215ee9c7d9dc229d2921a40e899ec5f  iPhotoLock.data
8e81d2b8fd3f851cb0e2d6dfe1c7f47d  com.apple.iPhoto.plist
710683e6a2520eb2605757f2c3c3027f  iPhotoMain.db
011b040a3a0de00abd50aee2f81f4ce2  iPhoto.ipspot
008d9f97c161f149c098f767f29f5c4b  Projects.db
8c52f4fe8975dbf90921baf1e049a276  Library.iPhoto
d8d97fc3a0c60e97ac4a66e57a1778f8  Thumbnails/IMG_2257_1024.jpg
a3680df33b27355bd18967860043a5b3  Thumbnails/IMG_2260.jpg
5d59b305703d861e299e06996509442d  Thumbnails/IMG_2258.jpg
ec27b1aa9afe097d397a0538d41d5006  Thumbnails/IMG_2256.jpg
674b50262670adf9b4198d0db8cb3d17  Thumbnails/IMG_2257.jpg
2ea4bae372e10f7677cfebf252d7ce00  Thumbnails/IMG_2255_1024.jpg
f202a9f4343aeac363fa4be8d21a0fc4  Thumbnails/IMG_2262.jpg
fc2e86d04eda8bd2c6b543ed9d266c52  Thumbnails/IMG_2262_1024.jpg
b1e0f5eebe2724d7627eb009a0f6ec67  Thumbnails/IMG_2263_1024.jpg
1acf5f06a9ea072d2de894e312f0be62  Thumbnails/IMG_2263.jpg
34e7f1cc0c191827cd69a2315400e7f8  Thumbnails/IMG_2264.jpg
b8fa406478f6e21d411f9d60e7a901e7  Thumbnails/IMG_2261_1024.jpg
01af20f2141bc7e45d299ee582126170  Thumbnails/iMovie Data
ca38101864190d4be827df219815052d  Thumbnails/IMG_2264_1024.jpg
3e98c476cf43639dbea49121450d8063  Thumbnails/IMG_2258_1024.jpg
611e1d0692dfe5535be10a0990c0608c  Thumbnails/IMG_2259.jpg
fb624e6bef00061223d0b3f8ce0d4850  Thumbnails/IMG_2256_1024.jpg
...

lsでファイルサイズ、md5sumでハッシュ値のリストが取れるのでこれで確認できそう。

sync

sync --dry-run

% rclone sync --checksum --dry-run ~/Pictures/写真\ Library.photoslibrary/ amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/ 2>&1 | tee sync-dry_`ts`.txt


2016/11/16 10:35:42 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Not making directory as dry run is set
2016/11/16 10:36:42
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:        1m5s

.....

2016/11/16 20:05:42
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:            323414
Transferred:         3932
Elapsed time:     9h30m5s
Checking:
 * Database/Versions/2014/04/22/20140422-110751/Vu2gSEhAQ0epbvs3tBrGAg/Version-0.apversion
 * Database/Versions/2014/10/05/20141005-123438/Zm7l3UzmTYG+WXNYXXhhog/Master.apmaster
 * Database/Versions/2014/10/05/20141005-123438/Zm7l3UzmTYG+WXNYXXhhog/Version-0.apversion
 * Database/Versions/2014/10/05/20141005-123438/Zm7l3UzmTYG+WXNYXXhhog/Version-1.apversion
 * Thumbnails/2014/04/22/20140422-110751/VZUHErpbTXuf5ou4RVYppw/P1090602_1024.jpg
 * Thumbnails/2014/04/22/20140422-110751/VyYA5CICSBSdI8uyYn7KQw/P1100171_1024.jpg
 * Thumbnails/2014/04/22/20140422-110751/W%BECsCHTIa2icv4VrBgfg/P1100271.jpg
 * Thumbnails/2014/04/22/20140422-110751/W%BECsCHTIa2icv4VrBgfg/P1100271_1024.jpg

2016/11/16 20:06:13 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Waiting for checks to finish
2016/11/16 20:06:13 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Waiting for transfers to finish
2016/11/16 20:06:13 Waiting for deletions to finish
2016/11/16 20:06:13 Thumbnails/2016/11/05/20161105-013504/d60KJ75wRfiHXtIWs1fDfQ/thumb_IMG_5728.jpg: Not deleting as --dry-run
2016/11/16 20:06:13 resources/moments/needsanalysis: Not deleting as --dry-run
2016/11/16 20:06:13 Masters/2016/11/05/20161105-013504/IMG_5728.JPG: Not deleting as --dry-run
2016/11/16 20:06:13 Thumbnails/2016/11/05/20161105-013504/d60KJ75wRfiHXtIWs1fDfQ/thumb_IMG_5728_1024.jpg: Not deleting as --dry-run
2016/11/16 20:06:13
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:            324670
Transferred:         3932
Elapsed time:  9h30m36.4s

% grep "dry-run" sync-dry_20161116_103537.txt | wc -l
    3936
% grep "dry-run" sync-dry_20161116_103537.txt | grep "Not copying" | wc -l
    3932
% grep "dry-run" sync-dry_20161116_103537.txt | grep "Not deleting" | wc -l
       4

だいたいあってる。

sync

% rclone sync --checksum ~/Pictures/写真\ Library.photoslibrary/ amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/ 2>&1 | tee sync_`ts`.txt

2016/11/18 12:14:04
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      1m2.5s

.....

2016/11/19 00:03:40 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Waiting for checks to finish
2016/11/19 00:03:40 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Waiting for transfers to finish
2016/11/19 00:03:40 Waiting for deletions to finish
2016/11/19 00:03:42
Transferred:   31.606 GBytes (777.236 kBytes/s)
Errors:                 0
Checks:            324674
Transferred:         3947
Elapsed time:  11h50m39.5s

再確認

% rclone check --checksum ~/Pictures/写真\ Library.photoslibrary/ amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/ 2>&1 | tee check_`ts`.txt
2016/11/19 08:11:43 Local file system at /Users/takeru/Pictures/写真 Library.photoslibrary: Building file list
2016/11/19 08:11:43 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Building file list
2016/11/19 08:12:43
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      1m6.8s

.......

2016/11/19 22:57:43
Transferred:      0 Bytes (0 Bytes/s)
Errors:                22
Checks:            328185
Transferred:            0
Elapsed time:  14h46m6.8s
Checking:
 * Database/Versions/2010/11/10/20101110-003755/4Z559LDtSXyuqbgL8FDWmg/Version-0.apversion
 * Database/Versions/2013/05/09/20130509-014056/IrQpTcJKTgCPwSwfc56+Yw/Version-1.apversion
 * Masters/2010/11/10/20101110-003720/r0300657.jpg
 * Masters/2014/04/22/20140422-110751/P1090489.JPG
 * Masters/2015/12/16/20151216-005122/IMG_3809.JPG
 * Masters/2016/05/04/20160504-092529/P1200785.JPG
 * Thumbnails/2013/12/07/20131207-142941/miwgrFYCRtOusU+R7Nuvlw/IMG_1373_1024.jpg
 * iPod Photo Cache/F43/T5241.ithmb

2016/11/19 22:58:22 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': 22 differences found
2016/11/19 22:58:22 Failed to check: 22 differences found

9:32:21 tkrimac2:~/tmp/acd-test% cat check_20161119_081136.txt | grep "File not in amazon drive"
2016/11/19 10:34:42 Masters/2016/11/18/20161118-124000/IMG_5925.PNG: File not in amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary'
2016/11/19 10:34:42 Thumbnails/2016/11/18/20161118-124000/WFpqzcDvTN6hE3b0dvoJVQ/thumb_IMG_5925.jpg: File not in amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary'
2016/11/19 10:34:42 Thumbnails/2016/11/18/20161118-124000/WFpqzcDvTN6hE3b0dvoJVQ/thumb_IMG_5925_1024.jpg: File not in amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary'
9:33:37 tkrimac2:~/tmp/acd-test% cat check_20161119_081136.txt | grep "Sizes differ"
2016/11/19 11:23:41 resources/recovery/RKVersion/0000000015.lij: Sizes differ
2016/11/19 11:28:34 resources/recovery/RKMaster/0000000001.lij: Sizes differ
2016/11/19 13:30:09 Apple TV Photo Cache/Apple TV Photo Database: Sizes differ
2016/11/19 14:14:52 Database/apdb/Library.apdb-wal: Sizes differ
2016/11/19 15:45:11 Database/metaSchema.db-wal: Sizes differ
2016/11/19 18:39:56 resources/recovery/RKImportGroup/0000000000.lij: Sizes differ
2016/11/19 20:16:57 resources/recovery/RKImageProxyState/0000000002.lij: Sizes differ
9:35:08 tkrimac2:~/tmp/acd-test% cat check_20161119_081136.txt | grep "MD5 differ"
2016/11/19 11:30:20 resources/recovery/RKPlaceForVersion/0000000000.lij: MD5 differ
2016/11/19 14:17:08 Database/apdb/Person.db: MD5 differ
2016/11/19 14:27:13 Database/apdb/Person.db-wal: MD5 differ
2016/11/19 15:31:52 Database/apdb/Properties.apdb: MD5 differ
2016/11/19 15:43:03 Database/apdb/ImageProxies.apdb: MD5 differ
2016/11/19 15:44:51 Database/metaSchema.db: MD5 differ
2016/11/19 17:40:35 Database/RKVersion_searchIndexText.skindex: MD5 differ
2016/11/19 18:37:04 Database/apdb/Properties.apdb-wal: MD5 differ
2016/11/19 18:48:50 Database/apdb/Library.apdb: MD5 differ
2016/11/19 20:02:35 Database/apdb/ImageProxies.apdb-wal: MD5 differ
2016/11/19 20:22:31 resources/moments/historicalmarker.plist: MD5 differ
9:35:21 tkrimac2:~/tmp/acd-test% cat check_20161119_081136.txt | grep "File not in Local file system"
9:35:39 tkrimac2:~/tmp/acd-test% cat check_20161119_081136.txt | grep "Failed to calculate src hash"
2016/11/19 22:14:37 Database/metaSchema.db.lock: Failed to calculate src hash: hash: failed to stat: lstat /Users/takeru/Pictures/写真 Library.photoslibrary/Database/metaSchema.db.lock: no such file or directory

OKぽい。

8
8
3

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
8
8