VM の削除について
vSphere Client の操作をする際、右クリックのメニューは
インベントリから削除 と ディスクから削除 が隣り合っている。
一応どちらも警告は出るが、インベントリから削除するつもりでうっかり
ディスクから削除してしまうと VM のデータが葬られて使用できなくなる。
削除自体はすぐ終わるので 0 を上書きなどはされていないと推測する。
データ自体は残っていると思うので
消してしまったファイルは復旧可能なのか調べてみたい。
調査の流れ
VMFS は仕様が公開されていないので、
vmfs-tools のソースを参考にして調査を試みる。
VMFS5 を取り扱うには完璧じゃないようだが、とりあえず挑戦。
- ホストの vSphere ESXi にゲストとして vSphere ESXi をインストールする
（このゲスト ESXi がインストールされた vmdk ファイルの VMFS 領域が調査対象になる） - Nested VM として CentOS をインストールする
（ゲスト ESXi に、さらにゲストとして CentOS を入れる） - Netsed VM および ゲスト ESXi をシャットダウンして ゲスト ESXi の vmdk をコピー
- Nested VM のファイルが存在する状態で、事前に様子を見ておく
- Nested VM をディスクから削除した後、どう変わったか調べる
(削除後の vmdk ファイルもコピーして VMFS 領域を調べてみる)
Nested 環境なら調査対象の VMFS フォーマットされたパーティションは vmdk ファイル内なので、
他のところに持っていって作業がし易い。
調査開始
<vmname>.vmdk ファイルは virtual disk descriptor file と呼ばれ、ディスクデータ本体ではないので、
<vmname>-flat.vmdk ファイルをコピーしてくる。
ここでのターゲットは __backup_vsphere-flat.vmdk という名前になっている。
さて、どう探せばいいだろうか。
まずは Nested VM が削除されていない状態で
Nested VM のデータを見つけられるか確認してみることにする。
捜索対象は、Nested VM の MBR にする。
ゲスト ESXi の VMDK を別の Linux に転送しておいて、そちらで調査開始。
と言いつつ MBR のことが良く分かっていないので、先に構造を確認しておく。
参考資料をもとに、ざっくりと下記を理解。
- 先頭から 446 byte が ブートローダ
- ブートローダの後ろ 64 byte がパーティションテーブル (16 byte ずつ 4 レコード)
- 最後の 2 byte がマジックナンバー
手元にある GRUB がインストールされた CentOS の MBR を読むと以下のものが見つかる
- GRUB が使うと思われる文字列
- パーティションテーブル
- マジックナンバー
これらを手掛かりにして MBR を探す。
Nested VM は 10GB の VMDK を作成して 2台インストール済み。
それぞれ異なるパーティションの切り方をした。
ゲスト ESXi の vmdk は 40GB で作っていて、とりあえず端から端まで探す。
searchmbr.py
# !/usr/bin/env python
# -*- coding: utf-8 -*-
import sys, os, re, struct
# target = './mbr.dd'
target = './__backup_vsphere-flat.vmdk'
MAGIC_OF_MBR = '\x55\xaa'
MARK_OF_GRUB = 'GRUB \x00Geom\x00Hard Disk\x00Read\x00 Error'
try:
if sys.argv[1] == '-d':
DEBUG = 1
except:
pass
def D(mesg):
try:
if DEBUG:
import inspect
print "[DEBUG:%d] %s" % (inspect.currentframe().f_back.f_lineno, mesg)
except NameError:
pass
def convert_num(num):
if len(str(int(num))) > 12:
return '%.1fT' % (num/(1024.0**4))
elif len(str(int(num))) > 9:
return '%.1fG' % (num/(1024.0**3))
elif len(str(int(num))) > 6:
return '%.1fM' % (num/(1024.0**2))
elif len(str(int(num))) > 3:
return '%.1fk' % (num/(1024.0**1))
else:
return str(num)
def check(data):
bootloader = data[0:446]
part_table = data[446:510]
magic = data[510:512]
if magic != MAGIC_OF_MBR:
return False
if re.search(MARK_OF_GRUB, bootloader):
print 'This is maybe MBR !!'
else:
return False
splitter = lambda s, n: [s[i:i+n] for i in range(0, len(s), n)]
print "Partition Boot %s %s Id Size" % ('Start'.rjust(10), 'End'.rjust(10))
i=1
for record in splitter(part_table, 16):
boot = record[0]
D('flag: %s' % hex(ord(boot)))
if boot == '\x80':
boot = '*'
elif boot == '\x00':
boot = ''
else:
print 'boot flag is wrong'
boot = hex(ord(boot))
id = record[4:5]
# LBA 方式のデータのみ確認 (CHS 方式のデータを見ない)
start_sector, num_sector = struct.unpack('II', record[8:16])
print "%s %s %10d %10d %s %sB" % (str(i).rjust(9), boot.rjust(4), start_sector, start_sector + num_sector, hex(ord(id)).rjust(4), convert_num(num_sector * 512))
i+=1
return True
with open(target , 'r') as f:
try:
fd = f.fileno()
except Exception as e:
print e
raise Exception('cannot get file size')
else:
size = os.fstat(fd).st_size
i=1
data = f.read(512)
while data:
if check(data):
print "position of above data: %d - %d" % (f.tell() - 512, f.tell())
data = f.read(512)
if f.tell() > size / 100.0 * i:
D("searching... %d / %d" % (f.tell(), size))
i+=1
MBR捜索
# python searchmbr.py
This is maybe MBR !!
Partition Boot Start End Id Size
1 * 2048 1026048 0x83 500.0MB
2 1026048 2074624 0x82 512.0MB
3 2074624 20971520 0x83 9.0GB
4 0 0 0x0 0B
position of above data: 38196477952 - 38196478464
This is maybe MBR !!
Partition Boot Start End Id Size
boot flag is wrong
1 0x24 332251185 2480556663 0x0 1.0TB
boot flag is wrong
2 0xf9 2548826057 4435176463 0xbe 899.5GB
boot flag is wrong
3 0x70 3053499650 4967399938 0x0 912.6GB
boot flag is wrong
4 0xd7 0 0 0x4f 0B
position of above data: 38792327168 - 38792327680
This is maybe MBR !!
Partition Boot Start End Id Size
boot flag is wrong
1 0x24 332251185 2480556663 0x0 1.0TB
boot flag is wrong
2 0xf9 2548826057 4435176463 0xbe 899.5GB
boot flag is wrong
3 0x70 3053499650 4967399938 0x0 912.6GB
boot flag is wrong
4 0xd7 0 0 0x4f 0B
position of above data: 39116325888 - 39116326400
This is maybe MBR !!
Partition Boot Start End Id Size
1 * 2048 1026048 0x83 500.0MB
2 1026048 20971520 0x8e 9.5GB
3 0 0 0x0 0B
4 0 0 0x0 0B
position of above data: 39419117568 - 39419118080
This is maybe MBR !!
Partition Boot Start End Id Size
boot flag is wrong
1 0x24 332251185 2480556663 0x0 1.0TB
boot flag is wrong
2 0xf9 2548826057 4435176463 0xbe 899.5GB
boot flag is wrong
3 0x70 3053499650 4967399938 0x0 912.6GB
boot flag is wrong
4 0xd7 0 0 0x4f 0B
position of above data: 40218379264 - 40218379776
This is maybe MBR !!
Partition Boot Start End Id Size
boot flag is wrong
1 0x24 332251185 2480556663 0x0 1.0TB
boot flag is wrong
2 0xf9 2548826057 4435176463 0xbe 899.5GB
boot flag is wrong
3 0x70 3053499650 4967399938 0x0 912.6GB
boot flag is wrong
4 0xd7 0 0 0x4f 0B
position of above data: 40242606080 - 40242606592
無事探し出せた。
フラグがおかしなものも見つかるが、Nested VM は
10GB の vmdk で作ったので、フラグが正しい 2つで間違いないだろう。
VMFS の探索
それでは VMFS を読んでいく。ここからが本番だ。
# parted __backup_vsphere-flat.vmdk unit s print
モデル: (file)
ディスク /root/recover/__backup_vsphere-flat.vmdk: 83886080s
セクタサイズ (論理/物理): 512B/512B
パーティションテーブル: gpt
番号 開始 終了 サイズ ファイルシステム 名前 フラグ
1 64s 8191s 8128s fat16 boot
5 8224s 520191s 511968s fat16
6 520224s 1032191s 511968s fat16
7 1032224s 1257471s 225248s
8 1257504s 1843199s 585696s fat16
2 1843200s 10229759s 8386560s fat16
3 10229760s 83886046s 73656287s
パーティションテーブルは GPT
一番サイズの大きい領域が VMFS のはずだが、parted では良く分からない。
GPT において、VMFS の GUID は AA31E02A400F11DB9590000C2911D1B8 が使われるそうだ。
表記を変えると AA31E02A-400F-11DB-9590-000C2911D1B8
サイズの大きい パーティション 3番に相当する場所を見てみると、
先頭の 3項目 はリトルエンディアンだそうだ。確かに GUID が一致した。
GPTの調査
# hexdump -C -s `expr 512 \* 2 + 128 \* 2` -n 128 vsphere-flat.vmdk | less
00000500 2a e0 31 aa 0f 40 db 11 95 90 00 0c 29 11 d1 b8 |*.1..@......)...|
00000510 45 01 35 2b 41 d6 59 45 aa 77 33 40 c0 94 f4 0d |E.5+A.YE.w3@....|
00000520 00 18 9c 00 00 00 00 00 de ff ff 04 00 00 00 00 |................|
00000530 00 00 00 00 00 00 00 00 00 00 ff ff ff ff ff ff |................|
00000540 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
VMFS に関する情報を読み進めていきたい。
vmfs_fs.h
# define VMFS_FSINFO_MAGIC 0x2fabf15e
struct vmfs_fsinfo_raw {
uint32_t magic;
uint32_t volver;
u_char ver;
uuid_t uuid;
uint32_t mode;
char label[128];
uint32_t dev_blocksize;
uint64_t blocksize;
uint32_t ctime; /* ctime? in seconds */
uint32_t _unknown3;
uuid_t lvm_uuid;
u_char _unknown4[16];
uint32_t fdc_header_size;
uint32_t fdc_bitmap_count;
uint32_t subblock_size;
} __attribute__((packed));
vmfs-tools を見るとマジックナンバーがあるようなので、MBR の時と同じ要領で探してみる。
searchvmfsinfo.py
# !/usr/bin/env python
# -*- coding: utf-8 -*-
import sys, os, re, struct
target = './__backup_vsphere-flat.vmdk'
MAGIC_OF_VMFSINFO = '\x5e\xf1\xab\x2f'
try:
if sys.argv[1] == '-d':
DEBUG = 1
except:
pass
def D(mesg):
try:
if DEBUG:
import inspect
print "[DEBUG:%d] %s" % (inspect.currentframe().f_back.f_lineno, mesg)
except NameError:
pass
def check(data):
if MAGIC_OF_VMFSINFO in data:
print 'Magic Number found. This is maybe VMFS !!'
return True
else:
return False
with open(target , 'r') as f:
try:
fd = f.fileno()
except Exception as e:
print e
raise Exception('cannot get file size')
else:
size = os.fstat(fd).st_size
i=1
data = f.read(512)
while data:
if check(data):
print "position of above data: %d - %d" % (f.tell() - 512, f.tell())
data = f.read(512)
if f.tell() > size / 100.0 * i:
D("searching... %d / %d" % (f.tell(), size))
i+=1
vmfsinfo捜索
# python searchvmfs.py
Magic Number found. This is maybe VMFS !!
position of above data: 5257560064 - 5257560576
Magic Number found. This is maybe VMFS !!
position of above data: 37435702784 - 37435703296
Magic Number found. This is maybe VMFS !!
position of above data: 38707637760 - 38707638272
Magic Number found. This is maybe VMFS !!
position of above data: 39859326464 - 39859326976
Magic Number found. This is maybe VMFS !!
position of above data: 40373243392 - 40373243904
いくつか見つかった。VMFS パーティションの先頭は 10229760 * 512 = 5237637120 なので先頭からすぐ始まっているわけではなかったようだ。
間が空いているのはよく分からないが、多分最初に見つかった場所に VMFS の情報があるのだろう。
その他は偶然一致しただけだろうか？
128 バイト分の label が見られそうなので、調べてみよう。
labelデータの確認
# hexdump -C -s `expr 5257560064 + 29` -n 128 __backup_vsphere-flat.vmdk
13960001d 64 61 74 61 73 74 6f 72 65 31 00 00 00 00 00 00 |datastore1......|
13960002d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
# hexdump -C -s `expr 37435702784 + 29` -n 128 __backup_vsphere-flat.vmdk
8b7577e1d 00 73 71 75 61 73 68 66 73 00 68 73 71 73 00 73 |.squashfs.hsqs.s|
8b7577e2d 68 73 71 00 25 75 2e 25 30 32 75 00 6e 73 73 00 |hsq.%u.%02u.nss.|
8b7577e3d 53 50 42 35 00 78 65 6e 69 78 00 2b 55 44 00 44 |SPB5.xenix.+UD.D|
8b7577e4d 55 2b 00 73 79 73 76 00 62 74 72 66 73 00 5f 42 |U+.sysv.btrfs._B|
8b7577e5d 48 52 66 53 5f 4d 00 4c 41 42 45 4c 4f 4e 45 00 |HRfS_M.LABELONE.|
8b7577e6d 4c 56 4d 32 5f 6d 65 6d 62 65 72 00 4c 56 4d 32 |LVM2_member.LVM2|
8b7577e7d 20 30 30 31 00 4c 56 4d 31 5f 6d 65 6d 62 65 72 | 001.LVM1_member|
8b7577e8d 00 48 4d 00 44 4d 5f 73 6e 61 70 73 68 6f 74 5f |.HM.DM_snapshot_|
8b7577e9d
# hexdump -C -s `expr 38707637760 + 29` -n 128 __backup_vsphere-flat.vmdk
90327ae1d 00 73 71 75 61 73 68 66 73 00 68 73 71 73 00 73 |.squashfs.hsqs.s|
90327ae2d 68 73 71 00 25 75 2e 25 30 32 75 00 6e 73 73 00 |hsq.%u.%02u.nss.|
90327ae3d 53 50 42 35 00 78 65 6e 69 78 00 2b 55 44 00 44 |SPB5.xenix.+UD.D|
90327ae4d 55 2b 00 73 79 73 76 00 62 74 72 66 73 00 5f 42 |U+.sysv.btrfs._B|
90327ae5d 48 52 66 53 5f 4d 00 4c 41 42 45 4c 4f 4e 45 00 |HRfS_M.LABELONE.|
90327ae6d 4c 56 4d 32 5f 6d 65 6d 62 65 72 00 4c 56 4d 32 |LVM2_member.LVM2|
90327ae7d 20 30 30 31 00 4c 56 4d 31 5f 6d 65 6d 62 65 72 | 001.LVM1_member|
90327ae8d 00 48 4d 00 44 4d 5f 73 6e 61 70 73 68 6f 74 5f |.HM.DM_snapshot_|
90327ae9d
# hexdump -C -s `expr 39859326464 + 29` -n 128 __backup_vsphere-flat.vmdk
947cd0e1d 00 73 71 75 61 73 68 66 73 00 68 73 71 73 00 73 |.squashfs.hsqs.s|
947cd0e2d 68 73 71 00 25 75 2e 25 30 32 75 00 6e 73 73 00 |hsq.%u.%02u.nss.|
947cd0e3d 53 50 42 35 00 78 65 6e 69 78 00 2b 55 44 00 44 |SPB5.xenix.+UD.D|
947cd0e4d 55 2b 00 73 79 73 76 00 62 74 72 66 73 00 5f 42 |U+.sysv.btrfs._B|
947cd0e5d 48 52 66 53 5f 4d 00 4c 41 42 45 4c 4f 4e 45 00 |HRfS_M.LABELONE.|
947cd0e6d 4c 56 4d 32 5f 6d 65 6d 62 65 72 00 4c 56 4d 32 |LVM2_member.LVM2|
947cd0e7d 20 30 30 31 00 4c 56 4d 31 5f 6d 65 6d 62 65 72 | 001.LVM1_member|
947cd0e8d 00 48 4d 00 44 4d 5f 73 6e 61 70 73 68 6f 74 5f |.HM.DM_snapshot_|
947cd0e9d
# hexdump -C -s `expr 40373243392 + 29` -n 128 __backup_vsphere-flat.vmdk
9666ece1d 00 73 71 75 61 73 68 66 73 00 68 73 71 73 00 73 |.squashfs.hsqs.s|
9666ece2d 68 73 71 00 25 75 2e 25 30 32 75 00 6e 73 73 00 |hsq.%u.%02u.nss.|
9666ece3d 53 50 42 35 00 78 65 6e 69 78 00 2b 55 44 00 44 |SPB5.xenix.+UD.D|
9666ece4d 55 2b 00 73 79 73 76 00 62 74 72 66 73 00 5f 42 |U+.sysv.btrfs._B|
9666ece5d 48 52 66 53 5f 4d 00 4c 41 42 45 4c 4f 4e 45 00 |HRfS_M.LABELONE.|
9666ece6d 4c 56 4d 32 5f 6d 65 6d 62 65 72 00 4c 56 4d 32 |LVM2_member.LVM2|
9666ece7d 20 30 30 31 00 4c 56 4d 31 5f 6d 65 6d 62 65 72 | 001.LVM1_member|
9666ece8d 00 48 4d 00 44 4d 5f 73 6e 61 70 73 68 6f 74 5f |.HM.DM_snapshot_|
9666ece9d
最初に見つけた場所からデフォルトで作成された datastore1 という label が確認できる。
他は何か分からないが、今は無視してよさそうだ。
ここからどうすべきか、正直まださっぱり分からないが、inode があるようだ。
vmfs_inode.h
struct vmfs_inode_raw {
struct vmfs_metadata_hdr_raw mdh;
uint32_t id;
uint32_t id2; /* seems to be VMFS_BLK_FD_ITEM(id) + 1 */
uint32_t nlink;
uint32_t type;
uint32_t flags;
uint64_t size;
uint64_t blk_size;
uint64_t blk_count;
uint32_t mtime;
uint32_t ctime;
uint32_t atime;
uint32_t uid;
uint32_t gid;
uint32_t mode;
uint32_t zla;
uint32_t tbz;
uint32_t cow;
u_char _unknown2[432];
union {
uint32_t blocks[VMFS_INODE_BLK_COUNT];
uint32_t rdm_id;
char content[VMFS_INODE_BLK_COUNT * sizeof(uint32_t)];
};
} __attribute__((packed));
vmfs_metadata.h
struct vmfs_metadata_hdr_raw {
uint32_t magic; /* Magic number */
uint64_t pos; /* Position in the volume */
uint64_t hb_pos; /* Heartbeat position */
uint64_t hb_seq; /* Heartbeat sequence */
uint64_t obj_seq; /* Object sequence */
uint32_t hb_lock; /* Heartbeat lock flag */
uuid_t hb_uuid; /* UUID of locking server */
uint64_t mtime;
u_char pad1[0x1c0]; /* Padding/unknown */
} __attribute__((packed));
とにかく inode が見つかれば何とかなる気がする。
vmfs_inode.h
# define VMFS_INODE_MAGIC 0x10c00001
inode もマジックナンバーがあるようなので、これを探してみる。
ついでに id と size を表示すれば手掛かりになりそうだ。
searchinode.py
# !/usr/bin/env python
# -*- coding: utf-8 -*-
import sys, os, re, struct
target = './__backup_vsphere-flat.vmdk'
POS_OF_VMFSINFOMAGIC = 5257560064
MAGIC_OF_VMFSINODE = '\x01\x00\xc0\x10'
try:
if sys.argv[1] == '-d':
DEBUG = 1
except:
pass
def D(mesg):
try:
if DEBUG:
import inspect
print "[DEBUG:%d] %s" % (inspect.currentframe().f_back.f_lineno, mesg)
except NameError:
pass
def convert_num(num):
if len(str(int(num))) > 12:
return '%.1fT' % (num/(1024.0**4))
elif len(str(int(num))) > 9:
return '%.1fG' % (num/(1024.0**3))
elif len(str(int(num))) > 6:
return '%.1fM' % (num/(1024.0**2))
elif len(str(int(num))) > 3:
return '%.1fk' % (num/(1024.0**1))
else:
return str(num)
with open(target , 'r') as f:
def check(data):
if MAGIC_OF_VMFSINODE in data:
print 'Magic Number found. This is maybe inode !!'
data = f.read(512)
id, id2, size = struct.unpack('IIQ', data[0:8] + data[20:28])
print 'id = %d, id2 = %d, filesize = %sB' % (id, id2, convert_num(size))
return True
else:
return False
try:
fd = f.fileno()
except Exception as e:
print e
raise Exception('cannot get file size')
else:
size = os.fstat(fd).st_size
f.seek(POS_OF_VMFSINFOMAGIC)
i=1
data = f.read(512)
while data:
if check(data):
print "position of above data: %d - %d" % (f.tell() - 1024, f.tell() - 512)
data = f.read(512)
if f.tell() > size / 100.0 * i:
D("searching... %d / %d" % (f.tell(), size))
i+=1
inode捜索
# python searchinode.py
Magic Number found. This is maybe inode !!
id = 4, id2 = 1, filesize = 1.4kB
position of above data: 5259788288 - 5259788800
Magic Number found. This is maybe inode !!
id = 4194308, id2 = 1, filesize = 256.0kB
position of above data: 5259790336 - 5259790848
Magic Number found. This is maybe inode !!
id = 8388612, id2 = 1, filesize = 254.7MB
position of above data: 5259792384 - 5259792896
Magic Number found. This is maybe inode !!
id = 12582916, id2 = 1, filesize = 256.0MB
position of above data: 5259794432 - 5259794944
Magic Number found. This is maybe inode !!
id = 16777220, id2 = 1, filesize = 250.6MB
position of above data: 5259796480 - 5259796992
Magic Number found. This is maybe inode !!
id = 20971524, id2 = 1, filesize = 4.0MB
position of above data: 5259798528 - 5259799040
Magic Number found. This is maybe inode !!
id = 25165828, id2 = 1, filesize = 1.1MB
position of above data: 5259800576 - 5259801088
Magic Number found. This is maybe inode !!
id = 32772, id2 = 1, filesize = 0B
position of above data: 5472124928 - 5472125440
Magic Number found. This is maybe inode !!
id = 4227076, id2 = 2, filesize = 0B
position of above data: 5472126976 - 5472127488
Magic Number found. This is maybe inode !!
id = 8421380, id2 = 3, filesize = 0B
position of above data: 5472129024 - 5472129536
Magic Number found. This is maybe inode !!
id = 12615684, id2 = 4, filesize = 0B
position of above data: 5472131072 - 5472131584
Magic Number found. This is maybe inode !!
id = 16809988, id2 = 5, filesize = 0B
position of above data: 5472133120 - 5472133632
Magic Number found. This is maybe inode !!
id = 21004292, id2 = 6, filesize = 0B
position of above data: 5472135168 - 5472135680
Magic Number found. This is maybe inode !!
id = 25198596, id2 = 7, filesize = 0B
position of above data: 5472137216 - 5472137728
Magic Number found. This is maybe inode !!
id = 29392900, id2 = 8, filesize = 0B
position of above data: 5472139264 - 5472139776
Magic Number found. This is maybe inode !!
id = 33587204, id2 = 9, filesize = 0B
position of above data: 5472141312 - 5472141824
Magic Number found. This is maybe inode !!
id = 37781508, id2 = 10, filesize = 0B
position of above data: 5472143360 - 5472143872
Magic Number found. This is maybe inode !!
id = 41975812, id2 = 11, filesize = 0B
position of above data: 5472145408 - 5472145920
Magic Number found. This is maybe inode !!
id = 46170116, id2 = 12, filesize = 0B
position of above data: 5472147456 - 5472147968
Magic Number found. This is maybe inode !!
id = 50364420, id2 = 14, filesize = 0B
position of above data: 5472149504 - 5472150016
Magic Number found. This is maybe inode !!
id = 54558724, id2 = 15, filesize = 0B
position of above data: 5472151552 - 5472152064
Magic Number found. This is maybe inode !!
id = 58753028, id2 = 17, filesize = 0B
position of above data: 5472153600 - 5472154112
Magic Number found. This is maybe inode !!
id = 62947332, id2 = 18, filesize = 0B
position of above data: 5472155648 - 5472156160
Magic Number found. This is maybe inode !!
id = 67141636, id2 = 20, filesize = 1.4kB
position of above data: 5472157696 - 5472158208
Magic Number found. This is maybe inode !!
id = 71335940, id2 = 21, filesize = 2.6kB
position of above data: 5472159744 - 5472160256
Magic Number found. This is maybe inode !!
id = 75530244, id2 = 22, filesize = 261B
position of above data: 5472161792 - 5472162304
Magic Number found. This is maybe inode !!
id = 79724548, id2 = 23, filesize = 0B
position of above data: 5472163840 - 5472164352
Magic Number found. This is maybe inode !!
id = 83918852, id2 = 24, filesize = 10.0GB
position of above data: 5472165888 - 5472166400
Magic Number found. This is maybe inode !!
id = 88113156, id2 = 25, filesize = 493B
position of above data: 5472167936 - 5472168448
Magic Number found. This is maybe inode !!
id = 92307460, id2 = 26, filesize = 0B
position of above data: 5472169984 - 5472170496
Magic Number found. This is maybe inode !!
id = 96501764, id2 = 27, filesize = 64.8kB
position of above data: 5472172032 - 5472172544
Magic Number found. This is maybe inode !!
id = 100696068, id2 = 28, filesize = 0B
position of above data: 5472174080 - 5472174592
Magic Number found. This is maybe inode !!
id = 104890372, id2 = 29, filesize = 8.5kB
position of above data: 5472176128 - 5472176640
Magic Number found. This is maybe inode !!
id = 109084676, id2 = 30, filesize = 0B
position of above data: 5472178176 - 5472178688
Magic Number found. This is maybe inode !!
id = 113278980, id2 = 31, filesize = 123.1kB
position of above data: 5472180224 - 5472180736
Magic Number found. This is maybe inode !!
id = 117473284, id2 = 33, filesize = 1.2kB
position of above data: 5472182272 - 5472182784
Magic Number found. This is maybe inode !!
id = 121667588, id2 = 34, filesize = 2.5kB
position of above data: 5472184320 - 5472184832
Magic Number found. This is maybe inode !!
id = 125861892, id2 = 35, filesize = 261B
position of above data: 5472186368 - 5472186880
Magic Number found. This is maybe inode !!
id = 130056196, id2 = 36, filesize = 0B
position of above data: 5472188416 - 5472188928
Magic Number found. This is maybe inode !!
id = 134250500, id2 = 37, filesize = 10.0GB
position of above data: 5472190464 - 5472190976
Magic Number found. This is maybe inode !!
id = 138444804, id2 = 38, filesize = 493B
position of above data: 5472192512 - 5472193024
Magic Number found. This is maybe inode !!
id = 142639108, id2 = 39, filesize = 0B
position of above data: 5472194560 - 5472195072
Magic Number found. This is maybe inode !!
id = 146833412, id2 = 40, filesize = 119.8kB
position of above data: 5472196608 - 5472197120
Magic Number found. This is maybe inode !!
id = 151027716, id2 = 41, filesize = 0B
position of above data: 5472198656 - 5472199168
Magic Number found. This is maybe inode !!
id = 155222020, id2 = 42, filesize = 8.5kB
position of above data: 5472200704 - 5472201216
^CTraceback (most recent call last):
File "searchinode.py", line 66, in <module>
data = f.read(512)
KeyboardInterrupt
それらしいものが見つかったので中断。
size を見ると 10.0GB を示すものがあるので、これが
<vmname>-flat.vmdk ファイルの inode と見てよさそうだ。
実際のブロックにどうマッピングされているのかは追いきれていないが、
この <vmname>-flat.vmdk の inode の 0x400 にあるのは
(vmfs-tools で言う所の) pointer block id で、その id の pointer block を読んで
色々処理した結果、実際のブロックが見つかる仕組みになっているようだ。
何にせよこのあたりを使ってマッピングしているようなので、
削除された後にそれらしい情報が残っていれば復旧できそうに思える。
いったんこの inode を ESXi shell から確認してみよう。
83918852 と 134250500 が <vmname>-flat.vmdk なら期待通りだ。
inode表示
# pwd
/vmfs/volumes/datastore1
# ls -li */*vmdk
83918852 -rw------- 1 root root 10737418240 Jun 13 13:31 test01/test01-flat.vmdk
88113156 -rw------- 1 root root 493 Jun 13 13:03 test01/test01.vmdk
134250500 -rw------- 1 root root 10737418240 Jun 13 13:57 test02/test02-flat.vmdk
138444804 -rw------- 1 root root 493 Jun 13 13:38 test02/test02.vmdk
致命的な思い違いは今のところ無さそうだ。さっそく削除してデータを確認する。
vSphere Client で接続して ディスクから削除 により削除する。
ゲスト ESXi が稼働中だと vmdk にアクセスできないようなので、シャットダウンして調べる。
ディスクから削除した
# pwd
/vmfs/volumes/datastore1
# ls -a
. .. .fbb.sf .fdc.sf .pb2.sf .pbc.sf .sbc.sf .vh.sf
削除前のデータと比較する。
削除前のinode情報
## <vmname>-flat.vmdk を示す inode
# hexdump -C -s 5472165888 -n 2048 __backup_vsphere-flat.vmdk
1462aa000 01 00 c0 10 00 a0 ea 0c 00 00 00 00 00 00 3e 00 |..............>.|
1462aa010 00 00 00 00 1f 00 00 00 00 00 00 00 20 00 00 00 |............ ...|
1462aa020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
1462aa030 00 00 00 00 00 00 00 00 91 05 00 00 00 00 00 00 |................|
1462aa040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
1462aa200 04 80 00 05 18 00 00 00 01 00 00 00 03 00 00 00 |................|
1462aa210 00 00 00 00 00 00 00 80 02 00 00 00 00 00 10 00 |................|
1462aa220 00 00 00 00 52 04 00 00 00 00 00 00 4d fd 9a 53 |....R.......M..S|
1462aa230 f6 f8 9a 53 4d fd 9a 53 00 00 00 00 00 00 00 00 |...SM..S........|
1462aa240 80 01 00 00 03 00 00 00 00 00 00 00 00 00 00 00 |................|
1462aa250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
1462aa400 03 ed 02 10 03 ed 02 30 03 ed 02 40 03 ed 02 50 |.......0...@...P|
1462aa410 03 ed 02 60 03 ed 02 70 03 ed 02 80 03 ed 02 90 |...`...p........|
1462aa420 03 ed 02 a0 03 ed 02 20 00 00 00 00 00 00 00 00 |....... ........|
1462aa430 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
1462aa800
削除後のinode情報
## vSphere のシェル上で直接 hexdump で skip できなかったので dd で skip してから渡す
## <vmname>-flat.vmdk を示す inode
# dd if=vsphere-flat.vmdk bs=512 skip=10687824 count=4
2> /dev/null | hexdump -C
00000000 01 00 c0 10 00 a0 ea 0c 00 00 00 00 00 f0 3d 00 |..............=.|
00000010 00 00 00 00 11 00 00 00 00 00 00 00 24 00 00 00 |............$...|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 d3 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000200 04 80 00 05 18 00 00 00 00 00 00 00 03 00 00 00 |................|
00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 10 00 |................|
00000220 00 00 00 00 00 00 00 00 00 00 00 00 4d fd 9a 53 |............M..S|
00000230 e8 a9 9d 53 e1 7b 9d 53 00 00 00 00 00 00 00 00 |...S.{.S........|
00000240 80 01 00 00 03 00 00 00 00 00 00 00 00 00 00 00 |................|
00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000800
マッピングに使われると思われる場所が 0 で上書きされている。
・・・こうなると私の手には負えない、諦めよう。そしてもう寝よう。
小さいファイルは inode に直接収まっていたようなので
そちらも確認してみたが、このファイルはデータがそのまま見つかった。
削除後のvmdkから小さいファイルのinodeを確認
## <vmname>.vmdk (descriptor の方) を示す inode
# dd if=vsphere-flat.vmdk bs=512 skip=10687828 count=4 2> /dev/null | hexdump -C
00000000 01 00 c0 10 00 a8 ea 0c 00 00 00 00 00 f0 3d 00 |..............=.|
00000010 00 00 00 00 11 00 00 00 00 00 00 00 2b 00 00 00 |............+...|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 d9 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 3e 00 00 00 00 00 1f 00 00 00 |......>.........|
00000050 00 00 00 00 ff e6 9a 53 cf 54 c0 67 9d 62 00 0c |.......S.T.g.b..|
00000060 29 d5 43 a8 00 00 00 00 00 00 00 00 00 00 00 00 |).C.............|
00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000200 04 80 40 05 19 00 00 00 00 00 00 00 03 00 00 00 |..@.............|
00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 |............. ..|
00000220 00 00 00 00 00 00 00 00 00 00 00 00 a9 f6 9a 53 |...............S|
00000230 8e f5 9a 53 a9 f6 9a 53 00 00 00 00 00 00 00 00 |...S...S........|
00000240 80 01 00 00 d1 10 00 00 00 00 00 00 00 00 00 00 |................|
00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000400 23 20 44 69 73 6b 20 44 65 73 63 72 69 70 74 6f |# Disk Descripto|
00000410 72 46 69 6c 65 0a 76 65 72 73 69 6f 6e 3d 31 0a |rFile.version=1.|
00000420 65 6e 63 6f 64 69 6e 67 3d 22 55 54 46 2d 38 22 |encoding="UTF-8"|
00000430 0a 43 49 44 3d 35 62 61 64 65 30 63 31 0a 70 61 |.CID=5bade0c1.pa|
00000440 72 65 6e 74 43 49 44 3d 66 66 66 66 66 66 66 66 |rentCID=ffffffff|
00000450 0a 69 73 4e 61 74 69 76 65 53 6e 61 70 73 68 6f |.isNativeSnapsho|
00000460 74 3d 22 6e 6f 22 0a 63 72 65 61 74 65 54 79 70 |t="no".createTyp|
00000470 65 3d 22 76 6d 66 73 22 0a 0a 23 20 45 78 74 65 |e="vmfs"..# Exte|
00000480 6e 74 20 64 65 73 63 72 69 70 74 69 6f 6e 0a 52 |nt description.R|
00000490 57 20 32 30 39 37 31 35 32 30 20 56 4d 46 53 20 |W 20971520 VMFS |
000004a0 22 74 65 73 74 30 31 2d 66 6c 61 74 2e 76 6d 64 |"test01-flat.vmd|
000004b0 6b 22 0a 0a 23 20 54 68 65 20 44 69 73 6b 20 44 |k"..# The Disk D|
000004c0 61 74 61 20 42 61 73 65 20 0a 23 44 44 42 0a 0a |ata Base .#DDB..|
000004d0 64 64 62 2e 61 64 61 70 74 65 72 54 79 70 65 20 |ddb.adapterType |
000004e0 3d 20 22 6c 73 69 6c 6f 67 69 63 22 0a 64 64 62 |= "lsilogic".ddb|
000004f0 2e 74 68 69 6e 50 72 6f 76 69 73 69 6f 6e 65 64 |.thinProvisioned|
00000500 20 3d 20 22 31 22 0a 64 64 62 2e 67 65 6f 6d 65 | = "1".ddb.geome|
00000510 74 72 79 2e 73 65 63 74 6f 72 73 20 3d 20 22 36 |try.sectors = "6|
00000520 33 22 0a 64 64 62 2e 67 65 6f 6d 65 74 72 79 2e |3".ddb.geometry.|
00000530 68 65 61 64 73 20 3d 20 22 32 35 35 22 0a 64 64 |heads = "255".dd|
00000540 62 2e 67 65 6f 6d 65 74 72 79 2e 63 79 6c 69 6e |b.geometry.cylin|
00000550 64 65 72 73 20 3d 20 22 31 33 30 35 22 0a 64 64 |ders = "1305".dd|
00000560 62 2e 75 75 69 64 20 3d 20 22 36 30 20 30 30 20 |b.uuid = "60 00 |
00000570 43 32 20 39 64 20 33 35 20 63 37 20 30 39 20 63 |C2 9d 35 c7 09 c|
00000580 66 2d 63 66 20 64 34 20 30 65 20 61 33 20 32 37 |f-cf d4 0e a3 27|
00000590 20 36 35 20 39 38 20 62 62 22 0a 64 64 62 2e 6c | 65 98 bb".ddb.l|
000005a0 6f 6e 67 43 6f 6e 74 65 6e 74 49 44 20 3d 20 22 |ongContentID = "|
000005b0 37 64 35 30 66 36 36 32 33 30 37 32 30 38 37 61 |7d50f6623072087a|
000005c0 37 31 64 36 61 63 63 39 35 62 61 64 65 30 63 31 |71d6acc95bade0c1|
000005d0 22 0a 64 64 62 2e 76 69 72 74 75 61 6c 48 57 56 |".ddb.virtualHWV|
000005e0 65 72 73 69 6f 6e 20 3d 20 22 38 22 0a 00 00 00 |ersion = "8"....|
000005f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000800
削除後のゲスト ESXi の vmdk をコピーして調査
inode のマジックナンバーは見つけられるので、サイズは 0 byte になっているものの
小さいファイルは片っ端から探せば復旧できそうだ。
ディスクから削除後のinode捜索
# python searchinode.py
Magic Number found. This is maybe inode !!
id = 4, id2 = 1, filesize = 1.1kB
position of above data: 5259788288 - 5259788800
Magic Number found. This is maybe inode !!
id = 4194308, id2 = 1, filesize = 256.0kB
position of above data: 5259790336 - 5259790848
Magic Number found. This is maybe inode !!
id = 8388612, id2 = 1, filesize = 254.7MB
position of above data: 5259792384 - 5259792896
Magic Number found. This is maybe inode !!
id = 12582916, id2 = 1, filesize = 256.0MB
position of above data: 5259794432 - 5259794944
Magic Number found. This is maybe inode !!
id = 16777220, id2 = 1, filesize = 250.6MB
position of above data: 5259796480 - 5259796992
Magic Number found. This is maybe inode !!
id = 20971524, id2 = 1, filesize = 4.0MB
position of above data: 5259798528 - 5259799040
Magic Number found. This is maybe inode !!
id = 25165828, id2 = 1, filesize = 1.1MB
position of above data: 5259800576 - 5259801088
Magic Number found. This is maybe inode !!
id = 32772, id2 = 1, filesize = 0B
position of above data: 5472124928 - 5472125440
Magic Number found. This is maybe inode !!
id = 4227076, id2 = 2, filesize = 0B
position of above data: 5472126976 - 5472127488
Magic Number found. This is maybe inode !!
id = 8421380, id2 = 3, filesize = 0B
position of above data: 5472129024 - 5472129536
Magic Number found. This is maybe inode !!
id = 12615684, id2 = 4, filesize = 0B
position of above data: 5472131072 - 5472131584
Magic Number found. This is maybe inode !!
id = 16809988, id2 = 5, filesize = 0B
position of above data: 5472133120 - 5472133632
Magic Number found. This is maybe inode !!
id = 21004292, id2 = 6, filesize = 0B
position of above data: 5472135168 - 5472135680
Magic Number found. This is maybe inode !!
id = 25198596, id2 = 7, filesize = 0B
position of above data: 5472137216 - 5472137728
Magic Number found. This is maybe inode !!
id = 29392900, id2 = 8, filesize = 0B
position of above data: 5472139264 - 5472139776
Magic Number found. This is maybe inode !!
id = 33587204, id2 = 9, filesize = 0B
position of above data: 5472141312 - 5472141824
Magic Number found. This is maybe inode !!
id = 37781508, id2 = 10, filesize = 0B
position of above data: 5472143360 - 5472143872
Magic Number found. This is maybe inode !!
id = 41975812, id2 = 11, filesize = 0B
position of above data: 5472145408 - 5472145920
Magic Number found. This is maybe inode !!
id = 46170116, id2 = 12, filesize = 0B
position of above data: 5472147456 - 5472147968
Magic Number found. This is maybe inode !!
id = 50364420, id2 = 14, filesize = 0B
position of above data: 5472149504 - 5472150016
Magic Number found. This is maybe inode !!
id = 54558724, id2 = 15, filesize = 0B
position of above data: 5472151552 - 5472152064
Magic Number found. This is maybe inode !!
id = 58753028, id2 = 17, filesize = 0B
position of above data: 5472153600 - 5472154112
Magic Number found. This is maybe inode !!
id = 62947332, id2 = 18, filesize = 0B
position of above data: 5472155648 - 5472156160
Magic Number found. This is maybe inode !!
id = 67141636, id2 = 20, filesize = 0B
position of above data: 5472157696 - 5472158208
Magic Number found. This is maybe inode !!
id = 71335940, id2 = 21, filesize = 0B
position of above data: 5472159744 - 5472160256
Magic Number found. This is maybe inode !!
id = 75530244, id2 = 22, filesize = 0B
position of above data: 5472161792 - 5472162304
Magic Number found. This is maybe inode !!
id = 79724548, id2 = 23, filesize = 0B
position of above data: 5472163840 - 5472164352
Magic Number found. This is maybe inode !!
id = 83918852, id2 = 24, filesize = 0B
position of above data: 5472165888 - 5472166400
Magic Number found. This is maybe inode !!
id = 88113156, id2 = 25, filesize = 0B
position of above data: 5472167936 - 5472168448
Magic Number found. This is maybe inode !!
id = 92307460, id2 = 26, filesize = 0B
position of above data: 5472169984 - 5472170496
Magic Number found. This is maybe inode !!
id = 96501764, id2 = 27, filesize = 0B
position of above data: 5472172032 - 5472172544
Magic Number found. This is maybe inode !!
id = 100696068, id2 = 28, filesize = 0B
position of above data: 5472174080 - 5472174592
Magic Number found. This is maybe inode !!
id = 104890372, id2 = 29, filesize = 0B
position of above data: 5472176128 - 5472176640
Magic Number found. This is maybe inode !!
id = 109084676, id2 = 30, filesize = 0B
position of above data: 5472178176 - 5472178688
Magic Number found. This is maybe inode !!
id = 113278980, id2 = 31, filesize = 0B
position of above data: 5472180224 - 5472180736
Magic Number found. This is maybe inode !!
id = 117473284, id2 = 33, filesize = 0B
position of above data: 5472182272 - 5472182784
Magic Number found. This is maybe inode !!
id = 121667588, id2 = 34, filesize = 0B
position of above data: 5472184320 - 5472184832
Magic Number found. This is maybe inode !!
id = 125861892, id2 = 35, filesize = 0B
position of above data: 5472186368 - 5472186880
Magic Number found. This is maybe inode !!
id = 130056196, id2 = 36, filesize = 0B
position of above data: 5472188416 - 5472188928
Magic Number found. This is maybe inode !!
id = 134250500, id2 = 37, filesize = 0B
position of above data: 5472190464 - 5472190976
Magic Number found. This is maybe inode !!
id = 138444804, id2 = 38, filesize = 0B
position of above data: 5472192512 - 5472193024
Magic Number found. This is maybe inode !!
id = 142639108, id2 = 39, filesize = 0B
position of above data: 5472194560 - 5472195072
Magic Number found. This is maybe inode !!
id = 146833412, id2 = 40, filesize = 0B
position of above data: 5472196608 - 5472197120
Magic Number found. This is maybe inode !!
id = 151027716, id2 = 41, filesize = 0B
position of above data: 5472198656 - 5472199168
Magic Number found. This is maybe inode !!
id = 155222020, id2 = 42, filesize = 0B
position of above data: 5472200704 - 5472201216
Magic Number found. This is maybe inode !!
id = 159416324, id2 = 43, filesize = 0B
position of above data: 5472202752 - 5472203264
Magic Number found. This is maybe inode !!
id = 163610628, id2 = 44, filesize = 0B
position of above data: 5472204800 - 5472205312
Magic Number found. This is maybe inode !!
id = 167804932, id2 = 45, filesize = 0B
position of above data: 5472206848 - 5472207360
Magic Number found. This is maybe inode !!
id = 171999236, id2 = 46, filesize = 0B
position of above data: 5472208896 - 5472209408
Magic Number found. This is maybe inode !!
id = 176193540, id2 = 47, filesize = 0B
position of above data: 5472210944 - 5472211456
Magic Number found. This is maybe inode !!
id = 180387844, id2 = 49, filesize = 0B
position of above data: 5472212992 - 5472213504
Magic Number found. This is maybe inode !!
id = 184582148, id2 = 50, filesize = 0B
position of above data: 5472215040 - 5472215552
Magic Number found. This is maybe inode !!
id = 188776452, id2 = 51, filesize = 0B
position of above data: 5472217088 - 5472217600
Magic Number found. This is maybe inode !!
id = 192970756, id2 = 52, filesize = 0B
position of above data: 5472219136 - 5472219648
Magic Number found. This is maybe inode !!
id = 197165060, id2 = 53, filesize = 0B
position of above data: 5472221184 - 5472221696
Magic Number found. This is maybe inode !!
id = 201359364, id2 = 54, filesize = 0B
position of above data: 5472223232 - 5472223744
^CTraceback (most recent call last):
File "searchinode.py", line 64, in <module>
data = f.read(512)
KeyboardInterrupt
まとめ
- データ自体は（削除時には）多分上書きされていない
- inode の情報は部分的に抹消される
- サイズが小さくて (1kB 以下?) inode 内にデータが収まるようなファイルは取り出せる
(.vmxf や virtual disk descriptor file の方の .vmdk くらいか) -
<vmname>-flat.vmdkの方の VMDK ファイルは inode の対応するブロックの情報が消されているのでほぼ無理 - MBR くらいだったら inode に関係なく直接探し出して復旧できる
- 大事なデータはバックアップを取りましょう
参考資料
VMFS 関連
- http://glandium.org/projects/vmfs-tools/
- http://buildvirtual.net/using-vmkfstools-to-manage-vmfs-datastores/
MBR 関連
GPT 関連