2019/6/27 新規作成
2019/7/23 コマンド微修正,図の微修正,singleでも確認
はじめに
OCIのDBaaSについて、作成直後のストレージ構成を確認してみました。
前提
- 以下の環境で確認
- OCI DBaaS(DB Systems)のVM.Standard2.2
- DB ver 11.2.0.4 (RAC/Single両方で確認。以下コマンド結果はRAC版)
- Oracle Linux 6.10
- 作成時に「Available Storage Size (GB)」は"256"を指定しました
- 2019/6/26作成のDB systemで確認しています
- PaaSなので変更される可能性があります。
確認したこと
- Block Volumeのアタッチ数
- ファイルシステムの切られ方
- ASM Disk Groupの切られ方
- ASM Volumeの切られ方
確認結果
概要
- Block Volumeは9つアタッチされている
- それぞれudevのSYMLINKにて、デバイス名を固定している
- sdb ~ sbiはASMディスクとして使用される
- sdb ~ sdeはASM Disk Groupの+DATA、およびASM VolumeのCOMMONSTOREとして利用
- +DATAにはデータファイル、パラメータファイル、OCR、Voting Diskなどが格納
- sdf ~ sdiはASM Disk Groupの+RECOとして利用
- +RECOにはオンラインREDO、アーカイブREDOなどが格納
- sdb ~ sdeはASM Disk Groupの+DATA、およびASM VolumeのCOMMONSTOREとして利用
- ASM VolumeのCOMMONSTOREはacfsとして/opt/oracle/dcs/commonstoreへマウントされている
- sdjはext4として/u01へマウントされている
- Boot Volumeはsda1,sda2,sda3とパーティションが切られている
- sda1は510MBで/boot/efiへマウント（Compute OL6.10の場合210MBでfat16）
- sda2は1488MBで/bootへマウント（Compute OL6.10の場合/boot/efiと分けられていない）
- sda3は42GBでLogVolRoot(35GB)とLogVolSwap(2GB)のLVが切られている（Compute OL6.10の場合LVは切られていない）
- LogVolRootは/へマウント
- LogVolSwapはSWAP
- ストレージを拡張すると、拡張されたブロックが追加アタッチされ、リバランスで移動させてから、元のブロックが削除される
コマンド結果
ブロックデバイスを確認
[opc@dbhost2 ~]$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 46.6G 0 disk
|-sda1 8:1 0 486M 0 part /boot/efi
|-sda2 8:2 0 1.4G 0 part /boot
`-sda3 8:3 0 39.1G 0 part
|-VolGroupSys-LogVolRoot (dm-0) 249:0 0 35G 0 lvm /
`-VolGroupSys-LogVolSwap (dm-1) 249:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 64G 0 disk
`-sdb1 8:17 0 64G 0 part
sdc 8:32 0 64G 0 disk
`-sdc1 8:33 0 64G 0 part
sdd 8:48 0 64G 0 disk
`-sdd1 8:49 0 64G 0 part
sde 8:64 0 64G 0 disk
`-sde1 8:65 0 64G 0 part
sdf 8:80 0 64G 0 disk
`-sdf1 8:81 0 64G 0 part
sdg 8:96 0 64G 0 disk
`-sdg1 8:97 0 64G 0 part
sdh 8:112 0 64G 0 disk
`-sdh1 8:113 0 64G 0 part
sdi 8:128 0 64G 0 disk
`-sdi1 8:129 0 64G 0 part
sdj 8:144 0 200G 0 disk
`-sdj1 8:145 0 200G 0 part
asm!commonstore-61 248:31233 0 5G 0 disk
パーティション情報を確認
[opc@dbhost2 ~]$ sudo parted -l
Warning: Not all of the space available to /dev/sda appears to be used, you can
fix the GPT to use all of the space (an extra 3305472 blocks) or continue with
the current setting?
Fix/Ignore? Ignore
Model: ORACLE BlockVolume (scsi)
Disk /dev/sda: 50.0GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
1 2097kB 512MB 510MB fat32 primary
2 512MB 2000MB 1488MB ext4 primary
3 2000MB 44.0GB 42.0GB primary
Error: /dev/sdb: unrecognised disk label
Error: /dev/sdc: unrecognised disk label
Error: /dev/sdd: unrecognised disk label
Error: /dev/sde: unrecognised disk label
Error: /dev/sdf: unrecognised disk label
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/VolGroupSys-LogVolSwap: 2147MB
Sector size (logical/physical): 512B/4096B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 2147MB 2147MB linux-swap(v1)
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/VolGroupSys-LogVolRoot: 37.6GB
Sector size (logical/physical): 512B/4096B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 37.6GB 37.6GB ext4
Error: /dev/sdg: unrecognised disk label
Error: /dev/sdh: unrecognised disk label
Error: /dev/sdi: unrecognised disk label
Model: ORACLE BlockVolume (scsi)
Disk /dev/sdj: 215GB
Sector size (logical/physical): 512B/4096B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 215GB 215GB ext4
Error: /dev/asm/commonstore-61: unrecognised disk label
Error: /dev/ofsctl: unrecognised disk label
ブロックのFSTYPEを確認
[opc@dbhost2 ~]$ sudo lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
sda
|-sda1 vfat bootefi CD3C-893F /boot/efi
|-sda2 ext4 bootfs 58ab4698-de1d-4fac-916f-df800d826a65 /boot
`-sda3 LVM2_member g7qOjo-0121-Qf1j-jUwv-0dde-5NqK-duVwQ1
|-VolGroupSys-LogVolRoot (dm-0) ext4 7a220ac5-5d95-48ed-b966-ef14fe36cc21 /
`-VolGroupSys-LogVolSwap (dm-1) swap f2eb68e5-ee82-404d-9a7c-613a8ff68d1b [SWAP]
sdb oracleasm
`-sdb1
sdc oracleasm
`-sdc1
sdd oracleasm
`-sdd1
sde oracleasm
`-sde1
sdf oracleasm
`-sdf1
sdg oracleasm
`-sdg1
sdh oracleasm
`-sdh1
sdi oracleasm
`-sdi1
sdj ext4 7d75da54-1388-483c-95e8-d5961ba03a31
`-sdj1
asm!commonstore-61
-> sdb ~ sdiがasmであることを確認
ASMを確認
- ASMディスク
SQL> select NAME,DISK_NUMBER,GROUP_NUMBER,PATH,MOUNT_STATUS,REDUNDANCY,OS_MB,TOTAL_MB,FAILGROUP from v$asm_disk order by NAME,DISK_NUMBER;
NAME DISK_NUMBER GROUP_NUMBER PATH MOUNT_S REDUNDA OS_MB TOTAL_MB FAILGROUP
-------------------- ----------- ------------ ------------------------------ ------- ------- ---------- ---------- ------------------------------
DATA_0000 0 1 /dev/DATADISK3 CACHED UNKNOWN 65536 65536 DATA_0000
DATA_0001 1 1 /dev/DATADISK2 CACHED UNKNOWN 65536 65536 DATA_0001
DATA_0002 2 1 /dev/DATADISK1 CACHED UNKNOWN 65536 65536 DATA_0002
DATA_0003 3 1 /dev/DATADISK4 CACHED UNKNOWN 65536 65536 DATA_0003
RECODISK1 0 2 /dev/RECODISK1 CACHED UNKNOWN 65536 65536 RECODISK1
RECODISK2 1 2 /dev/RECODISK2 CACHED UNKNOWN 65536 65536 RECODISK2
RECODISK3 2 2 /dev/RECODISK3 CACHED UNKNOWN 65536 65536 RECODISK3
RECODISK4 3 2 /dev/RECODISK4 CACHED UNKNOWN 65536 65536 RECODISK4
8 rows selected.
-> PATHが見慣れない文字列
- ASMディスクグループ
[grid@dbhost1 ~]$ asmcmd lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 512 4096 4194304 262144 255220 0 255220 0 Y DATA/
MOUNTED EXTERN N 512 512 4096 4194304 262144 257644 0 257644 0 N RECO/
SQL> select NAME,GROUP_NUMBER,STATE,TYPE,TOTAL_MB from v$asm_diskgroup;
NAME GROUP_NUMBER STATE TYPE TOTAL_MB
-------------------- ------------ ----------- --------------- ----------
DATA 1 MOUNTED EXTERN 262144
RECO 2 MOUNTED EXTERN 262144
- ASMボリューム
SQL> select VOLUME_NAME,VOLUME_NUMBER,GROUP_NUMBER,SIZE_MB,REDUNDANCY,VOLUME_DEVICE,MOUNTPATH from v$asm_volume;
VOLUME_NAME VOLUME_NUMBER GROUP_NUMBER SIZE_MB REDUND VOLUME_DEVICE MOUNTPATH
------------------------------ ------------- ------------ ---------- ------ ------------------------------ ------------------------------
COMMONSTORE 1 1 5120 UNPROT /dev/asm/commonstore-61 /opt/oracle/dcs/commonstore
- ASMファイル
SQL> select GROUP_NUMBER,FILE_NUMBER,TYPE from V$ASM_FILE;
GROUP_NUMBER FILE_NUMBER TYPE
------------ ----------- ------------------------------
1 253 ASMPARAMETERFILE
1 255 OCRFILE
1 256 PASSWORD
1 257 PASSWORD
1 258 ASMVOL
1 259 DATAFILE
1 260 DATAFILE
1 261 DATAFILE
1 262 TEMPFILE
1 263 DATAFILE
1 264 PARAMETERFILE
1 265 DATAFILE
1 266 OCRBACKUP
1 267 OCRBACKUP
1 268 OCRBACKUP
2 256 CONTROLFILE
2 257 ONLINELOG
2 258 ONLINELOG
2 259 ONLINELOG
2 260 ONLINELOG
2 261 ARCHIVELOG
2 262 ARCHIVELOG
2 263 ARCHIVELOG
2 264 ARCHIVELOG
2 265 ARCHIVELOG
2 266 ARCHIVELOG
2 267 ARCHIVELOG
27 rows selected.
- OCRの場所
[grid@dbhost1 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 491684
Used space (kbytes) : 84556
Available space (kbytes) : 407128
ID : 2018798373
Device/File Name : +DATA <--★
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
- Voting Diskの場所
[grid@dbhost1 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 0785acad848a4ff0bf084ab0600111a9 (/dev/DATADISK3) [DATA]<--★
Located 1 voting disk(s).
デバイスの命名を確認
- udevのルールをみてみる
[opc@dbhost2 ~]$ sudo ls -la /etc/udev/rules.d/
total 36
drwxr-xr-x 2 root root 4096 Jun 26 10:11 .
drwxr-xr-x 4 root root 4096 Apr 16 14:56 ..
-rw-r--r-- 1 root root 1574 Apr 16 16:03 40-redhat.rules
-rw-r--r-- 1 root root 190 Jun 26 10:11 55-usm.rules
-rw-r--r-- 1 root root 316 Jan 25 2018 60-raw.rules
-rw-r--r-- 1 root root 1431 Jun 26 09:49 70-names.rules
-rw-r--r-- 1 root root 589 Jun 26 09:16 70-persistent-net.rules
-rw-r--r-- 1 root root 83 Aug 17 2014 90-hal.rules
-rw-r--r-- 1 root root 308 Dec 2 2017 98-kexec.rules
- DISKと名前で、ルールがないか検索
[opc@dbhost2 ~]$ sudo find /etc/udev/rules.d/ -type f -name "*.rules" | xargs grep 'DISK' -n
/etc/udev/rules.d/70-names.rules:1:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360edc23284d54637b1b36221a19ce904", SYMLINK+="DATADISK1", OWNER="grid", GROUP="asmadmin", MODE="0660"
/etc/udev/rules.d/70-names.rules:2:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360a92e2a294d481497aeca6749839336", SYMLINK+="DATADISK2", OWNER="grid", GROUP="asmadmin", MODE="0660"
/etc/udev/rules.d/70-names.rules:3:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360b476cdd2504f25bf1f6e4dd03480f8", SYMLINK+="DATADISK3", OWNER="grid", GROUP="asmadmin", MODE="0660"
/etc/udev/rules.d/70-names.rules:4:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36056640a618a4f65993e5107a0086c8b", SYMLINK+="DATADISK4", OWNER="grid", GROUP="asmadmin", MODE="0660"
/etc/udev/rules.d/70-names.rules:5:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36010ff9c8af14dbea0a5937fef871e28", SYMLINK+="RECODISK1", OWNER="grid", GROUP="asmadmin", MODE="0660"
/etc/udev/rules.d/70-names.rules:6:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36027594d89c741c1bd71096ca2b423e9", SYMLINK+="RECODISK2", OWNER="grid", GROUP="asmadmin", MODE="0660"
/etc/udev/rules.d/70-names.rules:7:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360c467190d3b42ddaef111492240cdc9", SYMLINK+="RECODISK3", OWNER="grid", GROUP="asmadmin", MODE="0660"
/etc/udev/rules.d/70-names.rules:8:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606e77d57af6493ebee8c85ea10ac146", SYMLINK+="RECODISK4", OWNER="grid", GROUP="asmadmin", MODE="0660"
- ルールの中身を見る
[opc@dbhost2 ~]$ sudo cat /etc/udev/rules.d/55-usm.rules
#
# ADVM devices
KERNEL=="asm/*", GROUP="asmadmin", MODE="0770"
KERNEL=="asm/.*", GROUP="asmadmin", MODE="0770"
#
# ACFS devices
KERNEL=="ofsctl", GROUP="asmadmin", MODE="0664"
[opc@dbhost2 ~]$ sudo cat /etc/udev/rules.d/70-names.rules
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360edc23284d54637b1b36221a19ce904", SYMLINK+="DATADISK1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360a92e2a294d481497aeca6749839336", SYMLINK+="DATADISK2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360b476cdd2504f25bf1f6e4dd03480f8", SYMLINK+="DATADISK3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36056640a618a4f65993e5107a0086c8b", SYMLINK+="DATADISK4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36010ff9c8af14dbea0a5937fef871e28", SYMLINK+="RECODISK1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36027594d89c741c1bd71096ca2b423e9", SYMLINK+="RECODISK2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360c467190d3b42ddaef111492240cdc9", SYMLINK+="RECODISK3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606e77d57af6493ebee8c85ea10ac146", SYMLINK+="RECODISK4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360d566eec95b4c8f99aafa655558c6ad", SYMLINK+="localdisk", OWNER="grid", GROUP="asmadmin", MODE="0660"
- 対応も確認
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sda | grep ID_SERIAL=
E: ID_SERIAL=36049ae4cf5c945378713d7eff5bd2b97
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdb | grep ID_SERIAL=
E: ID_SERIAL=360edc23284d54637b1b36221a19ce904 --> DATADISK1
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdc | grep ID_SERIAL=
E: ID_SERIAL=360a92e2a294d481497aeca6749839336 --> DATADISK2
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdd | grep ID_SERIAL=
E: ID_SERIAL=360b476cdd2504f25bf1f6e4dd03480f8 --> DATADISK3
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sde | grep ID_SERIAL=
E: ID_SERIAL=36056640a618a4f65993e5107a0086c8b --> DATADISK4
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdf | grep ID_SERIAL=
E: ID_SERIAL=36010ff9c8af14dbea0a5937fef871e28 --> RECODISK1
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdg | grep ID_SERIAL=
E: ID_SERIAL=36027594d89c741c1bd71096ca2b423e9 --> RECODISK2
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdh | grep ID_SERIAL=
E: ID_SERIAL=360c467190d3b42ddaef111492240cdc9 --> RECODISK3
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdi | grep ID_SERIAL=
E: ID_SERIAL=3606e77d57af6493ebee8c85ea10ac146 --> RECODISK4
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdj | grep ID_SERIAL=
E: ID_SERIAL=360d566eec95b4c8f99aafa655558c6ad --> localdisk
[opc@dbhost2 ~]$ sudo ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 9 Jun 26 11:40 ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1 -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 26 09:49 ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 26 09:49 ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 26 09:49 ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Jun 26 13:00 ip-169.254.2.10:3260-iscsi-iqn.2015-12.com.oracleiaas:d0a95eeb-9faf-4129-9dbd-d65dedf94f65-lun-1 -> ../../sdc
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.10:3260-iscsi-iqn.2015-12.com.oracleiaas:d0a95eeb-9faf-4129-9dbd-d65dedf94f65-lun-1-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 Jun 26 11:40 ip-169.254.2.2:3260-iscsi-iqn.2015-12.com.oracleiaas:8ff76e53-8a0a-40ae-be86-3783f66fe750-lun-1 -> ../../sdj
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.2:3260-iscsi-iqn.2015-12.com.oracleiaas:8ff76e53-8a0a-40ae-be86-3783f66fe750-lun-1-part1 -> ../../sdj1
lrwxrwxrwx 1 root root 9 Jun 26 13:00 ip-169.254.2.3:3260-iscsi-iqn.2015-12.com.oracleiaas:05a4a7d3-e4cf-43f4-921d-83e31c1e364a-lun-1 -> ../../sdf
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.3:3260-iscsi-iqn.2015-12.com.oracleiaas:05a4a7d3-e4cf-43f4-921d-83e31c1e364a-lun-1-part1 -> ../../sdf1
lrwxrwxrwx 1 root root 9 Jun 26 12:23 ip-169.254.2.4:3260-iscsi-iqn.2015-12.com.oracleiaas:00467c13-215d-483d-8291-630a6ea256bd-lun-1 -> ../../sdd
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.4:3260-iscsi-iqn.2015-12.com.oracleiaas:00467c13-215d-483d-8291-630a6ea256bd-lun-1-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 9 Jun 26 13:00 ip-169.254.2.5:3260-iscsi-iqn.2015-12.com.oracleiaas:637ed59e-b517-44c2-9f8d-828f8159ed57-lun-1 -> ../../sdh
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.5:3260-iscsi-iqn.2015-12.com.oracleiaas:637ed59e-b517-44c2-9f8d-828f8159ed57-lun-1-part1 -> ../../sdh1
lrwxrwxrwx 1 root root 9 Jun 26 13:00 ip-169.254.2.6:3260-iscsi-iqn.2015-12.com.oracleiaas:ae744a7f-23df-469f-aa5f-6bfdb9685f37-lun-1 -> ../../sdg
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.6:3260-iscsi-iqn.2015-12.com.oracleiaas:ae744a7f-23df-469f-aa5f-6bfdb9685f37-lun-1-part1 -> ../../sdg1
lrwxrwxrwx 1 root root 9 Jun 26 12:23 ip-169.254.2.7:3260-iscsi-iqn.2015-12.com.oracleiaas:c933f73c-87bc-44f5-ad32-657a77e19198-lun-1 -> ../../sdb
lrwxrwxrwx 1 root root 10 Jun 26 09:49 ip-169.254.2.7:3260-iscsi-iqn.2015-12.com.oracleiaas:c933f73c-87bc-44f5-ad32-657a77e19198-lun-1-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 Jun 26 13:00 ip-169.254.2.8:3260-iscsi-iqn.2015-12.com.oracleiaas:d169d00d-926e-4ba1-ae79-d25ad6b57d56-lun-1 -> ../../sde
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.8:3260-iscsi-iqn.2015-12.com.oracleiaas:d169d00d-926e-4ba1-ae79-d25ad6b57d56-lun-1-part1 -> ../../sde1
lrwxrwxrwx 1 root root 9 Jun 26 13:00 ip-169.254.2.9:3260-iscsi-iqn.2015-12.com.oracleiaas:c84336f3-21f7-4062-8d3b-044f4a2d7aa0-lun-1 -> ../../sdi
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.9:3260-iscsi-iqn.2015-12.com.oracleiaas:c84336f3-21f7-4062-8d3b-044f4a2d7aa0-lun-1-part1 -> ../../sdi1
(おまけ) Storage拡張
Available Storage Sizeを256GBから512GBへ変更
拡張後
- data用のブロックが大きくなっている（今回のスケールアップではRECOに変わりはない）
[opc@dbhost2 ~]$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 46.6G 0 disk
|-sda1 8:1 0 486M 0 part /boot/efi
|-sda2 8:2 0 1.4G 0 part /boot
`-sda3 8:3 0 39.1G 0 part
|-VolGroupSys-LogVolRoot (dm-0) 249:0 0 35G 0 lvm /
`-VolGroupSys-LogVolSwap (dm-1) 249:1 0 2G 0 lvm [SWAP]
sdf 8:80 0 64G 0 disk
`-sdf1 8:81 0 64G 0 part
sdg 8:96 0 64G 0 disk
`-sdg1 8:97 0 64G 0 part
sdh 8:112 0 64G 0 disk
`-sdh1 8:113 0 64G 0 part
sdi 8:128 0 64G 0 disk
`-sdi1 8:129 0 64G 0 part
sdj 8:144 0 200G 0 disk
`-sdj1 8:145 0 200G 0 part
sdk 8:160 0 128G 0 disk ★
`-sdk1 8:161 0 128G 0 part
sdl 8:176 0 128G 0 disk ★
`-sdl1 8:177 0 128G 0 part
sdm 8:192 0 128G 0 disk ★
`-sdm1 8:193 0 128G 0 part
sdn 8:208 0 128G 0 disk ★
`-sdn1 8:209 0 128G 0 part
asm!commonstore-61 248:31233 0 5G 0 disk
-> RECOに使ってたディスク以外のサイズが64GBずつ増加。
-> sdb ~ sdeはなくなり、sdk ~ sdnへ
- シリアルIDが変わってる
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdk | grep ID_SERIAL=
E: ID_SERIAL=360de2a6205ef4b4b9d2368812b28abee
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdl | grep ID_SERIAL=
E: ID_SERIAL=360c9ebbedfff4f3b84554d41440b238c
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdm | grep ID_SERIAL=
E: ID_SERIAL=3605f79bae3174834a01bbf0017fa1cce
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdn | grep ID_SERIAL=
E: ID_SERIAL=360da995ed1c844a3a24438dbc83f09f2
[opc@dbhost2 ~]$ sudo cat /etc/udev/rules.d/70-names.rules
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36010ff9c8af14dbea0a5937fef871e28", SYMLINK+="RECODISK1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36027594d89c741c1bd71096ca2b423e9", SYMLINK+="RECODISK2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360c467190d3b42ddaef111492240cdc9", SYMLINK+="RECODISK3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606e77d57af6493ebee8c85ea10ac146", SYMLINK+="RECODISK4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360d566eec95b4c8f99aafa655558c6ad", SYMLINK+="localdisk", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360de2a6205ef4b4b9d2368812b28abee", SYMLINK+="DATADISK5", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360c9ebbedfff4f3b84554d41440b238c", SYMLINK+="DATADISK6", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3605f79bae3174834a01bbf0017fa1cce", SYMLINK+="DATADISK7", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360da995ed1c844a3a24438dbc83f09f2", SYMLINK+="DATADISK8", OWNER="grid", GROUP="asmadmin", MODE="0660"
-> 再度ストレージ拡張を行うと、DATA9,10,11,12となった。
- ASM
SQL> select NAME,DISK_NUMBER,GROUP_NUMBER,PATH,MOUNT_STATUS,REDUNDANCY,OS_MB,TOTAL_MB,FAILGROUP from v$asm_disk order by NAME,DISK_NUMBER;
NAME DISK_NUMBER GROUP_NUMBER PATH MOUNT_S REDUNDA OS_MB TOTAL_MB FAILGROUP
------------------------------ ----------- ------------ ------------------------------ ------- ------- ---------- ---------- ------------------------------
DATA_0004 4 1 /dev/DATADISK5 CACHED UNKNOWN 131072 131072 DATA_0004
DATA_0005 5 1 /dev/DATADISK6 CACHED UNKNOWN 131072 131072 DATA_0005
DATA_0006 6 1 /dev/DATADISK7 CACHED UNKNOWN 131072 131072 DATA_0006
DATA_0007 7 1 /dev/DATADISK8 CACHED UNKNOWN 131072 131072 DATA_0007
RECODISK1 0 2 /dev/RECODISK1 CACHED UNKNOWN 65536 65536 RECODISK1
RECODISK2 1 2 /dev/RECODISK2 CACHED UNKNOWN 65536 65536 RECODISK2
RECODISK3 2 2 /dev/RECODISK3 CACHED UNKNOWN 65536 65536 RECODISK3
RECODISK4 3 2 /dev/RECODISK4 CACHED UNKNOWN 65536 65536 RECODISK4
8 rows selected.
→ 再度ストレージ拡張を行うと、DATA_0000,0001,0002,0003へ戻った
SQL> select NAME,GROUP_NUMBER,STATE,TYPE,TOTAL_MB from v$asm_diskgroup;
NAME GROUP_NUMBER STATE TYPE TOTAL_MB
------------------------------ ------------ ----------- ------ ----------
DATA 1 MOUNTED EXTERN 524288
RECO 2 MOUNTED EXTERN 262144
SQL> select VOLUME_NAME,VOLUME_NUMBER,GROUP_NUMBER,SIZE_MB,REDUNDANCY,VOLUME_DEVICE,MOUNTPATH from v$asm_volume;
VOLUME_NAME VOLUME_NUMBER GROUP_NUMBER SIZE_MB REDUND VOLUME_DEVICE MOUNTPATH
--------------- ------------- ------------ ---------- ------ ------------------------------ ------------------------------
COMMONSTORE 1 1 5120 UNPROT /dev/asm/commonstore-61 /opt/oracle/dcs/commonstore
-> ASM Volumeは変化無し
拡張中の動き
一時的にデバイスが4つ増え、ASMでリバランスした後、不要となったデバイスを削除する動きの模様。
（以下、再拡張時のログのため上とデバイス名等が合っていない）
- デバイス
デバイスが4つ増え、もともとデータが入っていたディスクが削除された。
iostat 1
avg-cpu: %user %nice %system %iowait %steal %idle
11.42 0.00 11.93 1.78 0.00 74.87
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 21.00 0.00 192.00 0 192
dm-0 21.00 0.00 192.00 0 192
dm-1 0.00 0.00 0.00 0 0
sdf 80.00 640.00 0.00 640 0
sdg 80.00 640.00 0.00 640 0
sdh 71.00 568.00 0.00 568 0
sdi 81.00 672.00 0.00 672 0
sdj 87.00 640.00 144.00 640 144
asm!commonstore-61 0.00 0.00 0.00 0 0
sdk 84.00 642.00 9.00 642 9
sdl 80.00 640.00 0.00 640 0
sdm 80.00 640.00 0.00 640 0
sdn 80.00 640.00 0.00 640 0
sdo 83.00 664.00 0.00 664 0
sdp 83.00 664.00 0.00 664 0
sdq 83.00 664.00 0.00 664 0
sdr 83.00 664.00 0.00 664 0
- ASMディスク
一時的に8つになった
asmcmd iostat --io -G DATA 2
Group_Name Dsk_Name Reads Writes
DATA DATA_0000 3.00 0.00
DATA DATA_0001 3.00 0.00
DATA DATA_0002 3.00 0.00
DATA DATA_0003 3.00 0.00
DATA DATA_0004 6.00 0.00
DATA DATA_0005 3.00 0.00
DATA DATA_0006 3.00 0.00
DATA DATA_0007 3.00 4.00
Group_Name Dsk_Name Reads Writes
DATA DATA_0000 3.00 26.00
DATA DATA_0001 2.00 0.00
DATA DATA_0002 2.00 0.00
DATA DATA_0003 2.00 0.00
DATA DATA_0004 2.00 6.00
DATA DATA_0005 2.00 0.00
DATA DATA_0006 2.00 0.00
DATA DATA_0007 2.00 2.00
- リバランス処理
REBAL RUNをとらえられず…ｗ
SQL> r
1* select GROUP_NUMBER, OPERATION, STATE, POWER, ACTUAL,SOFAR, EST_WORK, EST_RATE, EST_MINUTES from V$ASM_OPERATION
GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- -----------
1 REBAL WAIT 1
1 REBAL WAIT 1
1 REBAL WAIT 1
以上