10
7

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

Oracle Cloud(OCI) DBaaSのデフォルト・ストレージ構成

Last updated at Posted at 2019-06-26

2019/6/27 新規作成
2019/7/23 コマンド微修正,図の微修正,singleでも確認

はじめに

OCIのDBaaSについて、作成直後のストレージ構成を確認してみました。

前提

  • 以下の環境で確認
    • OCI DBaaS(DB Systems)のVM.Standard2.2
    • DB ver 11.2.0.4 (RAC/Single両方で確認。以下コマンド結果はRAC版)
    • Oracle Linux 6.10
  • 作成時に「Available Storage Size (GB)」は"256"を指定しました
  • 2019/6/26作成のDB systemで確認しています
    • PaaSなので変更される可能性があります。

確認したこと

  • Block Volumeのアタッチ数
  • ファイルシステムの切られ方
  • ASM Disk Groupの切られ方
  • ASM Volumeの切られ方

確認結果

概要

image.png

  • Block Volumeは9つアタッチされている
    • それぞれudevのSYMLINKにて、デバイス名を固定している
  • sdb ~ sbiはASMディスクとして使用される
    • sdb ~ sdeはASM Disk Groupの+DATA、およびASM VolumeのCOMMONSTOREとして利用
      • +DATAにはデータファイル、パラメータファイル、OCR、Voting Diskなどが格納
    • sdf ~ sdiはASM Disk Groupの+RECOとして利用
      • +RECOにはオンラインREDO、アーカイブREDOなどが格納
  • ASM VolumeのCOMMONSTOREはacfsとして/opt/oracle/dcs/commonstoreへマウントされている
  • sdjはext4として/u01へマウントされている
  • Boot Volumeはsda1,sda2,sda3とパーティションが切られている
    • sda1は510MBで/boot/efiへマウント(Compute OL6.10の場合210MBでfat16)
    • sda2は1488MBで/bootへマウント(Compute OL6.10の場合/boot/efiと分けられていない)
    • sda3は42GBでLogVolRoot(35GB)とLogVolSwap(2GB)のLVが切られている(Compute OL6.10の場合LVは切られていない)
      • LogVolRootは/へマウント
      • LogVolSwapはSWAP
  • ストレージを拡張すると、拡張されたブロックが追加アタッチされ、リバランスで移動させてから、元のブロックが削除される

コマンド結果

ブロックデバイスを確認

[opc@dbhost2 ~]$ sudo lsblk
NAME                              MAJ:MIN   RM  SIZE RO TYPE MOUNTPOINT
sda                                 8:0      0 46.6G  0 disk
|-sda1                              8:1      0  486M  0 part /boot/efi
|-sda2                              8:2      0  1.4G  0 part /boot
`-sda3                              8:3      0 39.1G  0 part
  |-VolGroupSys-LogVolRoot (dm-0) 249:0      0   35G  0 lvm  /
  `-VolGroupSys-LogVolSwap (dm-1) 249:1      0    2G  0 lvm  [SWAP]
sdb                                 8:16     0   64G  0 disk
`-sdb1                              8:17     0   64G  0 part
sdc                                 8:32     0   64G  0 disk
`-sdc1                              8:33     0   64G  0 part
sdd                                 8:48     0   64G  0 disk
`-sdd1                              8:49     0   64G  0 part
sde                                 8:64     0   64G  0 disk
`-sde1                              8:65     0   64G  0 part
sdf                                 8:80     0   64G  0 disk
`-sdf1                              8:81     0   64G  0 part
sdg                                 8:96     0   64G  0 disk
`-sdg1                              8:97     0   64G  0 part
sdh                                 8:112    0   64G  0 disk
`-sdh1                              8:113    0   64G  0 part
sdi                                 8:128    0   64G  0 disk
`-sdi1                              8:129    0   64G  0 part
sdj                                 8:144    0  200G  0 disk
`-sdj1                              8:145    0  200G  0 part
asm!commonstore-61                248:31233  0    5G  0 disk

パーティション情報を確認

[opc@dbhost2 ~]$ sudo parted -l
Warning: Not all of the space available to /dev/sda appears to be used, you can
fix the GPT to use all of the space (an extra 3305472 blocks) or continue with
the current setting?
Fix/Ignore? Ignore
Model: ORACLE BlockVolume (scsi)
Disk /dev/sda: 50.0GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      2097kB  512MB   510MB   fat32        primary
 2      512MB   2000MB  1488MB  ext4         primary
 3      2000MB  44.0GB  42.0GB               primary


Error: /dev/sdb: unrecognised disk label

Error: /dev/sdc: unrecognised disk label

Error: /dev/sdd: unrecognised disk label

Error: /dev/sde: unrecognised disk label

Error: /dev/sdf: unrecognised disk label

Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/VolGroupSys-LogVolSwap: 2147MB
Sector size (logical/physical): 512B/4096B
Partition Table: loop

Number  Start  End     Size    File system     Flags
 1      0.00B  2147MB  2147MB  linux-swap(v1)


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/VolGroupSys-LogVolRoot: 37.6GB
Sector size (logical/physical): 512B/4096B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  37.6GB  37.6GB  ext4


Error: /dev/sdg: unrecognised disk label

Error: /dev/sdh: unrecognised disk label

Error: /dev/sdi: unrecognised disk label

Model: ORACLE BlockVolume (scsi)
Disk /dev/sdj: 215GB
Sector size (logical/physical): 512B/4096B
Partition Table: loop

Number  Start  End    Size   File system  Flags
 1      0.00B  215GB  215GB  ext4


Error: /dev/asm/commonstore-61: unrecognised disk label

Error: /dev/ofsctl: unrecognised disk label

ブロックのFSTYPEを確認

[opc@dbhost2 ~]$ sudo lsblk -f
NAME                              FSTYPE      LABEL   UUID                                   MOUNTPOINT
sda
|-sda1                            vfat        bootefi CD3C-893F                              /boot/efi
|-sda2                            ext4        bootfs  58ab4698-de1d-4fac-916f-df800d826a65   /boot
`-sda3                            LVM2_member         g7qOjo-0121-Qf1j-jUwv-0dde-5NqK-duVwQ1
  |-VolGroupSys-LogVolRoot (dm-0) ext4                7a220ac5-5d95-48ed-b966-ef14fe36cc21   /
  `-VolGroupSys-LogVolSwap (dm-1) swap                f2eb68e5-ee82-404d-9a7c-613a8ff68d1b   [SWAP]
sdb                               oracleasm
`-sdb1
sdc                               oracleasm
`-sdc1
sdd                               oracleasm
`-sdd1
sde                               oracleasm
`-sde1
sdf                               oracleasm
`-sdf1
sdg                               oracleasm
`-sdg1
sdh                               oracleasm
`-sdh1
sdi                               oracleasm
`-sdi1
sdj                               ext4                7d75da54-1388-483c-95e8-d5961ba03a31
`-sdj1
asm!commonstore-61

-> sdb ~ sdiがasmであることを確認

ASMを確認

  • ASMディスク
SQL> select NAME,DISK_NUMBER,GROUP_NUMBER,PATH,MOUNT_STATUS,REDUNDANCY,OS_MB,TOTAL_MB,FAILGROUP from v$asm_disk order by  NAME,DISK_NUMBER;

NAME                 DISK_NUMBER GROUP_NUMBER PATH                           MOUNT_S REDUNDA      OS_MB   TOTAL_MB FAILGROUP
-------------------- ----------- ------------ ------------------------------ ------- ------- ---------- ---------- ------------------------------
DATA_0000                      0            1 /dev/DATADISK3                 CACHED  UNKNOWN      65536      65536 DATA_0000
DATA_0001                      1            1 /dev/DATADISK2                 CACHED  UNKNOWN      65536      65536 DATA_0001
DATA_0002                      2            1 /dev/DATADISK1                 CACHED  UNKNOWN      65536      65536 DATA_0002
DATA_0003                      3            1 /dev/DATADISK4                 CACHED  UNKNOWN      65536      65536 DATA_0003
RECODISK1                      0            2 /dev/RECODISK1                 CACHED  UNKNOWN      65536      65536 RECODISK1
RECODISK2                      1            2 /dev/RECODISK2                 CACHED  UNKNOWN      65536      65536 RECODISK2
RECODISK3                      2            2 /dev/RECODISK3                 CACHED  UNKNOWN      65536      65536 RECODISK3
RECODISK4                      3            2 /dev/RECODISK4                 CACHED  UNKNOWN      65536      65536 RECODISK4

8 rows selected.

-> PATHが見慣れない文字列

  • ASMディスクグループ
[grid@dbhost1 ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304    262144   255220                0          255220              0             Y  DATA/
MOUNTED  EXTERN  N         512             512   4096  4194304    262144   257644                0          257644              0             N  RECO/
SQL> select NAME,GROUP_NUMBER,STATE,TYPE,TOTAL_MB from v$asm_diskgroup;
NAME GROUP_NUMBER STATE TYPE TOTAL_MB
-------------------- ------------ ----------- --------------- ----------
DATA 1 MOUNTED EXTERN 262144
RECO 2 MOUNTED EXTERN 262144
  • ASMボリューム
SQL> select VOLUME_NAME,VOLUME_NUMBER,GROUP_NUMBER,SIZE_MB,REDUNDANCY,VOLUME_DEVICE,MOUNTPATH from v$asm_volume;

VOLUME_NAME                    VOLUME_NUMBER GROUP_NUMBER    SIZE_MB REDUND VOLUME_DEVICE                  MOUNTPATH
------------------------------ ------------- ------------ ---------- ------ ------------------------------ ------------------------------
COMMONSTORE                                1            1       5120 UNPROT /dev/asm/commonstore-61        /opt/oracle/dcs/commonstore
  • ASMファイル
SQL> select GROUP_NUMBER,FILE_NUMBER,TYPE from V$ASM_FILE;

GROUP_NUMBER FILE_NUMBER TYPE
------------ ----------- ------------------------------
           1         253 ASMPARAMETERFILE
           1         255 OCRFILE
           1         256 PASSWORD
           1         257 PASSWORD
           1         258 ASMVOL
           1         259 DATAFILE
           1         260 DATAFILE
           1         261 DATAFILE
           1         262 TEMPFILE
           1         263 DATAFILE
           1         264 PARAMETERFILE
           1         265 DATAFILE
           1         266 OCRBACKUP
           1         267 OCRBACKUP
           1         268 OCRBACKUP
           2         256 CONTROLFILE
           2         257 ONLINELOG
           2         258 ONLINELOG
           2         259 ONLINELOG
           2         260 ONLINELOG
           2         261 ARCHIVELOG
           2         262 ARCHIVELOG
           2         263 ARCHIVELOG
           2         264 ARCHIVELOG
           2         265 ARCHIVELOG
           2         266 ARCHIVELOG
           2         267 ARCHIVELOG

27 rows selected.
  • OCRの場所
[grid@dbhost1 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     491684
         Used space (kbytes)      :      84556
         Available space (kbytes) :     407128
         ID                       : 2018798373
         Device/File Name         :      +DATA <--★
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

  • Voting Diskの場所
[grid@dbhost1 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   0785acad848a4ff0bf084ab0600111a9 (/dev/DATADISK3) [DATA]<--★
Located 1 voting disk(s).

デバイスの命名を確認

  • udevのルールをみてみる
[opc@dbhost2 ~]$ sudo ls -la /etc/udev/rules.d/
total 36
drwxr-xr-x 2 root root 4096 Jun 26 10:11 .
drwxr-xr-x 4 root root 4096 Apr 16 14:56 ..
-rw-r--r-- 1 root root 1574 Apr 16 16:03 40-redhat.rules
-rw-r--r-- 1 root root  190 Jun 26 10:11 55-usm.rules
-rw-r--r-- 1 root root  316 Jan 25  2018 60-raw.rules
-rw-r--r-- 1 root root 1431 Jun 26 09:49 70-names.rules
-rw-r--r-- 1 root root  589 Jun 26 09:16 70-persistent-net.rules
-rw-r--r-- 1 root root   83 Aug 17  2014 90-hal.rules
-rw-r--r-- 1 root root  308 Dec  2  2017 98-kexec.rules
  • DISKと名前で、ルールがないか検索
[opc@dbhost2 ~]$ sudo find /etc/udev/rules.d/ -type f -name "*.rules" | xargs grep 'DISK' -n
/etc/udev/rules.d/70-names.rules:1:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360edc23284d54637b1b36221a19ce904", SYMLINK+="DATADISK1",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:2:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360a92e2a294d481497aeca6749839336", SYMLINK+="DATADISK2",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:3:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360b476cdd2504f25bf1f6e4dd03480f8", SYMLINK+="DATADISK3",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:4:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36056640a618a4f65993e5107a0086c8b", SYMLINK+="DATADISK4",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:5:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36010ff9c8af14dbea0a5937fef871e28", SYMLINK+="RECODISK1",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:6:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36027594d89c741c1bd71096ca2b423e9", SYMLINK+="RECODISK2",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:7:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360c467190d3b42ddaef111492240cdc9", SYMLINK+="RECODISK3",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:8:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606e77d57af6493ebee8c85ea10ac146", SYMLINK+="RECODISK4",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
  • ルールの中身を見る
[opc@dbhost2 ~]$ sudo cat /etc/udev/rules.d/55-usm.rules
#
# ADVM devices
KERNEL=="asm/*",      GROUP="asmadmin", MODE="0770"
KERNEL=="asm/.*",     GROUP="asmadmin", MODE="0770"
#
# ACFS devices
KERNEL=="ofsctl",     GROUP="asmadmin", MODE="0664"
[opc@dbhost2 ~]$ sudo cat /etc/udev/rules.d/70-names.rules
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360edc23284d54637b1b36221a19ce904", SYMLINK+="DATADISK1",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360a92e2a294d481497aeca6749839336", SYMLINK+="DATADISK2",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360b476cdd2504f25bf1f6e4dd03480f8", SYMLINK+="DATADISK3",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36056640a618a4f65993e5107a0086c8b", SYMLINK+="DATADISK4",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36010ff9c8af14dbea0a5937fef871e28", SYMLINK+="RECODISK1",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36027594d89c741c1bd71096ca2b423e9", SYMLINK+="RECODISK2",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360c467190d3b42ddaef111492240cdc9", SYMLINK+="RECODISK3",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606e77d57af6493ebee8c85ea10ac146", SYMLINK+="RECODISK4",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360d566eec95b4c8f99aafa655558c6ad", SYMLINK+="localdisk",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
  • 対応も確認
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sda | grep ID_SERIAL=
E: ID_SERIAL=36049ae4cf5c945378713d7eff5bd2b97													
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdb | grep ID_SERIAL=
E: ID_SERIAL=360edc23284d54637b1b36221a19ce904														--> DATADISK1
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdc | grep ID_SERIAL=
E: ID_SERIAL=360a92e2a294d481497aeca6749839336														--> DATADISK2
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdd | grep ID_SERIAL=
E: ID_SERIAL=360b476cdd2504f25bf1f6e4dd03480f8														--> DATADISK3
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sde | grep ID_SERIAL=
E: ID_SERIAL=36056640a618a4f65993e5107a0086c8b														--> DATADISK4
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdf | grep ID_SERIAL=
E: ID_SERIAL=36010ff9c8af14dbea0a5937fef871e28														--> RECODISK1
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdg | grep ID_SERIAL=
E: ID_SERIAL=36027594d89c741c1bd71096ca2b423e9														--> RECODISK2
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdh | grep ID_SERIAL=
E: ID_SERIAL=360c467190d3b42ddaef111492240cdc9														--> RECODISK3
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdi | grep ID_SERIAL=
E: ID_SERIAL=3606e77d57af6493ebee8c85ea10ac146														--> RECODISK4
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdj | grep ID_SERIAL=
E: ID_SERIAL=360d566eec95b4c8f99aafa655558c6ad														--> localdisk

 [opc@dbhost2 ~]$ sudo  ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root  9 Jun 26 11:40 ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1 -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 26 09:49 ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 26 09:49 ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 26 09:49 ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1-part3 -> ../../sda3
lrwxrwxrwx 1 root root  9 Jun 26 13:00 ip-169.254.2.10:3260-iscsi-iqn.2015-12.com.oracleiaas:d0a95eeb-9faf-4129-9dbd-d65dedf94f65-lun-1 -> ../../sdc
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.10:3260-iscsi-iqn.2015-12.com.oracleiaas:d0a95eeb-9faf-4129-9dbd-d65dedf94f65-lun-1-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  9 Jun 26 11:40 ip-169.254.2.2:3260-iscsi-iqn.2015-12.com.oracleiaas:8ff76e53-8a0a-40ae-be86-3783f66fe750-lun-1 -> ../../sdj
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.2:3260-iscsi-iqn.2015-12.com.oracleiaas:8ff76e53-8a0a-40ae-be86-3783f66fe750-lun-1-part1 -> ../../sdj1
lrwxrwxrwx 1 root root  9 Jun 26 13:00 ip-169.254.2.3:3260-iscsi-iqn.2015-12.com.oracleiaas:05a4a7d3-e4cf-43f4-921d-83e31c1e364a-lun-1 -> ../../sdf
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.3:3260-iscsi-iqn.2015-12.com.oracleiaas:05a4a7d3-e4cf-43f4-921d-83e31c1e364a-lun-1-part1 -> ../../sdf1
lrwxrwxrwx 1 root root  9 Jun 26 12:23 ip-169.254.2.4:3260-iscsi-iqn.2015-12.com.oracleiaas:00467c13-215d-483d-8291-630a6ea256bd-lun-1 -> ../../sdd
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.4:3260-iscsi-iqn.2015-12.com.oracleiaas:00467c13-215d-483d-8291-630a6ea256bd-lun-1-part1 -> ../../sdd1
lrwxrwxrwx 1 root root  9 Jun 26 13:00 ip-169.254.2.5:3260-iscsi-iqn.2015-12.com.oracleiaas:637ed59e-b517-44c2-9f8d-828f8159ed57-lun-1 -> ../../sdh
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.5:3260-iscsi-iqn.2015-12.com.oracleiaas:637ed59e-b517-44c2-9f8d-828f8159ed57-lun-1-part1 -> ../../sdh1
lrwxrwxrwx 1 root root  9 Jun 26 13:00 ip-169.254.2.6:3260-iscsi-iqn.2015-12.com.oracleiaas:ae744a7f-23df-469f-aa5f-6bfdb9685f37-lun-1 -> ../../sdg
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.6:3260-iscsi-iqn.2015-12.com.oracleiaas:ae744a7f-23df-469f-aa5f-6bfdb9685f37-lun-1-part1 -> ../../sdg1
lrwxrwxrwx 1 root root  9 Jun 26 12:23 ip-169.254.2.7:3260-iscsi-iqn.2015-12.com.oracleiaas:c933f73c-87bc-44f5-ad32-657a77e19198-lun-1 -> ../../sdb
lrwxrwxrwx 1 root root 10 Jun 26 09:49 ip-169.254.2.7:3260-iscsi-iqn.2015-12.com.oracleiaas:c933f73c-87bc-44f5-ad32-657a77e19198-lun-1-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  9 Jun 26 13:00 ip-169.254.2.8:3260-iscsi-iqn.2015-12.com.oracleiaas:d169d00d-926e-4ba1-ae79-d25ad6b57d56-lun-1 -> ../../sde
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.8:3260-iscsi-iqn.2015-12.com.oracleiaas:d169d00d-926e-4ba1-ae79-d25ad6b57d56-lun-1-part1 -> ../../sde1
lrwxrwxrwx 1 root root  9 Jun 26 13:00 ip-169.254.2.9:3260-iscsi-iqn.2015-12.com.oracleiaas:c84336f3-21f7-4062-8d3b-044f4a2d7aa0-lun-1 -> ../../sdi
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.9:3260-iscsi-iqn.2015-12.com.oracleiaas:c84336f3-21f7-4062-8d3b-044f4a2d7aa0-lun-1-part1 -> ../../sdi1

(おまけ) Storage拡張

Available Storage Sizeを256GBから512GBへ変更

拡張後

  • data用のブロックが大きくなっている(今回のスケールアップではRECOに変わりはない)
[opc@dbhost2 ~]$ sudo lsblk
NAME                              MAJ:MIN   RM  SIZE RO TYPE MOUNTPOINT
sda                                 8:0      0 46.6G  0 disk
|-sda1                              8:1      0  486M  0 part /boot/efi
|-sda2                              8:2      0  1.4G  0 part /boot
`-sda3                              8:3      0 39.1G  0 part
  |-VolGroupSys-LogVolRoot (dm-0) 249:0      0   35G  0 lvm  /
  `-VolGroupSys-LogVolSwap (dm-1) 249:1      0    2G  0 lvm  [SWAP]
sdf                                 8:80     0   64G  0 disk
`-sdf1                              8:81     0   64G  0 part
sdg                                 8:96     0   64G  0 disk
`-sdg1                              8:97     0   64G  0 part
sdh                                 8:112    0   64G  0 disk
`-sdh1                              8:113    0   64G  0 part
sdi                                 8:128    0   64G  0 disk
`-sdi1                              8:129    0   64G  0 part
sdj                                 8:144    0  200G  0 disk 
`-sdj1                              8:145    0  200G  0 part
sdk                                 8:160    0  128G  0 disk ★
`-sdk1                              8:161    0  128G  0 part
sdl                                 8:176    0  128G  0 disk ★
`-sdl1                              8:177    0  128G  0 part
sdm                                 8:192    0  128G  0 disk ★
`-sdm1                              8:193    0  128G  0 part
sdn                                 8:208    0  128G  0 disk ★
`-sdn1                              8:209    0  128G  0 part
asm!commonstore-61                248:31233  0    5G  0 disk

-> RECOに使ってたディスク以外のサイズが64GBずつ増加。
-> sdb ~ sdeはなくなり、sdk ~ sdnへ

  • シリアルIDが変わってる
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdk | grep ID_SERIAL=
E: ID_SERIAL=360de2a6205ef4b4b9d2368812b28abee
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdl | grep ID_SERIAL=
E: ID_SERIAL=360c9ebbedfff4f3b84554d41440b238c
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdm | grep ID_SERIAL=
E: ID_SERIAL=3605f79bae3174834a01bbf0017fa1cce
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdn | grep ID_SERIAL=
E: ID_SERIAL=360da995ed1c844a3a24438dbc83f09f2
[opc@dbhost2 ~]$ sudo cat /etc/udev/rules.d/70-names.rules
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36010ff9c8af14dbea0a5937fef871e28", SYMLINK+="RECODISK1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36027594d89c741c1bd71096ca2b423e9", SYMLINK+="RECODISK2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360c467190d3b42ddaef111492240cdc9", SYMLINK+="RECODISK3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606e77d57af6493ebee8c85ea10ac146", SYMLINK+="RECODISK4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360d566eec95b4c8f99aafa655558c6ad", SYMLINK+="localdisk", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360de2a6205ef4b4b9d2368812b28abee", SYMLINK+="DATADISK5", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360c9ebbedfff4f3b84554d41440b238c", SYMLINK+="DATADISK6", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3605f79bae3174834a01bbf0017fa1cce", SYMLINK+="DATADISK7", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360da995ed1c844a3a24438dbc83f09f2", SYMLINK+="DATADISK8", OWNER="grid", GROUP="asmadmin", MODE="0660"

-> 再度ストレージ拡張を行うと、DATA9,10,11,12となった。

  • ASM
SQL> select NAME,DISK_NUMBER,GROUP_NUMBER,PATH,MOUNT_STATUS,REDUNDANCY,OS_MB,TOTAL_MB,FAILGROUP from v$asm_disk order by  NAME,DISK_NUMBER;

NAME                           DISK_NUMBER GROUP_NUMBER PATH                           MOUNT_S REDUNDA      OS_MB   TOTAL_MB FAILGROUP
------------------------------ ----------- ------------ ------------------------------ ------- ------- ---------- ---------- ------------------------------
DATA_0004                                4            1 /dev/DATADISK5                 CACHED  UNKNOWN     131072     131072 DATA_0004
DATA_0005                                5            1 /dev/DATADISK6                 CACHED  UNKNOWN     131072     131072 DATA_0005
DATA_0006                                6            1 /dev/DATADISK7                 CACHED  UNKNOWN     131072     131072 DATA_0006
DATA_0007                                7            1 /dev/DATADISK8                 CACHED  UNKNOWN     131072     131072 DATA_0007
RECODISK1                                0            2 /dev/RECODISK1                 CACHED  UNKNOWN      65536      65536 RECODISK1
RECODISK2                                1            2 /dev/RECODISK2                 CACHED  UNKNOWN      65536      65536 RECODISK2
RECODISK3                                2            2 /dev/RECODISK3                 CACHED  UNKNOWN      65536      65536 RECODISK3
RECODISK4                                3            2 /dev/RECODISK4                 CACHED  UNKNOWN      65536      65536 RECODISK4

8 rows selected.

→ 再度ストレージ拡張を行うと、DATA_0000,0001,0002,0003へ戻った

SQL> select NAME,GROUP_NUMBER,STATE,TYPE,TOTAL_MB from v$asm_diskgroup;

NAME                           GROUP_NUMBER STATE       TYPE     TOTAL_MB
------------------------------ ------------ ----------- ------ ----------
DATA                                      1 MOUNTED     EXTERN     524288
RECO                                      2 MOUNTED     EXTERN     262144
SQL> select VOLUME_NAME,VOLUME_NUMBER,GROUP_NUMBER,SIZE_MB,REDUNDANCY,VOLUME_DEVICE,MOUNTPATH from v$asm_volume;

VOLUME_NAME     VOLUME_NUMBER GROUP_NUMBER    SIZE_MB REDUND VOLUME_DEVICE                  MOUNTPATH
--------------- ------------- ------------ ---------- ------ ------------------------------ ------------------------------
COMMONSTORE                 1            1       5120 UNPROT /dev/asm/commonstore-61        /opt/oracle/dcs/commonstore

-> ASM Volumeは変化無し

拡張中の動き

一時的にデバイスが4つ増え、ASMでリバランスした後、不要となったデバイスを削除する動きの模様。
(以下、再拡張時のログのため上とデバイス名等が合っていない)

  • デバイス

デバイスが4つ増え、もともとデータが入っていたディスクが削除された。

iostat 1

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.42    0.00   11.93    1.78    0.00   74.87

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              21.00         0.00       192.00          0        192
dm-0             21.00         0.00       192.00          0        192
dm-1              0.00         0.00         0.00          0          0
sdf              80.00       640.00         0.00        640          0
sdg              80.00       640.00         0.00        640          0
sdh              71.00       568.00         0.00        568          0
sdi              81.00       672.00         0.00        672          0
sdj              87.00       640.00       144.00        640        144
asm!commonstore-61     0.00         0.00         0.00          0          0
sdk              84.00       642.00         9.00        642          9
sdl              80.00       640.00         0.00        640          0
sdm              80.00       640.00         0.00        640          0
sdn              80.00       640.00         0.00        640          0
sdo              83.00       664.00         0.00        664          0
sdp              83.00       664.00         0.00        664          0
sdq              83.00       664.00         0.00        664          0
sdr              83.00       664.00         0.00        664          0
  • ASMディスク
    一時的に8つになった

asmcmd iostat --io -G DATA 2

Group_Name  Dsk_Name   Reads  Writes
DATA        DATA_0000  3.00   0.00
DATA        DATA_0001  3.00   0.00
DATA        DATA_0002  3.00   0.00
DATA        DATA_0003  3.00   0.00
DATA        DATA_0004  6.00   0.00
DATA        DATA_0005  3.00   0.00
DATA        DATA_0006  3.00   0.00
DATA        DATA_0007  3.00   4.00

Group_Name  Dsk_Name   Reads  Writes
DATA        DATA_0000  3.00   26.00
DATA        DATA_0001  2.00   0.00
DATA        DATA_0002  2.00   0.00
DATA        DATA_0003  2.00   0.00
DATA        DATA_0004  2.00   6.00
DATA        DATA_0005  2.00   0.00
DATA        DATA_0006  2.00   0.00
DATA        DATA_0007  2.00   2.00
  • リバランス処理

REBAL RUNをとらえられず…w

SQL> r
  1* select GROUP_NUMBER, OPERATION, STATE, POWER, ACTUAL,SOFAR, EST_WORK, EST_RATE, EST_MINUTES from V$ASM_OPERATION

GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- -----------
           1 REBAL WAIT          1
           1 REBAL WAIT          1
           1 REBAL WAIT          1

以上

10
7
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
10
7

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?