仮想環境でiSCSI target/initiator、multipath、LVMを試してみた。

環境

CentOS 7.6 (VirtualBox, vagrant box を利用)

Hostname	IP address	Description
svr00	192.168.33.100, 192.168.33.110	iSCSI ターゲット
svr01	192.168.33.101	iSCSI イニシエータ

svr00: iSCSIターゲットの構成

targetcli の導入と target.service の起動

# yum -y install targetcli
# systemctl enable target.service
# systemctl start target.service
# systemctl status target.service
● target.service - Restore LIO kernel target configuration
   Loaded: loaded (/usr/lib/systemd/system/target.service; enabled; vendor preset: disabled)
   Active: active (exited) since Tue 2019-02-05 12:22:22 UTC; 8min ago
  Process: 24553 ExecStart=/usr/bin/targetctl restore (code=exited, status=0/SUCCESS)
 Main PID: 24553 (code=exited, status=0/SUCCESS)

iSCSIターゲット初期状態

# targetcli ls /
o- / ............................................................................. [...]
  o- backstores .................................................................. [...]
  | o- block ...................................................... [Storage Objects: 0]
  | o- fileio ..................................................... [Storage Objects: 0]
  | o- pscsi ...................................................... [Storage Objects: 0]
  | o- ramdisk .................................................... [Storage Objects: 0]
  o- iscsi ................................................................ [Targets: 0]
  o- loopback ............................................................. [Targets: 0]

backstoreを作成（面倒だったのでFILEIOで作成、ディスク付ける場合はBLOCKIOで作成する）

# mkdir /var/iscsi
# targetcli /backstores/fileio create file00 /var/iscsi/disk00.img 10MB write_back=false
# targetcli ls /
o- / ............................................................................. [...]
  o- backstores .................................................................. [...]
  | o- block ...................................................... [Storage Objects: 0]
  | o- fileio ..................................................... [Storage Objects: 1]
  | | o- file00 ............... [/var/iscsi/disk00.img (10.0MiB) write-thru deactivated]
  | |   o- alua ....................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ........................... [ALUA state: Active/optimized]
  | o- pscsi ...................................................... [Storage Objects: 0]
  | o- ramdisk .................................................... [Storage Objects: 0]
  o- iscsi ................................................................ [Targets: 0

IQNの定義（命名規則はRFC 3721を参照）

# targetcli /iscsi create iqn.2000-08.jp.co.example:svr00
# targetcli ls /
o- / ............................................................................. [...]
  o- backstores .................................................................. [...]
  | o- block ...................................................... [Storage Objects: 0]
  | o- fileio ..................................................... [Storage Objects: 1]
  | | o- file00 ............... [/var/iscsi/disk00.img (10.0MiB) write-thru deactivated]
  | |   o- alua ....................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ........................... [ALUA state: Active/optimized]
  | o- pscsi ...................................................... [Storage Objects: 0]
  | o- ramdisk .................................................... [Storage Objects: 0]
  o- iscsi ................................................................ [Targets: 1]
  | o- iqn.2000-08.jp.co.example:svr00 ....................................... [TPGs: 1]
  |   o- tpg1 ................................................... [no-gen-acls, no-auth]
  |     o- acls .............................................................. [ACLs: 0]
  |     o- luns .............................................................. [LUNs: 0]
  |     o- portals ........................................................ [Portals: 1]
  |       o- 0.0.0.0:3260 ......................................................... [OK]
  o- loopback ............................................................. [Targets: 0]

IQN定義時にTPG（Target Portal Group)が1つ作成される。
Portalは「0.0.0.0:3260」が定義されているのでこのまま利用。
LUNは先に作成したbackstore(file00)を紐付ける。

# targetcli /iscsi/iqn.2000-08.jp.co.example:svr00/tpg1/luns create /backstores/fileio/file00
# targetcli ls /
o- / ............................................................................. [...]
  o- backstores .................................................................. [...]
  | o- block ...................................................... [Storage Objects: 0]
  | o- fileio ..................................................... [Storage Objects: 1]
  | | o- file00 ................. [/var/iscsi/disk00.img (10.0MiB) write-thru activated]
  | |   o- alua ....................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ........................... [ALUA state: Active/optimized]
  | o- pscsi ...................................................... [Storage Objects: 0]
  | o- ramdisk .................................................... [Storage Objects: 0]
  o- iscsi ................................................................ [Targets: 1]
  | o- iqn.2000-08.jp.co.example:svr00 ....................................... [TPGs: 1]
  |   o- tpg1 ................................................... [no-gen-acls, no-auth]
  |     o- acls .............................................................. [ACLs: 0]
  |     o- luns .............................................................. [LUNs: 1]
  |     | o- lun0 ........... [fileio/file00 (/var/iscsi/disk00.img) (default_tg_pt_gp)]
  |     o- portals ........................................................ [Portals: 1]
  |       o- 0.0.0.0:3260 ......................................................... [OK]
  o- loopback ............................................................. [Targets: 0]

ACLを定義（svr01のiSCSIイニシエータ名はiqn.2000-08.jp.co.example:svr01とする）

# targetcli /iscsi/iqn.2000-08.jp.co.example:svr00/tpg1/acls create iqn.2000-08.jp.co.example:svr01
# targetcli ls /
o- / ............................................................................. [...]
  o- backstores .................................................................. [...]
  | o- block ...................................................... [Storage Objects: 0]
  | o- fileio ..................................................... [Storage Objects: 1]
  | | o- file00 ................. [/var/iscsi/disk00.img (10.0MiB) write-thru activated]
  | |   o- alua ....................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ........................... [ALUA state: Active/optimized]
  | o- pscsi ...................................................... [Storage Objects: 0]
  | o- ramdisk .................................................... [Storage Objects: 0]
  o- iscsi ................................................................ [Targets: 1]
  | o- iqn.2000-08.jp.co.example:svr00 ....................................... [TPGs: 1]
  |   o- tpg1 ................................................... [no-gen-acls, no-auth]
  |     o- acls .............................................................. [ACLs: 1]
  |     | o- iqn.2000-08.jp.co.example:svr01 .......................... [Mapped LUNs: 1]
  |     |   o- mapped_lun0 ................................... [lun0 fileio/file00 (rw)]
  |     o- luns .............................................................. [LUNs: 1]
  |     | o- lun0 ........... [fileio/file00 (/var/iscsi/disk00.img) (default_tg_pt_gp)]
  |     o- portals ........................................................ [Portals: 1]
  |       o- 0.0.0.0:3260 ......................................................... [OK]
  o- loopback ............................................................. [Targets: 0]

設定を保存

# targetcli saveconfig
Configuration saved to /etc/target/saveconfig.json

svr01: iSCSイニシエータの構成

iscsi-initiator-utils のインストール、イニシエータ名の設定

# yum -y install iscsi-initiator-utils
# echo "InitiatorName=iqn.2000-08.jp.co.example:svr01" > /etc/iscsi/initiatorname.iscsi

iSCSIデーモン(iscsid.service)の起動

# systemctl enable iscsid.service
# systemctl start iscsid.service
# systemctl status iscsid.service
● iscsid.service - Open-iSCSI
   Loaded: loaded (/usr/lib/systemd/system/iscsid.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-02-05 13:20:14 UTC; 3s ago
     Docs: man:iscsid(8)
           man:iscsiadm(8)
  Process: 24840 ExecStart=/usr/sbin/iscsid (code=exited, status=0/SUCCESS)
 Main PID: 24842 (iscsid)
   CGroup: /system.slice/iscsid.service
           ├─24841 /usr/sbin/iscsid
           └─24842 /usr/sbin/iscsid

登録済みiSCSIターゲットの自動認識(iscsi.service)の有効化確認 ※ServiceTypeはoneshotなのでenableのみ

# systemctl status iscsi.service
● iscsi.service - Login and scanning of iSCSI devices
   Loaded: loaded (/usr/lib/systemd/system/iscsi.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:iscsid(8)
           man:iscsiadm(8)

block deviceの作業前確認

# lsblk -S
NAME HCTL       TYPE VENDOR   MODEL             REV TRAN
sda  0:0:0:0    disk ATA      VBOX HARDDISK    1.0  ata

iSCSIターゲットで指定したIQNを指定してログイン

# iscsiadm -m discovery -t st -p 192.168.33.100:3260
# iscsiadm -m discovery -t st -p 192.168.33.110:3260

# iscsiadm -m node -T iqn.2000-08.jp.co.example:svr00 -p 192.168.33.100:3260 --login
# iscsiadm -m node -T iqn.2000-08.jp.co.example:svr00 -p 192.168.33.110:3260 --login

# iscsiadm -m node
192.168.33.100:3260,1 iqn.2000-08.jp.co.example:svr00
192.168.33.110:3260,1 iqn.2000-08.jp.co.example:svr00

block deviceが増えたのを確認

# lsblk -S
NAME HCTL       TYPE VENDOR   MODEL             REV TRAN
sda  0:0:0:0    disk ATA      VBOX HARDDISK    1.0  ata
sdb  2:0:0:0    disk LIO-ORG  file00           4.0  iscsi
sdc  3:0:0:0    disk LIO-ORG  file00           4.0  iscsi

WWIDは揃っているので同じデバイスを見ている

# /usr/lib/udev/scsi_id --whitelisted --device=/dev/sdb
360014053ca1100494274260830c9109f
# /usr/lib/udev/scsi_id --whitelisted --device=/dev/sdc
360014053ca1100494274260830c9109f

svr01: multipathを組む

DM-Multipath をインストール

# yum -y install device-mapper-multipath

multipath.confを作成（mpathXという名前で自動作成する設定）

/etc/multipath.conf

defaults {
    user_friendly_names yes
    find_multipaths yes
}
blacklist {
}

dry-runして構成を確認

# multipath -v2 -d
create: mpatha (360014053ca1100494274260830c9109f) undef LIO-ORG ,file00
size=10M features='0' hwhandler='0' wp=undef
|-+- policy='service-time 0' prio=1 status=undef
| `- 2:0:0:0 sdb 8:16 undef ready running
`-+- policy='service-time 0' prio=1 status=undef
  `- 3:0:0:0 sdc 8:32 undef ready running

multipathd.serviceを起動してmultipath deviceが作成されたのを確認

# systemctl enable multipathd.service
# systemctl start multipathd.service
# systemctl status multipathd.service
● multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-02-05 13:52:17 UTC; 17s ago
  Process: 25312 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS)
  Process: 25310 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS)
  Process: 25308 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS)
 Main PID: 25315 (multipathd)
   CGroup: /system.slice/multipathd.service
           └─25315 /sbin/multipathd

# multipath -ll
mpatha (360014053ca1100494274260830c9109f) dm-0 LIO-ORG ,file00
size=10M features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=enabled
| `- 2:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 3:0:0:0 sdc 8:32 active ready running
Feb 05 13:52:20 | multipath device maps are present, but 'multipathd' service is not running
Feb 05 13:52:20 | IO failover/failback will not work without 'multipathd' service running

# ls -l /dev/mapper/mpatha
lrwxrwxrwx. 1 root root 7 Feb  5 13:52 /dev/mapper/mpatha -> ../dm-0

svr01: LVMを作成する

事前準備

折角なので multipath device を追加しておく。

svr00: backstore 作成
svr00: lun に追加
svr01: iSCSI デバイスの再スキャン iscsiadm -m session --rescan
svr01: (multipathd 起動していない場合) multipath device を作成

とりあえず今回は３つ用意（10MB x 2だとPE size変更してもファイルシステム作成時にblock数が足りなかったので後からmpathc足したのは内緒）

# multipath -ll
mpathc (360014058d5fa0d9125b45ec9aa4491d4) dm-2 LIO-ORG ,file02
size=100M features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 3:0:0:2 sdg 8:96 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 2:0:0:2 sdf 8:80 active ready running
mpathb (360014056b1f65811723430ebc736012b) dm-1 LIO-ORG ,file01
size=10M features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 2:0:0:1 sde 8:64 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 3:0:0:1 sdd 8:48 active ready running
mpatha (360014053ca1100494274260830c9109f) dm-0 LIO-ORG ,file00
size=10M features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 2:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 3:0:0:0 sdc 8:32 active ready running

PV,VG,LVの作成

PVの作成

# pvcreate /dev/mapper/mpatha
  Physical volume "/dev/mapper/mpatha" successfully created.

# pvcreate /dev/mapper/mpathb
  Physical volume "/dev/mapper/mpathb" successfully created.

# pvcreate /dev/mapper/mpathc
  Physical volume "/dev/mapper/mpathc" successfully created.

# pvs
  PV                 VG Fmt  Attr PSize  PFree
  /dev/mapper/mpatha    lvm2 ---  10.00m 10.00m
  /dev/mapper/mpathb    lvm2 ---  10.00m 10.00m
  /dev/mapper/mpathc    lvm2 ---  100.00m 100.00m

VGの作成（PE Size　を1MBに変更）

# vgcreate -s 1M VG0 /dev/mapper/mpatha /dev/mapper/mpathb
  Volume group "VG0" successfully created

# vgextend VG0 /dev/mapper/mpathc
  Volume group "VG0" successfully extended

# vgs
  VG  #PV #LV #SN Attr   VSize   VFree
  VG0   3   0   0 wz--n- 108.00m 108.00m

LVの作成

# lvcreate -l 100%FREE -n lv00 VG0
  Logical volume "lv00" created.

# lvs
  LV   VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv00 VG0 -wi-ao---- 108.00m

ファイルシステムの作成

ファイルシステム作成

# mkfs.xfs /dev/VG0/lv00
meta-data=/dev/VG0/lv00          isize=512    agcount=4, agsize=6912 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=27648, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=855, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

UUIDを確認、fstabに追記してmount

# blkid /dev/VG0/lv00
/dev/VG0/lv00: UUID="9351d7da-a701-4011-9553-15feb27ecc5c" TYPE="xfs"

# echo "UUID=9351d7da-a701-4011-9553-15feb27ecc5c /work xfs defaults 0 0" >> /etc/fstab
# mkdir /work
# mount /work
# df -h | grep /work
/dev/mapper/VG0-lv00  105M  5.7M  100M   6% /work

おしまい。