はじめに
Kubernetesには、永続データを扱うために、PVやPVCといったストレージ概念が存在します。
PV, PVC を利用する際には、Backendで稼働するストレージを Plugin という形で選択します。
今回は、iscsiボリュームを利用する方法を技術調査します。
- iscsi には Dynamic Provisioner が、基本的には存在しないため、Dynamic Provisioner を使用しない連携方法を確認します
targetdを構築
Kubernetesと連携するために、iscsi のターゲットとして、targerd を構築します。
前提として、CentOSを利用します。
yum -y install targetcli
backstoreの定義
targetcli /backstores/block create name=block_backend_vdb dev=/dev/vdb
targetcli /backstores/block create name=block_backend_vdc dev=/dev/vdc
確認
[root@k8s-targetd01 dev]# targetcli ls /backstores/block
o- block ...................................................................................................... [Storage Objects: 2]
o- block_backend_vdb ................................................................. [/dev/vdb (10.0GiB) write-thru deactivated]
| o- alua ....................................................................................................... [ALUA Groups: 1]
| o- default_tg_pt_gp ........................................................................... [ALUA state: Active/optimized]
o- block_backend_vdc ................................................................. [/dev/vdc (15.0GiB) write-thru deactivated]
o- alua ....................................................................................................... [ALUA Groups: 1]
o- default_tg_pt_gp ........................................................................... [ALUA state: Active/optimized]
target の iqn を定義
targetcli /iscsi create iqn.2018-08.localdomain.hoge:k8s-targetd01
確認
[root@k8s-targetd01 dev]# targetcli ls /
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 2]
| | o- block_backend_vdb ............................................................. [/dev/vdb (10.0GiB) write-thru deactivated]
| | | o- alua ................................................................................................... [ALUA Groups: 1]
| | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| | o- block_backend_vdc ............................................................. [/dev/vdc (15.0GiB) write-thru deactivated]
| | o- alua ................................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 1]
| o- iqn.2018-08.localdomain.hoge:k8s-targetd01 ........................................................................ [TPGs: 1]
| o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
| o- acls .......................................................................................................... [ACLs: 0]
| o- luns .......................................................................................................... [LUNs: 0]
| o- portals .................................................................................................... [Portals: 1]
| o- 0.0.0.0:3260 ..................................................................................................... [OK]
o- loopback ......................................................................................................... [Targets: 0]
backstoreとiSCSI targetの紐付け設定
targetcli /iscsi/iqn.2018-08.localdomain.hoge:k8s-targetd01/tpg1/luns create /backstores/block/block_backend_vdb
targetcli /iscsi/iqn.2018-08.localdomain.hoge:k8s-targetd01/tpg1/luns create /backstores/block/block_backend_vdc
確認
lun 0, 1 が作成されています
[root@k8s-targetd01 dev]# targetcli ls /
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 2]
| | o- block_backend_vdb ............................................................... [/dev/vdb (10.0GiB) write-thru activated]
| | | o- alua ................................................................................................... [ALUA Groups: 1]
| | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| | o- block_backend_vdc ............................................................... [/dev/vdc (15.0GiB) write-thru activated]
| | o- alua ................................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 1]
| o- iqn.2018-08.localdomain.hoge:k8s-targetd01 ........................................................................ [TPGs: 1]
| o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
| o- acls .......................................................................................................... [ACLs: 0]
| o- luns .......................................................................................................... [LUNs: 2]
| | o- lun0 .......................................................... [block/block_backend_vdb (/dev/vdb) (default_tg_pt_gp)]
| | o- lun1 .......................................................... [block/block_backend_vdc (/dev/vdc) (default_tg_pt_gp)]
| o- portals .................................................................................................... [Portals: 1]
| o- 0.0.0.0:3260 ..................................................................................................... [OK]
o- loopback ......................................................................................................... [Targets: 0]
aclの設定
Kubernetesの全Nodeを設定します
- iqn.2018-08.localdomain.hoge:k8s-master01
- iqn.2018-08.localdomain.hoge:k8s-nodevm01
- iqn.2018-08.localdomain.hoge:k8s-nodegpu01
targetcli /iscsi/iqn.2018-08.localdomain.hoge:k8s-targetd01/tpg1/acls create iqn.2018-08.localdomain.hoge:k8s-master01
targetcli /iscsi/iqn.2018-08.localdomain.hoge:k8s-targetd01/tpg1/acls create iqn.2018-08.localdomain.hoge:k8s-nodevm01
targetcli /iscsi/iqn.2018-08.localdomain.hoge:k8s-targetd01/tpg1/acls create iqn.2018-08.localdomain.hoge:k8s-nodegpu01
確認します
[root@k8s-targetd01 dev]# targetcli ls /
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 2]
| | o- block_backend_vdb ............................................................... [/dev/vdb (10.0GiB) write-thru activated]
| | | o- alua ................................................................................................... [ALUA Groups: 1]
| | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| | o- block_backend_vdc ............................................................... [/dev/vdc (15.0GiB) write-thru activated]
| | o- alua ................................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 1]
| o- iqn.2018-08.localdomain.hoge:k8s-targetd01 ........................................................................ [TPGs: 1]
| o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
| o- acls .......................................................................................................... [ACLs: 3]
| | o- iqn.2018-08.localdomain.hoge:k8s-master01 ........................................................ [Mapped LUNs: 2]
| | | o- mapped_lun0 ..................................................................... [lun0 block/block_backend_vdb (rw)]
| | | o- mapped_lun1 ..................................................................... [lun1 block/block_backend_vdc (rw)]
| | o- iqn.2018-08.localdomain.hoge:k8s-nodegpu01 ....................................................... [Mapped LUNs: 2]
| | | o- mapped_lun0 ..................................................................... [lun0 block/block_backend_vdb (rw)]
| | | o- mapped_lun1 ..................................................................... [lun1 block/block_backend_vdc (rw)]
| | o- iqn.2018-08.localdomain.hoge:k8s-nodevm01 ........................................................ [Mapped LUNs: 2]
| | o- mapped_lun0 ..................................................................... [lun0 block/block_backend_vdb (rw)]
| | o- mapped_lun1 ..................................................................... [lun1 block/block_backend_vdc (rw)]
| o- luns .......................................................................................................... [LUNs: 2]
| | o- lun0 .......................................................... [block/block_backend_vdb (/dev/vdb) (default_tg_pt_gp)]
| | o- lun1 .......................................................... [block/block_backend_vdc (/dev/vdc) (default_tg_pt_gp)]
| o- portals .................................................................................................... [Portals: 1]
| o- 0.0.0.0:3260 ..................................................................................................... [OK]
o- loopback ......................................................................................................... [Targets: 0]
設定を保存します
targetcli saveconfig
serviceの起動
systemctl enable target
systemctl start target
systemctl status target
Kubernetes クラスタの iqn 設定
initiatorとして必要なものがinstallされているか確認
root@k8s-nodevm01:~# apt list | grep -i open-iscsi
open-iscsi/xenial-updates,now 2.0.873+git0.3b4b4500-14ubuntu3.4 amd64 [installed]
iqnの設定 (Master)
sed -i -e "s/InitiatorName=.*/InitiatorName=iqn.2018-08.localdomain.hoge:k8s-master01/g" /etc/iscsi/initiatorname.iscsi
iqnの設定 (k8s-nodevm01)
sed -i -e "s/InitiatorName=.*/InitiatorName=iqn.2018-08.localdomain.hoge:k8s-nodevm01/g" /etc/iscsi/initiatorname.iscsi
iqnの設定 (k8s-nodegpu01)
sed -i -e "s/InitiatorName=.*/InitiatorName=iqn.2018-08.localdomain.hoge:k8s-nodegpu01/g" /etc/iscsi/initiatorname.iscsi
serviceの再起動
systemctl restart iscsid open-iscsi
target を discoveryして、正常にtargetdが構成されているか確認します
iscsiadm -m discovery -t sendtargets -p 10.44.194.72
実行例
root@k8s-master01(default kubernetes-admin):~# iscsiadm -m discovery -t sendtargets -p 10.44.194.72
10.44.194.72:3260,1 iqn.2018-08.localdomain.hoge:k8s-targetd01
Snapshotを取得し、Loginしてみます。 (Kubernetesクラスタの各Nodeで手動実行する必要はありません。Kubelet等が自動的に行ってくれます)
root@k8s-master01(default kubernetes-admin):~# iscsiadm -m node --login
Logging in to [iface: default, target: iqn.2018-08.localdomain.hoge:k8s-targetd01, portal: 10.44.194.72,3260] (multiple)
Login to [iface: default, target: iqn.2018-08.localdomain.hoge:k8s-targetd01, portal: 10.44.194.72,3260] successful.
セッションの確認
root@k8s-master01(default kubernetes-admin):~# iscsiadm -m session -o show
tcp: [1] 10.44.194.72:3260,1 iqn.2018-08.localdomain.hoge:k8s-targetd01 (non-flash)
block storage を確認
root@k8s-master01(default kubernetes-admin):~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 10G 0 disk <---- here
sdb 8:16 0 15G 0 disk <---- here
vda 253:0 0 100G 0 disk
└─vda1 253:1 0 100G 0 part /
Kubernetes上でPersistentVolumeとして扱う
Volumeの定義
PVを定義します。
spec.capacity.storage の容量指定は、はあくまで自己申告制。実体と異なっても定義できます。
実際に使用されるのは、spec.iscsi.lunで指定しているLUNが使用されます。
cat <<'EOF' > /root/manifests/persistent_volume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: iscsi-15gb
spec:
capacity:
storage: 15Gi
volumeMode: BlockVolume
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: slow
iscsi:
targetPortal: 10.44.194.72:3260
portals: ['0.0.0.0:3260']
iqn: iqn.2018-08.localdomain.hoge:k8s-targetd01
lun: 0
fsType: xfs
readOnly: false
EOF
kubectl apply -f /root/manifests/persistent_volume.yaml
PVCの定義
requests で指定する容量は、自己申告制で定義したPVの数値を基に、要求を満たしているPVがboundされます。
cat <<'EOF' > /root/manifests/persistent_volume_claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: iscsi-15gb-claim
annotations:
volume.kubernetes.io/storage-class: "iscsi"
spec:
accessModes:
- ReadWriteOnce
volumeMode: BlockVolume
storageClassName: slow
resources:
requests:
storage: 11Gi
EOF
kubectl create -f /root/manifests/persistent_volume_claims.yaml
PVCにPVが紐づけられたことが分かります
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
iscsi-15gb 15Gi RWO Recycle Bound default/iscsi-15gb-claim slow 4m
root@k8s-master01(default kubernetes-admin):~#
root@k8s-master01(default kubernetes-admin):~# kubectl get persistentvolumeclaims -o wide
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
iscsi-15gb-claim Bound iscsi-15gb 15Gi RWO slow 40s
詳細を確認します
[Volume: iscsi-15gb]を表示されており、紐づけが確認できます
root@k8s-master01(default kubernetes-admin):~# kubectl describe persistentvolumeclaims iscsi-15gb-claim
Name: iscsi-15gb-claim
Namespace: default
StorageClass: slow
Status: Bound
Volume: iscsi-15gb
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed=yes
pv.kubernetes.io/bound-by-controller=yes
volume.kubernetes.io/storage-class=iscsi
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 15Gi
Access Modes: RWO
Events: <none>
Deploymentの作成
cat <<'EOF' > /root/manifests/persistent_deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: persistent-deployment
spec:
selector:
matchLabels:
app: persistest_test
replicas: 1
template:
metadata:
labels:
app: persistest_test
spec:
containers:
- name: master
image: nginx
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 80
volumeMounts:
- mountPath: /persistent-volume
name: pvc-volume
volumes:
- name: pvc-volume
persistentVolumeClaim:
claimName: iscsi-15gb-claim
restartPolicy: Always
EOF
kubectl apply -f /root/manifests/persistent_deployment.yaml
Podが作成されたことを確認
root@k8s-master01(default kubernetes-admin):~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-test-58586b9f9c-4qqk7 1/1 Running 2 8d 192.168.15.23 k8s-nodegpu01
nginx-test-58586b9f9c-9kf7v 1/1 Running 2 9d 192.168.15.16 k8s-nodegpu01
persistent-deployment-769b6dbb7c-bp7kv 1/1 Running 0 42s 192.168.15.28 k8s-nodegpu01 <--- here
private-reg 1/1 Running 0 4d 192.168.15.26 k8s-nodegpu01
private-reg2 1/1 Running 0 4d 192.168.15.27 k8s-nodegpu01
Node(k8s-nodegpu01)で自動的にtargetdのPortalへLoginし、Mountしていることが分かります
root@k8s-nodegpu01:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 464.7G 0 disk
└─sda1 8:1 0 464.7G 0 part /
sdb 8:16 1 7.2G 0 disk
sdf 8:80 0 10G 0 disk /var/lib/kubelet/pods/f8079bf7-9994-11e8-8a8a-525400139e17/volumes/kubernetes.io~iscsi/iscsi-15gb
sdg 8:96 0 15G 0 disk
sr0 11:0 1 4.2G 0 rom
sr1 11:1 1 1024M 0 rom
docker inspect で確認すると、dockerレイヤーでmountしていることを確認できます
snip
{
"Type": "bind",
"Source": "/var/lib/kubelet/pods/f8079bf7-9994-11e8-8a8a-525400139e17/volumes/kubernetes.io~iscsi/iscsi-15gb",
"Destination": "/persistent-volume",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
snip
また、df で確認すると、iscsiデバイスをxfsとしてMountしていることが分かります
root@k8s-nodegpu01:~# df -hT | grep iscsi-15gb
/dev/sdf xfs 10G 33M 10G 1% /var/lib/kubelet/pods/f8079bf7-9994-11e8-8a8a-525400139e17/volumes/kubernetes.io~iscsi/iscsi-15gb
データのライフサイクルについて確認
Podの障害やNodeの障害時に、正常にPersistentVolume上にデータを退避出来るか確認します。
結論から記載すると、現段階の検証内容では、PV(iscsi)を使用しているPodは、正常にセルフヒーリングされなかった。Nodeの障害時に、ハングアップしているように見えました
テストデータの作成
PodへLogin
kubectl exec -it persistent-deployment-769b6dbb7c-bp7kv bash
テストデータの作成
echo '2018-08-27 mieru?1' >> /persistent-volume/test.txt
確認
root@persistent-deployment-769b6dbb7c-bp7kv:~# cat /persistent-volume/test.txt
2018-08-27 mieru?1
Podから離脱
exit
Podの削除
Podの削除を行います
kubectl delete pod persistent-deployment-769b6dbb7c-bp7kv
同一のNodeで再作成されました
root@k8s-master01(default kubernetes-admin):~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-test-58586b9f9c-4qqk7 1/1 Running 2 8d 192.168.15.23 k8s-nodegpu01
nginx-test-58586b9f9c-9kf7v 1/1 Running 2 9d 192.168.15.16 k8s-nodegpu01
persistent-deployment-769b6dbb7c-m7p9z 1/1 Running 0 15s 192.168.15.29 k8s-nodegpu01
private-reg 1/1 Running 0 4d 192.168.15.26 k8s-nodegpu01
private-reg2 1/1 Running 0 4d 192.168.15.27 k8s-nodegpu01
PodにLoginして、persistentvolumeを確認すると、データが確認できました
root@persistent-deployment-769b6dbb7c-m7p9z:~# cat /persistent-volume/test.txt
2018-08-27 mieru?1
Nodeのシャットダウン
k8s-nodegpu01 をシャットダウンすると、PV(iscsi)も含めて他のNodeへ移動するかと思ったが、移動しませんでした。iscsiのpvがずっと使用中ということで、移動されなかった(20時間放置しても変わらず)
PVをマウントしていないPodの場合：Nodeの障害時には、NodeのStatusがNotReadyとなってから約5分後にPodの移動が発生した。パラメータで短く出来るのかも
root@k8s-master01(default kubernetes-admin):~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-test-58586b9f9c-4qqk7 1/1 Unknown 2 8d 192.168.15.23 k8s-nodegpu01
nginx-test-58586b9f9c-9kf7v 1/1 Unknown 2 9d 192.168.15.16 k8s-nodegpu01
nginx-test-58586b9f9c-wwvnb 1/1 Running 0 5m 192.168.125.79 k8s-nodevm01
nginx-test-58586b9f9c-x42qp 1/1 Running 0 5m 192.168.125.78 k8s-nodevm01
persistent-deployment-769b6dbb7c-d98dv 0/1 ContainerCreating 0 9m <none> k8s-nodevm01
persistent-deployment-769b6dbb7c-m7p9z 1/1 Unknown 0 12m 192.168.15.29 k8s-nodegpu01
private-reg 1/1 Unknown 0 4d 192.168.15.26 k8s-nodegpu01
private-reg2 1/1 Unknown 0 4d 192.168.15.27 k8s-nodegpu01
その後、nodegpu01を起動し直すと、nodevm01へ移動された (iscsiは、Node障害に対してまだ機能が不足しているのかも)
root@k8s-master01(default kubernetes-admin):~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-test-58586b9f9c-wwvnb 1/1 Running 0 7m 192.168.125.79 k8s-nodevm01
nginx-test-58586b9f9c-x42qp 1/1 Running 0 7m 192.168.125.78 k8s-nodevm01
persistent-deployment-769b6dbb7c-d98dv 1/1 Running 0 10m 192.168.125.83 k8s-nodevm01
移動先の nodevm01 でも、正常に iscsi を mount している
root@k8s-nodevm01:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 10G 0 disk /var/lib/kubelet/pods/2b056bdb-9997-11e8-8a8a-525400139e17/volumes/kubernetes.io~iscsi/iscsi-15gb
sdb 8:16 0 15G 0 disk
vda 253:0 0 100G 0 disk
└─vda1 253:1 0 100G 0 part /
root@k8s-nodevm01:~# df -hT | grep -i iscsi-15gb
/dev/sda xfs 10G 33M 10G 1% /var/lib/kubelet/pods/2b056bdb-9997-11e8-8a8a-525400139e17/volumes/kubernetes.io~iscsi/iscsi-15gb
PodにLogin
kubectl exec -it persistent-deployment-769b6dbb7c-d98dv bash
Podが再作成されても、テストデータを確認することが出来る
root@persistent-deployment-769b6dbb7c-d98dv:/# cat /persistent-volume/test.txt
2018-08-27 mieru?1
Deploymentを削除して、再作成してみる
kubectl delete deployments persistent-deployment
kubectl apply -f /root/manifests/persistent_deployment.yaml
PodにLogin
kubectl exec -it persistent-deployment-769b6dbb7c-hvqkg bash
Deploymentを再作成しても、Persistent Volume 上のデータは正しく引き継がれる
root@persistent-deployment-769b6dbb7c-hvqkg:/# cat /persistent-volume/test.txt
2018-08-27 mieru?1
まとめ
iscsiを使用して、PV, PVCを利用することが出来たが、Node障害時の挙動が怪しい部分がありました。
商用利用の際には、iscsi利用は厳しいと思われるので、他のストレージを検討した方が良いと思います。
参考URL
kubernetes docs
https://kubernetes.io/docs/concepts/storage/storage-classes/
Targetd の構築
https://qiita.com/sugimount/items/60a7ac50247f6bf7130e
iscsi initiatorの構築
https://www.server-world.info/query?os=Ubuntu_16.04&p=iscsi&f=3
kubernetes iscsi
https://github.com/kubernetes/examples/blob/master/staging/volumes/iscsi/iscsi.yaml