はじめに

Oracle Cloud で提供されている Kubernentes マネージドサービス OKE を使って、 Persistent Volume (PV) と Persistent Volume Claim (PVC) の動作を確認していきます。確認するのは次のポイントです。

PV と Deployment, StatefulSet の紐づけを確認
PVC を作って、Block Volume と連携されることを確認
Block Volume の Backup から PV を作成して、事前にデータを格納できるか確認

PV と Deployment, StatefulSet の紐づけ

まず結論から書きますが、Persistent Volume と Deployment, StatefulSet の紐づけは次のようになっています。Deployment は、1個の Persistent Volume を複数の Pod で共有します。StatefulSet は、1 Pod 1 Persistent Volume となります。

Deployment で複数 Pod を使用したい場合は、NFS Protocol で ReadWriteMany が利用できる、File Storage Service を使います。Block Volume の場合は、ReadWriteOnce なので、複数 Pod から利用できません。

それでは実際に動作を確認していきます。

Storage Class の確認

OKE を作成した Default の状態で、既に Storage Class が用意されています

suguru_sug@cloudshell:.kube (ap-tokyo-1)$ kubectl get storageclass
NAME            PROVISIONER      AGE
oci (default)   oracle.com/oci   19d
suguru_sug@cloudshell:.kube (ap-tokyo-1)$

YAML で確認します

suguru_sug@cloudshell:.kube (ap-tokyo-1)$ kubectl get storageclass -o yaml
apiVersion: v1
items:
- apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{"storageclass.beta.kubernetes.io/is-default-class":"true"},"name":"oci"},"provisioner":"oracle.com/oci"}
      storageclass.beta.kubernetes.io/is-default-class: "true"
    creationTimestamp: "2020-06-22T16:05:43Z"
    name: oci
    resourceVersion: "414"
    selfLink: /apis/storage.k8s.io/v1/storageclasses/oci
    uid: 02aeb840-55be-4de2-adaa-10ad3ba3b6e2
  provisioner: oracle.com/oci
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
suguru_sug@cloudshell:.kube (ap-tokyo-1)$

PVC + Deployment

Create

確認した Storage Class oci を使って、Deployment に割り当てを行います。
まず、PVCのマニフェストファイルを作ります。
ちなみに、Request を 10Gi のように少なくしても、実際に作成されるのは 50GB の Block Volume になります。

cat <<'EOF' > ~/workdir/pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: cowweb-volume
spec:
  storageClassName: "oci"
  selector:
    matchLabels:
      failure-domain.beta.kubernetes.io/zone: "AP-TOKYO-1-AD-1"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
EOF

apply

kubectl apply -f ~/workdir/pvc.yaml

確認

suguru_sug@cloudshell:workdir (ap-tokyo-1)$ kubectl get pvc -o wide
NAME            STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
cowweb-volume   Pending                                      oci            13s   Filesystem
suguru_sug@cloudshell:workdir (ap-tokyo-1)$

一定時間後、Bond されています

suguru_sug@cloudshell:workdir (ap-tokyo-1)$ kubectl get pvc -o wide
NAME            STATUS   VOLUME                                                                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
cowweb-volume   Bound    ocid1.volume.oc1.ap-tokyo-1.abxhiljr76vvzu6b4xyfut7ftxjscg4gurlfeawdyfkgtsa6dpdzzkbjr3ja   50Gi       RWO            oci            39s   Filesystem
suguru_sug@cloudshell:workdir (ap-tokyo-1)$

PV も自動生成されています
Name が、Block Volume の OCID になっています

suguru_sug@cloudshell:workdir (ap-tokyo-1)$ kubectl get pv -o wide
NAME                                                                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS   REASON   AGE   VOLUMEMODE
ocid1.volume.oc1.ap-tokyo-1.abxhiljr76vvzu6b4xyfut7ftxjscg4gurlfeawdyfkgtsa6dpdzzkbjr3ja   50Gi       RWO            Delete           Bound    default/cowweb-volume   oci                     44s   Filesystem
suguru_sug@cloudshell:workdir (ap-tokyo-1)$

YAMLで確認
ociVolumeID という指定で、Block Volume と紐づけているようです。

suguru_sug@cloudshell:workdir (ap-tokyo-1)$ kubectl get pv -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: PersistentVolume
  metadata:
    annotations:
      ociAvailabilityDomain: AP-TOKYO-1-AD-1
      ociCompartment: ocid1.compartment.oc1..aaaaaaaalxxd67fpsduvby7s3cj2ykm3bbl4myg37atkky27ipvfxv5iepda
      ociProvisionerIdentity: ociProvisionerIdentity
      ociVolumeID: ocid1.volume.oc1.ap-tokyo-1.abxhiljr76vvzu6b4xyfut7ftxjscg4gurlfeawdyfkgtsa6dpdzzkbjr3ja
      pv.kubernetes.io/provisioned-by: oracle.com/oci
    creationTimestamp: "2020-07-11T19:07:08Z"
    finalizers:
    - kubernetes.io/pv-protection
    labels:
      failure-domain.beta.kubernetes.io/region: ap-tokyo-1
      failure-domain.beta.kubernetes.io/zone: AP-TOKYO-1-AD-1
    name: ocid1.volume.oc1.ap-tokyo-1.abxhiljr76vvzu6b4xyfut7ftxjscg4gurlfeawdyfkgtsa6dpdzzkbjr3ja
    resourceVersion: "3997342"
    selfLink: /api/v1/persistentvolumes/ocid1.volume.oc1.ap-tokyo-1.abxhiljr76vvzu6b4xyfut7ftxjscg4gurlfeawdyfkgtsa6dpdzzkbjr3ja
    uid: fb926db2-8538-48ed-b05f-fe8f5dbf97c3
  spec:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 50Gi
    claimRef:
      apiVersion: v1
      kind: PersistentVolumeClaim
      name: cowweb-volume
      namespace: default
      resourceVersion: "3997295"
      uid: a4a5dfdb-8de4-4b69-85c1-4e7ad9363209
    flexVolume:
      driver: oracle/oci
      fsType: ext4
    persistentVolumeReclaimPolicy: Delete
    storageClassName: oci
    volumeMode: Filesystem
  status:
    phase: Bound
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
suguru_sug@cloudshell:workdir (ap-tokyo-1)$

OCI 上でも、同じ Compartment 上に自動的に Block Volume が作成されています

詳細画面

それでは、作成した PVC を使って、Deployment に紐づけてみます。まず、replicas は1です。

cat <<'EOF' > ~/workdir/deployment-pvc.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cowweb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cowweb
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: cowweb
    spec:
      containers:
      - name: cowweb
        image: sugimount/cowweb:v1.0
        ports:
        - name: api
          containerPort: 8080
        readinessProbe:
          httpGet:
            path: /cowsay/ping
            port: api
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /cowsay/ping
            port: api
          initialDelaySeconds: 15
          periodSeconds: 20
        imagePullPolicy: Always
        volumeMounts:
        - mountPath: /persistent-volume
          name: cowweb-storage
      volumes:
      - name: cowweb-storage
        persistentVolumeClaim:
          claimName: cowweb-volume
      securityContext:
        fsGroup: 1000
EOF

apply

kubectl apply -f ~/workdir/deployment-pvc.yaml

deployment

suguru_sug@cloudshell:~ (ap-tokyo-1)$ kubectl get deployment -o wide
NAME     READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                  SELECTOR
cowweb   1/1     1            1           35m   cowweb       sugimount/cowweb:v1.0   app=cowweb
suguru_sug@cloudshell:~ (ap-tokyo-1)$

pod

suguru_sug@cloudshell:~ (ap-tokyo-1)$ kubectl get pods -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP           NODE        NOMINATED NODE   READINESS GATES
cowweb-644d5fc5c6-qrxv4   1/1     Running   0          36m   10.244.0.5   10.0.10.8   <none>           <none>
suguru_sug@cloudshell:~ (ap-tokyo-1)$

作成した pod の中に入ります

kubectl exec -it cowweb-644d5fc5c6-qrxv4 sh

Persistent Volume が mount されています。Path はマニフェストファイルの指定通りです。

~ $ df -hT
Filesystem           Type            Size      Used Available Use% Mounted on
overlay              overlay        38.4G      3.8G     34.6G  10% /
tmpfs                tmpfs          64.0M         0     64.0M   0% /dev
tmpfs                tmpfs           7.2G         0      7.2G   0% /sys/fs/cgroup
/dev/sdb             ext4           49.1G     52.0M     46.5G   0% /persistent-volume <=================== ここにMountされている
/dev/sda3            xfs            38.4G      3.8G     34.6G  10% /dev/termination-log
/dev/sda3            xfs            38.4G      3.8G     34.6G  10% /etc/resolv.conf
/dev/sda3            xfs            38.4G      3.8G     34.6G  10% /etc/hostname
/dev/sda3            xfs            38.4G      3.8G     34.6G  10% /etc/hosts
shm                  tmpfs          64.0M         0     64.0M   0% /dev/shm
tmpfs                tmpfs           7.2G     12.0K      7.2G   0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                tmpfs           7.2G         0      7.2G   0% /proc/acpi
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/kcore
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/keys
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/timer_list
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                tmpfs           7.2G         0      7.2G   0% /proc/scsi
tmpfs                tmpfs           7.2G         0      7.2G   0% /sys/firmware

Deployment を作成した直後、Pod が稼働している Worker Node に Block Volume が Attach されているのが確認できる

Worker Node から見た時

コンテナはプロセスなので、Worker Node から iSCSI で認識している Block Volume が見えます。
scsi device の一覧を確認します。

[root@oke-crdazdbgvsw-n3geolfmm3d-s6bnuvahoda-0 ~]# lsblk --scsi
NAME HCTL       TYPE VENDOR   MODEL             REV TRAN
sdb  3:0:0:1    disk ORACLE   BlockVolume      1.0  iscsi <======= Persistent Volume
sda  1:0:0:1    disk ORACLE   BlockVolume      1.0
[root@oke-crdazdbgvsw-n3geolfmm3d-s6bnuvahoda-0 ~]#

lsblk で見ると、MOUNTPOINT が見えて、kubelet 配下に mount していることが分かります。

[root@oke-crdazdbgvsw-n3geolfmm3d-s6bnuvahoda-0 ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdb      8:16   0   50G  0 disk /var/lib/kubelet/pods/aa72eb6f-edf9-4cd4-ab4a-caf98e7e7a37/volumes/oracle~oci/ocid1.volume.oc1.ap-tokyo-1.abxhiljr76vvzu6b4xyfut7ftxjscg4gurlfeawdyfkgtsa6dpdzzkbjr3ja  <======= Persistent Volume
sda      8:0    0 46.6G  0 disk
├─sda2   8:2    0    8G  0 part
├─sda3   8:3    0 38.4G  0 part /
└─sda1   8:1    0  200M  0 part /boot/efi
[root@oke-crdazdbgvsw-n3geolfmm3d-s6bnuvahoda-0 ~]#

Path も含めて表示します

[root@oke-crdazdbgvsw-n3geolfmm3d-s6bnuvahoda-0 ~]# lsblk -P
NAME="sdb" MAJ:MIN="8:16" RM="0" SIZE="50G" RO="0" TYPE="disk" MOUNTPOINT="/var/lib/kubelet/pods/aa72eb6f-edf9-4cd4-ab4a-caf98e7e7a37/volumes/oracle~oci/ocid1.volume.oc1.ap-tokyo-1.abxhiljr76vvzu6b4xyfut7ftxjscg4gurlfeawdyfkgtsa6dpdzzkbjr3ja" <======= Persistent Volume
NAME="sda" MAJ:MIN="8:0" RM="0" SIZE="46.6G" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="sda2" MAJ:MIN="8:2" RM="0" SIZE="8G" RO="0" TYPE="part" MOUNTPOINT=""
NAME="sda3" MAJ:MIN="8:3" RM="0" SIZE="38.4G" RO="0" TYPE="part" MOUNTPOINT="/"
NAME="sda1" MAJ:MIN="8:1" RM="0" SIZE="200M" RO="0" TYPE="part" MOUNTPOINT="/boot/efi"
[root@oke-crdazdbgvsw-n3geolfmm3d-s6bnuvahoda-0 ~]#

df で確認すると、ext4 として 50GB のファイルシステムをマウントしていることが見えます

[root@oke-crdazdbgvsw-n3geolfmm3d-s6bnuvahoda-0 ~]# df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
devtmpfs       devtmpfs  7.2G     0  7.2G   0% /dev
tmpfs          tmpfs     7.3G     0  7.3G   0% /dev/shm
tmpfs          tmpfs     7.3G   33M  7.2G   1% /run
tmpfs          tmpfs     7.3G     0  7.3G   0% /sys/fs/cgroup
/dev/sda3      xfs        39G  3.8G   35G  10% /
/dev/sda1      vfat      200M  9.7M  191M   5% /boot/efi
tmpfs          tmpfs     7.3G   12K  7.3G   1% /var/lib/kubelet/pods/224cded9-1b6b-45a5-89f2-6e0c0fc642ab/volumes/kubernetes.io~secret/kube-proxy-token-f9hnj
tmpfs          tmpfs     7.3G   12K  7.3G   1% /var/lib/kubelet/pods/40a85423-7cc8-45ac-b628-a34cee14aa5b/volumes/kubernetes.io~secret/flannel-token-swf2n
tmpfs          tmpfs     7.3G   12K  7.3G   1% /var/lib/kubelet/pods/8ef6eaa9-418c-4a01-9ef2-17cd54c86ac2/volumes/kubernetes.io~secret/default-token-bj8nz
overlay        overlay    39G  3.8G   35G  10% /u01/data/docker/overlay2/6010dde33448e6cf70b136f8085c86ef35e37c01cc7122a22fec451c223219a1/merged
overlay        overlay    39G  3.8G   35G  10% /u01/data/docker/overlay2/a76f81dc09a6441e74633001abe48d86216904e7c18f1a741df145281d590560/merged
shm            tmpfs      64M     0   64M   0% /u01/data/docker/containers/121e5664df21f3b3e7dea1d7b035b062538efb2f03810a49df87b19d01e7fb79/mounts/shm
shm            tmpfs      64M     0   64M   0% /u01/data/docker/containers/c241fea2620190f34b4ee0216d8a71da96d12dabec9cf91e4fa0a864e72d3d3b/mounts/shm
overlay        overlay    39G  3.8G   35G  10% /u01/data/docker/overlay2/8ec841feaa2b22f303fb527f8f2a28a24cdb3a3aac3432c2ad483be4b7fc0f8b/merged
shm            tmpfs      64M     0   64M   0% /u01/data/docker/containers/bc3b5a5ed6d763e7cee803c14d4fba8baf5c278849bf4b5e0d072d71cb2d8d7f/mounts/shm
overlay        overlay    39G  3.8G   35G  10% /u01/data/docker/overlay2/41ecb906a4ec14fb1e4834ed4e18a672e1cf60328054264af531f061806681f7/merged
overlay        overlay    39G  3.8G   35G  10% /u01/data/docker/overlay2/0714ab0710e5c855e1c3ae54bae78160c1da5c29be821b6a6d981d18922b5a91/merged
overlay        overlay    39G  3.8G   35G  10% /u01/data/docker/overlay2/ceed125e97ff0ad5ce2b68e8650761a05cef9020495ecde0fb83edb06a3a4c92/merged
tmpfs          tmpfs     7.3G   12K  7.3G   1% /var/lib/kubelet/pods/aa72eb6f-edf9-4cd4-ab4a-caf98e7e7a37/volumes/kubernetes.io~secret/default-token-wqzmq
/dev/sdb       ext4       50G   53M   47G   1% /var/lib/kubelet/plugins/kubernetes.io/flexvolume/oracle/oci/mounts/ocid1.volume.oc1.ap-tokyo-1.abxhiljr76vvzu6b4xyfut7ftxjscg4gurlfeawdyfkgtsa6dpdzzkbjr3ja <======= Persistent Volume
overlay        overlay    39G  3.8G   35G  10% /u01/data/docker/overlay2/38c5c69d4147da1136032480b97778c44a687b58007d37e2fd21601dc8436fa3/merged
shm            tmpfs      64M     0   64M   0% /u01/data/docker/containers/c5714fdf8673d4e2775f6b76c07a034d5acd44263f74509ee38356fe26a0842a/mounts/shm
overlay        overlay    39G  3.8G   35G  10% /u01/data/docker/overlay2/c2286133eedd9516d0e68d3e4bd0c62888d82c75b4e24bd10226ebdd1577c224/merged
tmpfs          tmpfs     1.5G     0  1.5G   0% /run/user/1000
[root@oke-crdazdbgvsw-n3geolfmm3d-s6bnuvahoda-0 ~]#

docker ps で、該当のコンテナを特定します。sugimount/cowweb が該当のコンテナです。

[root@oke-crdazdbgvsw-n3geolfmm3d-s6bnuvahoda-0 etc]# docker ps
CONTAINER ID        IMAGE                                                   COMMAND                  CREATED             STATUS              PORTS               NAMES
7cf1807afdca        sugimount/cowweb                                        "java -jar /home/app…"   2 hours ago         Up 2 hours                              k8s_cowweb_cowweb-644d5fc5c6-qrxv4_default_aa72eb6f-edf9-4cd4-ab4a-caf98e7e7a37_0
c5714fdf8673        ap-tokyo-1.ocir.io/odx-oke/oke-public/pause-amd64:3.1   "/pause"                 2 hours ago         Up 2 hours                              k8s_POD_cowweb-644d5fc5c6-qrxv4_default_aa72eb6f-edf9-4cd4-ab4a-caf98e7e7a37_0
f9f87a531a9e        f0fad859c909                                            "/opt/bin/flanneld -…"   15 hours ago        Up 15 hours                             k8s_kube-flannel_kube-flannel-ds-bqsdc_kube-system_40a85423-7cc8-45ac-b628-a34cee14aa5b_3
cd90f9f0fd7f        849af609e0c6                                            "/usr/local/bin/kube…"   15 hours ago        Up 15 hours                             k8s_kube-proxy_kube-proxy-g47m7_kube-system_224cded9-1b6b-45a5-89f2-6e0c0fc642ab_1
d300cee122d0        c31b051add64                                            "/bin/proxymux.sh --…"   15 hours ago        Up 15 hours                             k8s_proxymux-client_proxymux-client-47fpz_kube-system_8ef6eaa9-418c-4a01-9ef2-17cd54c86ac2_1
bc3b5a5ed6d7        ap-tokyo-1.ocir.io/odx-oke/oke-public/pause-amd64:3.1   "/pause"                 15 hours ago        Up 15 hours                             k8s_POD_proxymux-client-47fpz_kube-system_8ef6eaa9-418c-4a01-9ef2-17cd54c86ac2_1
c241fea26201        ap-tokyo-1.ocir.io/odx-oke/oke-public/pause-amd64:3.1   "/pause"                 15 hours ago        Up 15 hours                             k8s_POD_kube-flannel-ds-bqsdc_kube-system_40a85423-7cc8-45ac-b628-a34cee14aa5b_1
121e5664df21        ap-tokyo-1.ocir.io/odx-oke/oke-public/pause-amd64:3.1   "/pause"                 15 hours ago        Up 15 hours                             k8s_POD_kube-proxy-g47m7_kube-system_224cded9-1b6b-45a5-89f2-6e0c0fc642ab_1
[root@oke-crdazdbgvsw-n3geolfmm3d-s6bnuvahoda-0 etc]#

inspect で詳細を確認します

# docker inspect 7cf1807afdca

Mounts 配下に情報が見えます。Worker Node 側で Block Volume を ext4 でマウントしている Directory をコンテナ側に見せています。

詳細
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/aa72eb6f-edf9-4cd4-ab4a-caf98e7e7a37/volumes/oracle~oci/ocid1.volume.oc1.ap-tokyo-1.abxhiljr76vvzu6b4xyfut7ftxjscg4gurlfeawdyfkgtsa6dpdzzkbjr3ja",
                "Destination": "/persistent-volume",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }

2個 Pod の動作

では、先ほどまで Deployment 配下の Pod は1個でした。2個にしてみて、動作を見てみましょう。2個目の Pod は稼働しないのが、想定している挙動です。

cat <<'EOF' > ~/workdir/deployment-pvc.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cowweb
spec:
  replicas: 2
  selector:
    matchLabels:
      app: cowweb
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: cowweb
    spec:
      containers:
      - name: cowweb
        image: sugimount/cowweb:v1.0
        ports:
        - name: api
          containerPort: 8080
        readinessProbe:
          httpGet:
            path: /cowsay/ping
            port: api
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /cowsay/ping
            port: api
          initialDelaySeconds: 15
          periodSeconds: 20
        imagePullPolicy: Always
        volumeMounts:
        - mountPath: /persistent-volume
          name: cowweb-storage
      volumes:
      - name: cowweb-storage
        persistentVolumeClaim:
          claimName: cowweb-volume
EOF

apply

kubectl apply -f ~/workdir/deployment-pvc.yaml

deployment

suguru_sug@cloudshell:~ (ap-tokyo-1)$ kubectl get deployment -o wide
NAME     READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                  SELECTOR
cowweb   1/2     2            1           72m   cowweb       sugimount/cowweb:v1.0   app=cowweb

pod
2個目の Pod は Running にはなりません

suguru_sug@cloudshell:~ (ap-tokyo-1)$ kubectl get pods -o wide
NAME                      READY   STATUS              RESTARTS   AGE   IP           NODE        NOMINATED NODE   READINESS GATES
cowweb-644d5fc5c6-dshhf   0/1     ContainerCreating   0          73s   <none>       10.0.10.7   <none>           <none>
cowweb-644d5fc5c6-qrxv4   1/1     Running             0          72m   10.244.0.5   10.0.10.8   <none>           <none>
suguru_sug@cloudshell:~ (ap-tokyo-1)$

Pod の Events を見ると、エラーが出ています

Multi-Attach error for volume "ocid1.volume.oc1.ap-tokyo-1.abxhiljr76vvzu6b4xyfut7ftxjscg4gurlfeawdyfkgtsa6dpdzzkbjr3ja" Volume is already used by pod(s) cowweb-644d5fc5c6-qrxv4 と表示されており、ReadWriteOnce のため、複数の Pod からアクセスできないことが分かります。

suguru_sug@cloudshell:~ (ap-tokyo-1)$ kubectl describe pod cowweb-644d5fc5c6-dshhf
Name:           cowweb-644d5fc5c6-dshhf
Namespace:      default
Priority:       0
Node:           10.0.10.7/10.0.10.7
Start Time:     Sun, 12 Jul 2020 08:55:39 +0000
Labels:         app=cowweb
                pod-template-hash=644d5fc5c6
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/cowweb-644d5fc5c6
Containers:
  cowweb:
    Container ID:   
    Image:          sugimount/cowweb:v1.0
    Image ID:       
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:api/cowsay/ping delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:api/cowsay/ping delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /persistent-volume from cowweb-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wqzmq (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  cowweb-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  cowweb-volume
    ReadOnly:   false
  default-token-wqzmq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wqzmq
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason              Age        From                     Message
  ----     ------              ----       ----                     -------
  Normal   Scheduled           <unknown>  default-scheduler        Successfully assigned default/cowweb-644d5fc5c6-dshhf to 10.0.10.7
  Warning  FailedAttachVolume  93s        attachdetach-controller  Multi-Attach error for volume "ocid1.volume.oc1.ap-tokyo-1.abxhiljr76vvzu6b4xyfut7ftxjscg4gurlfeawdyfkgtsa6dpdzzkbjr3ja" Volume is already used by pod(s) cowweb-644d5fc5c6-qrxv4
suguru_sug@cloudshell:~ (ap-tokyo-1)$

PVC from Backup

OKE では、Block Volume の Backup を使って PVC を作成できます。動作するアプリケーションによっては、必要なデータを外部から持ってきた方が良い場合があります。様々な方法がありますが、Block Volume Backup に必要なデータを事前に格納しておくことが、一つの方法です。(他には、initContainer で Object Storage からダウンロードすることも考えられます)

Create

既存の Persistent Volume に適当にデータを入れます。

[opc@bastion ~]$ kubectl exec -it cowweb-7f8d748898-wsh92 sh
~ $ echo "mieru?" > /persistent-volume/CouldYouSeeMe?
~ $ cat /persistent-volume/CouldYouSeeMe?
mieru?
~ $

PV の OCID を確認

[opc@bastion ~]$ kubectl get pv -o wide
NAME                                                                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS   REASON   AGE     VOLUMEMODE
ocid1.volume.oc1.ap-tokyo-1.abxhiljr3anwkx6ye6i5eeyijzqapqqdr5esnrz6zouvfphpgcaitg32pkta   50Gi       RWO            Delete           Bound    default/cowweb-volume   oci                     9m55s   Filesystem

OCID が一致している Block Volume を特定

Backup を作成

Block Volume Backup 作成

一定時間後に、Available となります
OCID をコピーしておきます

ocid1.volumebackup.oc1.ap-tokyo-1.abxhiljrokdlb3nungbzuh25bcnliwm2rwle3pui3ft27soy6tqor2xep4ha

PVC from Backup のための、マニフェストファイルを作成します。volume.beta.kubernetes.io/oci-volume-source に OCID を指定します。

cat <<'EOF' > ~/workdir/pvcfrombackup.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvcfrombackup
  annotations:
    volume.beta.kubernetes.io/oci-volume-source: ocid1.volumebackup.oc1.ap-tokyo-1.abxhiljrokdlb3nungbzuh25bcnliwm2rwle3pui3ft27soy6tqor2xep4ha
spec:
  selector:
    matchLabels:
      failure-domain.beta.kubernetes.io/zone: "AP-TOKYO-1-AD-1"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
EOF

PVC Create

kubectl apply -f ~/workdir/pvcfrombackup.yaml

Pending

[opc@bastion ~]$ kubectl get pvc -o wide
NAME            STATUS    VOLUME                                                                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
cowweb-volume   Bound     ocid1.volume.oc1.ap-tokyo-1.abxhiljr3anwkx6ye6i5eeyijzqapqqdr5esnrz6zouvfphpgcaitg32pkta   50Gi       RWO            oci            25m   Filesystem
pvcfrombackup   Pending                                                                                                                        oci            9s    Filesystem
[opc@bastion ~]$

Bound

[opc@bastion ~]$ kubectl get pvc -o wide
NAME            STATUS   VOLUME                                                                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
cowweb-volume   Bound    ocid1.volume.oc1.ap-tokyo-1.abxhiljr3anwkx6ye6i5eeyijzqapqqdr5esnrz6zouvfphpgcaitg32pkta   50Gi       RWO            oci            26m   Filesystem
pvcfrombackup   Bound    ocid1.volume.oc1.ap-tokyo-1.abxhiljrvjisjbpz2p3kihjmhyttgpffmpf4txpmuterw3lrr2ihpd5znefa   50Gi       RWO            oci            39s   Filesystem
[opc@bastion ~]$

PV

[opc@bastion ~]$ kubectl get pv -o wide
NAME                                                                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS   REASON   AGE   VOLUMEMODE
ocid1.volume.oc1.ap-tokyo-1.abxhiljr3anwkx6ye6i5eeyijzqapqqdr5esnrz6zouvfphpgcaitg32pkta   50Gi       RWO            Delete           Bound    default/cowweb-volume   oci                     25m   Filesystem
ocid1.volume.oc1.ap-tokyo-1.abxhiljrvjisjbpz2p3kihjmhyttgpffmpf4txpmuterw3lrr2ihpd5znefa   50Gi       RWO            Delete           Bound    default/pvcfrombackup   oci                     39s   Filesystem

Deployment
Replicas : 1

cat <<'EOF' > ~/workdir/deployment-pvcfrombackup.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cowweb2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cowweb2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: cowweb2
    spec:
      containers:
      - name: cowweb
        image: sugimount/cowweb:v1.0
        ports:
        - name: api
          containerPort: 8080
        readinessProbe:
          httpGet:
            path: /cowsay/ping
            port: api
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /cowsay/ping
            port: api
          initialDelaySeconds: 15
          periodSeconds: 20
        imagePullPolicy: Always
        volumeMounts:
        - mountPath: /persistent-volume
          name: cowweb-storage
      volumes:
      - name: cowweb-storage
        persistentVolumeClaim:
          claimName: pvcfrombackup
      securityContext:
        fsGroup: 1000
EOF

apply

kubectl apply -f ~/workdir/deployment-pvcfrombackup.yaml

deployment

[opc@bastion ~]$ kubectl get deployment -o wide
NAME      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                  SELECTOR
cowweb    1/1     1            1           30m   cowweb       sugimount/cowweb:v1.0   app=cowweb
cowweb2   0/1     1            0           30s   cowweb       sugimount/cowweb:v1.0   app=cowweb2

pod

$ kubectl get pods -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP           NODE        NOMINATED NODE   READINESS GATES
cowweb-7f8d748898-wsh92   1/1     Running   0          30m   10.244.0.7   10.0.10.8   <none>           <none>
cowweb2-7f74d8cf-gxwpd    1/1     Running   0          68s   10.244.0.8   10.0.10.8   <none>           <none>

pod の中にアクセスしてみて、Backup 前に作成したデータが有るか確認します。

kubectl exec -it cowweb2-7f74d8cf-gxwpd sh

無事に見えました。

/persistent-volume $ ls -la /persistent-volume/
total 24
drwxrwsr-x    3 root     app           4096 Jul 12 11:26 .
drwxr-xr-x    1 root     root            65 Jul 12 11:49 ..
-rw-rw-r--    1 app      app              7 Jul 12 11:26 CouldYouSeeMe?
drwxrws---    2 root     app          16384 Jul 12 11:20 lost+found
/persistent-volume $

無事に見えました。

/persistent-volume $ cat /persistent-volume/CouldYouSeeMe?
mieru?

PVC + StatefulSet

StatefulSet と PVC の挙動を確認します。

Create

StetefulSet の Manifest を作成します。volumeClaimTemplates がポイントです。replicas は 3 です。

cat <<'EOF' > statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cowweb-statefulset
spec:
  serviceName: cowweb
  replicas: 3
  selector:
    matchLabels:
      app: cowweb-statefulset
  template:
    metadata:
      labels:
        app: cowweb-statefulset
    spec:
      containers:
      - name: cowweb
        image: sugimount/cowweb:v1.0
        ports:
        - name: api
          containerPort: 8080
        readinessProbe:
          httpGet:
            path: /cowsay/ping
            port: api
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /cowsay/ping
            port: api
          initialDelaySeconds: 15
          periodSeconds: 20
        imagePullPolicy: Always
        volumeMounts:
        - mountPath: /persistent-volume
          name: cowweb-storage
      securityContext:
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: cowweb-storage
      annotations:
        volume.beta.kubernetes.io/storage-class: "oci"
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 50G
EOF

apply

kubectl apply -f statefulset.yaml

確認

[opc@bastion workdir]$ kubectl get sts -o wide
NAME                 READY   AGE    CONTAINERS   IMAGES
cowweb-statefulset   1/3     105s   cowweb       sugimount/cowweb:v1.0

PodManagementPolicy が Default の OrderedReady なので、1個ずつ順番に作成される

[opc@bastion workdir]$ kubectl get pod -o wide
NAME                   READY   STATUS              RESTARTS   AGE    IP           NODE        NOMINATED NODE   READINESS GATES
cowweb-statefulset-0   1/1     Running             0          3m7s   10.244.1.9   10.0.10.7   <none>           <none>
cowweb-statefulset-1   1/1     Running             0          91s    10.244.0.9   10.0.10.8   <none>           <none>
cowweb-statefulset-2   0/1     ContainerCreating   0          15s    <none>       10.0.10.7   <none>           <none>

PCVが3個自動作成される

[opc@bastion workdir]$ kubectl get pvc -o wide
NAME                                  STATUS   VOLUME                                                                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE     VOLUMEMODE
cowweb-storage-cowweb-statefulset-0   Bound    ocid1.volume.oc1.ap-tokyo-1.abxhiljrnrmimc6jjgrcjdyhxbge6cyvi2ekq3qljs36vbd3dzh2j77c4lrq   50Gi       RWO            oci            8m36s   Filesystem
cowweb-storage-cowweb-statefulset-1   Bound    ocid1.volume.oc1.ap-tokyo-1.abxhiljrrhgra64eelaepnmlhrq6tgs7ooajxq4rrtnf5ikmshraoa67eqra   50Gi       RWO            oci            7m      Filesystem
cowweb-storage-cowweb-statefulset-2   Bound    ocid1.volume.oc1.ap-tokyo-1.abxhiljrkhglvrysh4aevdcrnnajmk42ha3rjozgeo76i2bybodgajxvfesa   50Gi       RWO            oci            5m44s   Filesystem

PVが3個自動作成される

[opc@bastion workdir]$ kubectl get pv -o wide
NAME                                                                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                         STORAGECLASS   REASON   AGE     VOLUMEMODE
ocid1.volume.oc1.ap-tokyo-1.abxhiljrkhglvrysh4aevdcrnnajmk42ha3rjozgeo76i2bybodgajxvfesa   50Gi       RWO            Delete           Bound    default/cowweb-storage-cowweb-statefulset-2   oci
    5m50s   Filesystem
ocid1.volume.oc1.ap-tokyo-1.abxhiljrnrmimc6jjgrcjdyhxbge6cyvi2ekq3qljs36vbd3dzh2j77c4lrq   50Gi       RWO            Delete           Bound    default/cowweb-storage-cowweb-statefulset-0   oci
    8m42s   Filesystem
ocid1.volume.oc1.ap-tokyo-1.abxhiljrrhgra64eelaepnmlhrq6tgs7ooajxq4rrtnf5ikmshraoa67eqra   50Gi       RWO            Delete           Bound    default/cowweb-storage-cowweb-statefulset-1   oci
    6m57s   Filesystem

OCI

pods

[opc@bastion workdir]$ kubectl get pods -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
cowweb-statefulset-0   1/1     Running   0          25m   10.244.1.9    10.0.10.7   <none>           <none>
cowweb-statefulset-1   1/1     Running   0          23m   10.244.0.9    10.0.10.8   <none>           <none>
cowweb-statefulset-2   1/1     Running   0          22m   10.244.1.10   10.0.10.7   <none>           <none>

Mount している

[opc@bastion workdir]$ kubectl exec -it cowweb-statefulset-0 sh
~ $
~ $
~ $ df -hT
Filesystem           Type            Size      Used Available Use% Mounted on
overlay              overlay        38.4G      3.8G     34.6G  10% /
tmpfs                tmpfs          64.0M         0     64.0M   0% /dev
tmpfs                tmpfs           7.2G         0      7.2G   0% /sys/fs/cgroup
/dev/sdb             ext4           49.1G     52.0M     46.5G   0% /persistent-volume
/dev/sda3            xfs            38.4G      3.8G     34.6G  10% /dev/termination-log
/dev/sda3            xfs            38.4G      3.8G     34.6G  10% /etc/resolv.conf
/dev/sda3            xfs            38.4G      3.8G     34.6G  10% /etc/hostname
/dev/sda3            xfs            38.4G      3.8G     34.6G  10% /etc/hosts
shm                  tmpfs          64.0M         0     64.0M   0% /dev/shm
tmpfs                tmpfs           7.2G     12.0K      7.2G   0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                tmpfs           7.2G         0      7.2G   0% /proc/acpi
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/kcore
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/keys
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/timer_list
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                tmpfs           7.2G         0      7.2G   0% /proc/scsi
tmpfs                tmpfs           7.2G         0      7.2G   0% /sys/firmware

参考URL