Help us understand the problem. What is going on with this article?

(GKE)KubernetesにおけるProduction環境構築 (III)分散ストレージ Rook Cephの構築

背景: 
pod間でデータを共有したいときに、AWSではAmazon Elastic Block Store (Amazon EBS)のNFSをMountすれば良いが、GCPにはpodごとにPersistent DiskをMountすることがしかできない。 GCPのNFSとしてはCloud Filestoreあるんですが、The minimum Standard tier instance size is 1 terabyte (TB)と書いてあるので、非常に値段が高い。なので、Ceph分散ストレージをGKEにdeployするのが選択肢になった。そこでCephのdepolyは非常に面倒なので、Cloud NativeのRookを活かす。
さらに Taint/Toleration とNode selectorを用いて、指定したNode poolにCeph ClusterをDeployし、
他のPodがCeph Clusterに侵入しないようにDeployした。

手順:

もしすでに本記事の詳細部分を見ましたら、以下のコマンドで便利に組み立てる

====================================================

1.
helm install --namespace rook-ceph-system --name rook-ceph rook-stable/rook-ceph -f values.yaml
kubectl --namespace rook-ceph-system get pods -l "app=rook-ceph-operator"
kubectl get po -n rook-ceph-system
2.
kubectl apply -f cluster.yaml
###rook-ceph-mon-* のSTATUSがCrashLoopBackOffのときに VMにsshでloginし sudo rm -rf /var/lib/rookを実行してください ###
kubectl get po -n rook-ceph -o wide
kubectl logs `kubectl get pod -n rook-ceph-system | grep -F -i 'rook-ceph-operator' | awk '{print $1}'` -n rook-ceph-system
3.
kubectl apply -f filesystem.yaml
###CephFilesystemのpodを立ち上がるまでに3~5分かかる###
kubectl get po -n rook-ceph

4.

cat <<EOF > testfs.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-data
          mountPath: /usr/share/nginx/html
      volumes:
      - name: nginx-data
        flexVolume:
          driver: ceph.rook.io/rook
          fsType: ceph
          options:
            fsName: myfs
            clusterNamespace: rook-ceph
EOF
kubectl apply -f testfs.yaml

kubectl delete -f testfs.yaml
kubectl delete -f filesystem.yaml
kubectl delete -f cluster.yaml
kubectl -n rook-ceph patch cephclusters.ceph.rook.io rook-ceph -p '{"metadata":{"finalizers": []}}' --type=merge
kubectl delete ns rook-ceph
kubectl delete ns rook-ceph-system
helm del rook-ceph --purge

====================================================

以下は詳細

I Helmを用いてRook OperatorのDeploy

cat <<EOF > values.yaml
nodeSelector:
  role: app-node
tolerations:
- key: node-type
  operator: Equal
  value: app-node
  effect: "NoSchedule"


agent:
  flexVolumeDirPath:
    /home/kubernetes/flexvolume
  toleration:
    "NoSchedule"
discover:
  toleration:
    "NoSchedule"
EOF

helm install --namespace rook-ceph-system --name rook-ceph rook-stable/rook-ceph -f values.yaml

II. Node AffinityとNode Taintsを用いてCeph ClusterのDeploy
ここのspec.placement.all.nodeAffinityはosd(object storage deamon),mon,mgrがどの物理nodeに優先deploy意味であり、物理nodeのtoleration(values.yaml)と必ず整合性を取ってください
cluster.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: rook-ceph
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-ceph-osd
  namespace: rook-ceph
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-ceph-mgr
  namespace: rook-ceph
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rook-ceph-osd
  namespace: rook-ceph
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: [ "get", "list", "watch", "create", "update", "delete" ]
---
# Aspects of ceph-mgr that require access to the system namespace
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rook-ceph-mgr-system
  namespace: rook-ceph
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - list
  - watch
---
# Aspects of ceph-mgr that operate within the cluster's namespace
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rook-ceph-mgr
  namespace: rook-ceph
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - delete
- apiGroups:
  - ceph.rook.io
  resources:
  - "*"
  verbs:
  - "*"
---
# Allow the operator to create resources in this cluster's namespace
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rook-ceph-cluster-mgmt
  namespace: rook-ceph
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rook-ceph-cluster-mgmt
subjects:
- kind: ServiceAccount
  name: rook-ceph-system
  namespace: rook-ceph-system
---
# Allow the osd pods in this namespace to work with configmaps
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rook-ceph-osd
  namespace: rook-ceph
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rook-ceph-osd
subjects:
- kind: ServiceAccount
  name: rook-ceph-osd
  namespace: rook-ceph
---
# Allow the ceph mgr to access the cluster-specific resources necessary for the mgr modules
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rook-ceph-mgr
  namespace: rook-ceph
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rook-ceph-mgr
subjects:
- kind: ServiceAccount
  name: rook-ceph-mgr
  namespace: rook-ceph
---
# Allow the ceph mgr to access the rook system resources necessary for the mgr modules
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rook-ceph-mgr-system
  namespace: rook-ceph-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rook-ceph-mgr-system
subjects:
- kind: ServiceAccount
  name: rook-ceph-mgr
  namespace: rook-ceph
---
# Allow the ceph mgr to access cluster-wide resources necessary for the mgr modules
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rook-ceph-mgr-cluster
  namespace: rook-ceph
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rook-ceph-mgr-cluster
subjects:
- kind: ServiceAccount
  name: rook-ceph-mgr
  namespace: rook-ceph
---
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    # The container image used to launch the Ceph daemon pods (mon, mgr, osd, mds, rgw).
    # v12 is luminous, v13 is mimic, and v14 is nautilus.
    # RECOMMENDATION: In production, use a specific version tag instead of the general v13 flag, which pulls the latest release and could result in different
    # versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/.
    image: ceph/ceph:v13.2.4-20190109
    # Whether to allow unsupported versions of Ceph. Currently only luminous and mimic are supported.
    # After nautilus is released, Rook will be updated to support nautilus.
    # Do not set to true in production.
    allowUnsupported: false
  # The path on the host where configuration files will be persisted. If not specified, a kubernetes emptyDir will be created (not recommended).
  # Important: if you reinstall the cluster, make sure you delete this directory from each host or else the mons will fail to start on the new cluster.
  # In Minikube, the '/data' directory is configured to persist across reboots. Use "/data/rook" in Minikube environment.
  dataDirHostPath: /var/lib/rook
  # set the amount of mons to be started
  mon:
    count: 3
    allowMultiplePerNode: true
  # enable the ceph dashboard for viewing cluster status
  dashboard:
    enabled: true
    # serve the dashboard under a subpath (useful when you are accessing the dashboard via a reverse proxy)
    # urlPrefix: /ceph-dashboard
    # serve the dashboard at the given port.
    # port: 8443
    # serve the dashboard using SSL
    # ssl: true
  network:
    # toggle to use hostNetwork
    hostNetwork: false
  rbdMirroring:
    # The number of daemons that will perform the rbd mirroring.
    # rbd mirroring must be configured with "rbd mirror" from the rook toolbox.
    workers: 0
  # To control where various services will be scheduled by kubernetes, use the placement configuration sections below.
  # The example under 'all' would have all services scheduled on kubernetes nodes labeled with 'role=storage-node' and
  # tolerate taints with a key of 'storage-node'.
  placement:
    osd:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: role
              operator: In
              values:
              - app-node
      tolerations:
      - key: node-type
        operator: Equal
        value: app-node
        effect: "NoSchedule"
    all:
      tolerations:
      - key: node-type
        operator: Equal
        value: app-node
        effect: "NoSchedule"
# The above placement information can also be specified for mon, osd, and mgr components
#    mon:
#    osd:
#    mgr:
  resources:
# The requests and limits set here, allow the mgr pod to use half of one CPU core and 1 gigabyte of memory
#    mgr:
#      limits:
#        cpu: "500m"
#        memory: "1024Mi"
#      requests:
#        cpu: "500m"
#        memory: "1024Mi"
# The above example requests/limits can also be added to the mon and osd components
#    mon:
#    osd:
  storage: # cluster level storage configuration and selection
    useAllNodes: true
    useAllDevices: false
    deviceFilter:
    location:
    config:
      # The default and recommended storeType is dynamically set to bluestore for devices and filestore for directories.
      # Set the storeType explicitly only if it is required not to use the default.
      # storeType: bluestore
      databaseSizeMB: "1024" # this value can be removed for environments with normal sized disks (100 GB or larger)
      journalSizeMB: "1024"  # this value can be removed for environments with normal sized disks (20 GB or larger)
      osdsPerDevice: "1" # this value can be overridden at the node or device level
# Cluster level list of directories to use for storage. These values will be set for all nodes that have no `directories` set.
#    directories:
#    - path: /rook/storage-dir
# Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named
# nodes below will be used as storage resources.  Each node's 'name' field should match their 'kubernetes.io/hostname' label.
#    nodes:
#    - name: "172.17.4.101"
#      directories: # specific directories to use for storage can be specified for each node
#      - path: "/rook/storage-dir"
#      resources:
#        limits:
#          cpu: "500m"
#          memory: "1024Mi"
#        requests:
#          cpu: "500m"
#          memory: "1024Mi"
#    - name: "172.17.4.201"
#      devices: # specific devices to use for storage can be specified for each node
#      - name: "sdb"
#      - name: "nvme01" # multiple osds can be created on high performance devices
#        config:
#          osdsPerDevice: "5"
#      config: # configuration can be specified at the node level which overrides the cluster level config
#        storeType: filestore
#    - name: "172.17.4.301"
#      deviceFilter: "^sd."
kubectl apply -f cluster.yaml

ここで、もし前回ceph-clusterをdeployしたことがあった場合、次のエラー出る:
The keyring does not match the existing keyring in /var/lib/rook/mon-a/data/keyring. You may need to delete the contents of dataDirHostPath on the host from a previous deployment.
対処案:
GCPのVM INSTANCEでSSHでnodeにloginし, sudo rm -rf /var/lib/rookを実行する

III
Rook Operatorを更新したいときに

helm upgrade --namespace rook-ceph-system rook-ceph rook-stable/rook-ceph -f values.yaml

IV Node AffinityとNode Taintを用いてFilesystemをProvisioningする
filesystem.yaml

apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: myfs
  namespace: rook-ceph
spec:   

  # The metadata pool spec
  metadataPool:
    replicated:
      # Increase the replication size if you have more than one osd
      size: 1
  # The list of data pool specs
  dataPools:
    - failureDomain: osd
      replicated:
        size: 1
  # The metadata service (mds) configuration
  metadataServer:
    # The number of active MDS instances
    activeCount: 1
    # Whether each active MDS instance will have an active standby with a warm metadata cache for faster failover.
    # If false, standbys will be available, but will not have a warm cache.
    activeStandby: true
    # The affinity rules to apply to the mds deployment

    placement:
      # nodeAffinity:
      #   requiredDuringSchedulingIgnoredDuringExecution:
      #     nodeSelectorTerms:
      #     - matchExpressions:
      #       - key: role
      #         operator: In
      #         values:
      #         - storage-node
      tolerations:
      - key: "node-type"
        operator: "Equal"
        value: "app-node"
        effect: "NoSchedule"
    #  tolerations:
    #  - key: mds-node
    #    operator: Exists
    #  podAffinity:
    #  podAntiAffinity:
    resources:
    # The requests and limits set here, allow the filesystem MDS Pod(s) to use half of one CPU core and 1 gigabyte of memory
    #  limits:
    #    cpu: "500m"
    #    memory: "1024Mi"
    #  requests:
    #    cpu: "500m"
    #    memory: "1024Mi"
 kubectl apply -f filesystem.yaml

V.Rook CephをUninstallする
泥臭いのやり方:
0.kubectl delete -f filesystem.yaml
1.kubectl delete -f cluster.yaml
2.helm del rook-ceph --purge
3.kubectl delete daemonset --all -n rook-ceph-system
4.kubectl delete ns rook-ceph
5.kubectl delete crd cephclusters.ceph.rook.io
6.kubectl delete ns rook-ceph-system
Step 4を実行した後に、kubectl get nsしたらrook-cephずっとTerminatingのStatusになることがあるそこで kubectl -n rook-ceph patch cephclusters.ceph.rook.io rook-ceph -p '{"metadata":{"finalizers": []}}' --type=merge を実行してください

やや綺麗なやり方:
0.kubectl delete -f filesystem.yaml
1.kubectl delete -f cluster.yaml
2.kubectl delete ns rook-ceph-system
3.helm del rook-ceph --purge

非常にいい参考になるページ

Rook
Rook本家のQuick Start

optimind
配送ルート最適化クラウドサービス「Loogia」を提供する、物流×AIのスタートアップです
https://www.optimind.tech/
Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
Comments
No comments
Sign up for free and join this conversation.
If you already have a Qiita account
Why do not you register as a user and use Qiita more conveniently?
You need to log in to use this function. Qiita can be used more conveniently after logging in.
You seem to be reading articles frequently this month. Qiita can be used more conveniently after logging in.
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
ユーザーは見つかりませんでした