背景：
pod間でデータを共有したいときに、AWSではAmazon Elastic Block Store (Amazon EBS)のNFSをMountすれば良いが、GCPにはpodごとにPersistent DiskをMountすることがしかできない。 GCPのNFSとしてはCloud Filestoreあるんですが、The minimum Standard tier instance size is 1 terabyte (TB)と書いてあるので、非常に値段が高い。なので、Ceph分散ストレージをGKEにdeployするのが選択肢になった。そこでCephのdepolyは非常に面倒なので、Cloud NativeのRookを活かす。
さらに Taint/Toleration とNode selectorを用いて、指定したNode poolにCeph ClusterをDeployし、
他のPodがCeph Clusterに侵入しないようにDeployした。
手順:
もしすでに本記事の詳細部分を見ましたら、以下のコマンドで便利に組み立てる
====================================================
1.
helm install --namespace rook-ceph-system --name rook-ceph rook-stable/rook-ceph -f values.yaml
kubectl --namespace rook-ceph-system get pods -l "app=rook-ceph-operator"
kubectl get po -n rook-ceph-system
2.
kubectl apply -f cluster.yaml
###rook-ceph-mon-* のSTATUSがCrashLoopBackOffのときに VMにsshでloginし sudo rm -rf /var/lib/rookを実行してください ###
kubectl get po -n rook-ceph -o wide
kubectl logs `kubectl get pod -n rook-ceph-system | grep -F -i 'rook-ceph-operator' | awk '{print $1}'` -n rook-ceph-system
3.
kubectl apply -f filesystem.yaml
###CephFilesystemのpodを立ち上がるまでに3~5分かかる###
kubectl get po -n rook-ceph
4.
cat <<EOF > testfs.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
volumeMounts:
- name: nginx-data
mountPath: /usr/share/nginx/html
volumes:
- name: nginx-data
flexVolume:
driver: ceph.rook.io/rook
fsType: ceph
options:
fsName: myfs
clusterNamespace: rook-ceph
EOF
kubectl apply -f testfs.yaml
kubectl delete -f testfs.yaml
kubectl delete -f filesystem.yaml
kubectl delete -f cluster.yaml
kubectl -n rook-ceph patch cephclusters.ceph.rook.io rook-ceph -p '{"metadata":{"finalizers": []}}' --type=merge
kubectl delete ns rook-ceph
kubectl delete ns rook-ceph-system
helm del rook-ceph --purge
====================================================
以下は詳細
I Helmを用いてRook OperatorのDeploy
cat <<EOF > values.yaml
nodeSelector:
role: app-node
tolerations:
- key: node-type
operator: Equal
value: app-node
effect: "NoSchedule"
agent:
flexVolumeDirPath:
/home/kubernetes/flexvolume
toleration:
"NoSchedule"
discover:
toleration:
"NoSchedule"
EOF
helm install --namespace rook-ceph-system --name rook-ceph rook-stable/rook-ceph -f values.yaml
II. Node AffinityとNode Taintsを用いてCeph ClusterのDeploy
ここのspec.placement.all.nodeAffinityはosd(object storage deamon),mon,mgrがどの物理nodeに優先deploy意味であり、物理nodeのtoleration(values.yaml)と必ず整合性を取ってください
cluster.yaml
apiVersion: v1
kind: Namespace
metadata:
name: rook-ceph
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rook-ceph-osd
namespace: rook-ceph
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rook-ceph-mgr
namespace: rook-ceph
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-ceph-osd
namespace: rook-ceph
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: [ "get", "list", "watch", "create", "update", "delete" ]
---
# Aspects of ceph-mgr that require access to the system namespace
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-ceph-mgr-system
namespace: rook-ceph
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
---
# Aspects of ceph-mgr that operate within the cluster's namespace
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-ceph-mgr
namespace: rook-ceph
rules:
- apiGroups:
- ""
resources:
- pods
- services
verbs:
- get
- list
- watch
- apiGroups:
- batch
resources:
- jobs
verbs:
- get
- list
- watch
- create
- update
- delete
- apiGroups:
- ceph.rook.io
resources:
- "*"
verbs:
- "*"
---
# Allow the operator to create resources in this cluster's namespace
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-ceph-cluster-mgmt
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: rook-ceph-cluster-mgmt
subjects:
- kind: ServiceAccount
name: rook-ceph-system
namespace: rook-ceph-system
---
# Allow the osd pods in this namespace to work with configmaps
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-ceph-osd
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rook-ceph-osd
subjects:
- kind: ServiceAccount
name: rook-ceph-osd
namespace: rook-ceph
---
# Allow the ceph mgr to access the cluster-specific resources necessary for the mgr modules
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-ceph-mgr
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rook-ceph-mgr
subjects:
- kind: ServiceAccount
name: rook-ceph-mgr
namespace: rook-ceph
---
# Allow the ceph mgr to access the rook system resources necessary for the mgr modules
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-ceph-mgr-system
namespace: rook-ceph-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rook-ceph-mgr-system
subjects:
- kind: ServiceAccount
name: rook-ceph-mgr
namespace: rook-ceph
---
# Allow the ceph mgr to access cluster-wide resources necessary for the mgr modules
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-ceph-mgr-cluster
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: rook-ceph-mgr-cluster
subjects:
- kind: ServiceAccount
name: rook-ceph-mgr
namespace: rook-ceph
---
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
# The container image used to launch the Ceph daemon pods (mon, mgr, osd, mds, rgw).
# v12 is luminous, v13 is mimic, and v14 is nautilus.
# RECOMMENDATION: In production, use a specific version tag instead of the general v13 flag, which pulls the latest release and could result in different
# versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/.
image: ceph/ceph:v13.2.4-20190109
# Whether to allow unsupported versions of Ceph. Currently only luminous and mimic are supported.
# After nautilus is released, Rook will be updated to support nautilus.
# Do not set to true in production.
allowUnsupported: false
# The path on the host where configuration files will be persisted. If not specified, a kubernetes emptyDir will be created (not recommended).
# Important: if you reinstall the cluster, make sure you delete this directory from each host or else the mons will fail to start on the new cluster.
# In Minikube, the '/data' directory is configured to persist across reboots. Use "/data/rook" in Minikube environment.
dataDirHostPath: /var/lib/rook
# set the amount of mons to be started
mon:
count: 3
allowMultiplePerNode: true
# enable the ceph dashboard for viewing cluster status
dashboard:
enabled: true
# serve the dashboard under a subpath (useful when you are accessing the dashboard via a reverse proxy)
# urlPrefix: /ceph-dashboard
# serve the dashboard at the given port.
# port: 8443
# serve the dashboard using SSL
# ssl: true
network:
# toggle to use hostNetwork
hostNetwork: false
rbdMirroring:
# The number of daemons that will perform the rbd mirroring.
# rbd mirroring must be configured with "rbd mirror" from the rook toolbox.
workers: 0
# To control where various services will be scheduled by kubernetes, use the placement configuration sections below.
# The example under 'all' would have all services scheduled on kubernetes nodes labeled with 'role=storage-node' and
# tolerate taints with a key of 'storage-node'.
placement:
osd:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: role
operator: In
values:
- app-node
tolerations:
- key: node-type
operator: Equal
value: app-node
effect: "NoSchedule"
all:
tolerations:
- key: node-type
operator: Equal
value: app-node
effect: "NoSchedule"
# The above placement information can also be specified for mon, osd, and mgr components
# mon:
# osd:
# mgr:
resources:
# The requests and limits set here, allow the mgr pod to use half of one CPU core and 1 gigabyte of memory
# mgr:
# limits:
# cpu: "500m"
# memory: "1024Mi"
# requests:
# cpu: "500m"
# memory: "1024Mi"
# The above example requests/limits can also be added to the mon and osd components
# mon:
# osd:
storage: # cluster level storage configuration and selection
useAllNodes: true
useAllDevices: false
deviceFilter:
location:
config:
# The default and recommended storeType is dynamically set to bluestore for devices and filestore for directories.
# Set the storeType explicitly only if it is required not to use the default.
# storeType: bluestore
databaseSizeMB: "1024" # this value can be removed for environments with normal sized disks (100 GB or larger)
journalSizeMB: "1024" # this value can be removed for environments with normal sized disks (20 GB or larger)
osdsPerDevice: "1" # this value can be overridden at the node or device level
# Cluster level list of directories to use for storage. These values will be set for all nodes that have no `directories` set.
# directories:
# - path: /rook/storage-dir
# Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named
# nodes below will be used as storage resources. Each node's 'name' field should match their 'kubernetes.io/hostname' label.
# nodes:
# - name: "172.17.4.101"
# directories: # specific directories to use for storage can be specified for each node
# - path: "/rook/storage-dir"
# resources:
# limits:
# cpu: "500m"
# memory: "1024Mi"
# requests:
# cpu: "500m"
# memory: "1024Mi"
# - name: "172.17.4.201"
# devices: # specific devices to use for storage can be specified for each node
# - name: "sdb"
# - name: "nvme01" # multiple osds can be created on high performance devices
# config:
# osdsPerDevice: "5"
# config: # configuration can be specified at the node level which overrides the cluster level config
# storeType: filestore
# - name: "172.17.4.301"
# deviceFilter: "^sd."
kubectl apply -f cluster.yaml
ここで、もし前回ceph-clusterをdeployしたことがあった場合、次のエラー出る:
The keyring does not match the existing keyring in /var/lib/rook/mon-a/data/keyring. You may need to delete the contents of dataDirHostPath on the host from a previous deployment.
対処案:
GCPのVM INSTANCEでSSHでnodeにloginし, sudo rm -rf /var/lib/rookを実行する
III
Rook Operatorを更新したいときに
helm upgrade --namespace rook-ceph-system rook-ceph rook-stable/rook-ceph -f values.yaml
IV Node AffinityとNode Taintを用いてFilesystemをProvisioningする
filesystem.yaml
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
name: myfs
namespace: rook-ceph
spec:
# The metadata pool spec
metadataPool:
replicated:
# Increase the replication size if you have more than one osd
size: 1
# The list of data pool specs
dataPools:
- failureDomain: osd
replicated:
size: 1
# The metadata service (mds) configuration
metadataServer:
# The number of active MDS instances
activeCount: 1
# Whether each active MDS instance will have an active standby with a warm metadata cache for faster failover.
# If false, standbys will be available, but will not have a warm cache.
activeStandby: true
# The affinity rules to apply to the mds deployment
placement:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: role
# operator: In
# values:
# - storage-node
tolerations:
- key: "node-type"
operator: "Equal"
value: "app-node"
effect: "NoSchedule"
# tolerations:
# - key: mds-node
# operator: Exists
# podAffinity:
# podAntiAffinity:
resources:
# The requests and limits set here, allow the filesystem MDS Pod(s) to use half of one CPU core and 1 gigabyte of memory
# limits:
# cpu: "500m"
# memory: "1024Mi"
# requests:
# cpu: "500m"
# memory: "1024Mi"
kubectl apply -f filesystem.yaml
V.Rook CephをUninstallする
泥臭いのやり方：
0.kubectl delete -f filesystem.yaml
1.kubectl delete -f cluster.yaml
2.helm del rook-ceph --purge
3.kubectl delete daemonset --all -n rook-ceph-system
4.kubectl delete ns rook-ceph
5.kubectl delete crd cephclusters.ceph.rook.io
6.kubectl delete ns rook-ceph-system
Step 4を実行した後に、kubectl get nsしたらrook-cephずっとTerminatingのStatusになることがあるそこで kubectl -n rook-ceph patch cephclusters.ceph.rook.io rook-ceph -p '{"metadata":{"finalizers": []}}' --type=merge を実行してください
やや綺麗なやり方:
0.kubectl delete -f filesystem.yaml
1.kubectl delete -f cluster.yaml
2.kubectl delete ns rook-ceph-system
3.helm del rook-ceph --purge
非常にいい参考になるページ