OpenShift を含む Kubernetes の環境で様々なリソースを削除する場合、Deletion is in progress のようなメッセージと共に処理が滞っているように見える事があります。この原因として考えられるのは Finalizer の処理における問題であるというのが、最も一般的な見解です。
"Deletion is in progress" sample
Deletion is in progress のサンプルとして OpenShift Data Foundation (ODF) の StorageSystem CRD を削除する場合を見てみます。
$ oc delete StorageSystem my-flashsystemcluster-storagesystem -n openshift-storage
storagesystem.odf.openshift.io "my-flashsystemcluster-storagesystem" deleted
$ oc delete --grace-period=0 --force StorageSystem my-flashsystemcluster-storagesystem -n openshift-storage
storagesystem.odf.openshift.io "my-flashsystemcluster-storagesystem" deleted
上記のいずれを実行してもリソースは削除されず、詳細を確認すると .status.conditions[].message に Deletion is in progress があり、.metadata.finalizers が設定されている事が分かります。
$ oc -o yaml get storsys my-flashsystemcluster-storagesystem -n openshift-storage
apiVersion: odf.openshift.io/v1alpha1
kind: StorageSystem
metadata:
creationTimestamp: "2024-10-23T07:07:24Z"
deletionGracePeriodSeconds: 0
deletionTimestamp: "2024-10-23T07:08:38Z"
finalizers:
- storagesystem.odf.openshift.io
generation: 2
name: my-flashsystemcluster-storagesystem
namespace: openshift-storage
resourceVersion: "4304211"
uid: 149b7f69-37ea-4636-95c2-0d423c2c6795
spec:
kind: flashsystemcluster.odf.ibm.com/v1alpha1
name: my-flashsystemcluster
namespace: openshift-storage
status:
conditions:
- lastHeartbeatTime: "2024-10-24T23:44:35Z"
lastTransitionTime: "2024-10-23T07:07:24Z"
message: Deletion is in progress
reason: Deleting
status: "False"
type: Available
- lastHeartbeatTime: "2024-10-24T23:44:35Z"
lastTransitionTime: "2024-10-23T07:07:24Z"
message: Deletion is in progress
reason: Deleting
status: "True"
type: Progressing
.metadata.finalizers は Kubernetes 共通であるため、どのリソースでも同じ内容である事が確認できます。
$ oc explain StorageSystem.metadata.finalizers
GROUP: odf.openshift.io
KIND: StorageSystem
VERSION: v1alpha1
FIELD: finalizers <[]string>
DESCRIPTION:
Must be empty before the object is deleted from the registry. Each entry is
an identifier for the responsible component that will remove the entry from
the list. If the deletionTimestamp of the object is non-nil, entries in this
list can only be removed. Finalizers may be processed and removed in any
order. Order is NOT enforced because it introduces significant risk of
stuck finalizers. finalizers is a shared field, any actor with permission
can reorder it. If the finalizer list is processed in order, then this can
lead to a situation in which the component responsible for the first
finalizer in the list is waiting for a signal (field value, external system,
or other) produced by a component responsible for a finalizer later in the
list, resulting in a deadlock. Without enforced ordering finalizers are free
to order amongst themselves and are not vulnerable to ordering changes in
the list.
oc explain Pod.metadata.finalizers
KIND: Pod
VERSION: v1
FIELD: finalizers <[]string>
DESCRIPTION:
Must be empty before the object is deleted from the registry. Each entry is
an identifier for the responsible component that will remove the entry from
the list. If the deletionTimestamp of the object is non-nil, entries in this
list can only be removed. Finalizers may be processed and removed in any
order. Order is NOT enforced because it introduces significant risk of
stuck finalizers. finalizers is a shared field, any actor with permission
can reorder it. If the finalizer list is processed in order, then this can
lead to a situation in which the component responsible for the first
finalizer in the list is waiting for a signal (field value, external system,
or other) produced by a component responsible for a finalizer later in the
list, resulting in a deadlock. Without enforced ordering finalizers are free
to order amongst themselves and are not vulnerable to ordering changes in
the list.
このような状態のリソースを削除する方法の1つとして、oc edit や oc patch による .metadata.finalizers の設定解除があります。
$ oc edit StorageSystem my-flashsystemcluster-storagesystem -n openshift-storage
$ oc patch StorageSystem my-flashsystemcluster-storagesystem -n openshift-storage -p '{"metadata":{"finalizers":null}}' --type=merge
.metadata.finalizers sample
.metadata.finalizers の設定状況は環境によって異なります。ここでは2つの環境の例をご紹介します。
Cluster sample #1
KIND FINALIZER
ClusterQueue [kueue.x-k8s.io/resource-in-use]
ClusterRole [tigera.io/cni-protector]
ClusterRoleBinding [tigera.io/cni-protector]
ClusterSecret [kopf.zalando.org/KopfFinalizerMarker]
Config [imageregistry.operator.openshift.io/finalizer]
Config [samples.operator.openshift.io/finalizer]
DNS [dns.operator.openshift.io/dns-controller]
DNSRecord [operator.openshift.io/ingress-dns]
DSCInitialization [dscinitialization.opendatahub.io/finalizer]
DataScienceCluster [datasciencecluster.opendatahub.io/finalizer]
DataSciencePipelinesApplication [datasciencepipelinesapplications.opendatahub.io/finalizer]
InferenceService [inferenceservice.finalizers]
Ingress [ingresses.networking.internal.knative.dev ocp-ingress]
IngressController [ingresscontroller.operator.openshift.io/finalizer-ingresscontroller]
Installation [tigera.io/operator-cleanup]
KnativeServing [knativeservings.operator.knative.dev knative-serving-openshift]
KubeStorageVersionMigrator [cluster-kube-storage-version-migrator-operator.operator.openshift.io/KubeStorageVersionMigrator]
NodeFeatureDiscovery [foreground-deletion]
PersistentVolume [kubernetes.io/pv-protection external-attacher/vpc-block-csi-ibm-io]
PersistentVolume [kubernetes.io/pv-protection]
PersistentVolumeClaim [kubernetes.io/pvc-protection]
ResourceFlavor [kueue.x-k8s.io/resource-in-use]
Route [routes.serving.knative.dev]
Service [service.kubernetes.io/load-balancer-cleanup]
ServiceAccount [tigera.io/cni-protector]
ServiceMeshControlPlane [maistra.io/istio-operator]
ServiceMeshMember [maistra.io/istio-operator]
ServiceMeshMemberRoll [maistra.io/istio-operator]
VolumeAttachment [external-attacher/vpc-block-csi-ibm-io]
Cluster sample #2
KIND FINALIZER
BackingStore [noobaa.io/finalizer]
BucketClass [noobaa.io/finalizer]
CephBlockPool [cephblockpool.ceph.rook.io]
CephCluster [cephcluster.ceph.rook.io]
CephFilesystem [cephfilesystem.ceph.rook.io]
CephFilesystemSubVolumeGroup [cephfilesystemsubvolumegroup.ceph.rook.io]
ClusterServiceVersion [operators.coreos.com/csv-cleanup]
Config [imageregistry.operator.openshift.io/finalizer]
ConfigMap [ceph.rook.io/disaster-protection]
Config [samples.operator.openshift.io/finalizer]
CredentialsRequest [cloudcredential.openshift.io/deprovision]
CSIAddonsNode [csiaddons.openshift.io/csiaddonsnode]
DNS [dns.operator.openshift.io/dns-controller]
DNSRecord [operator.openshift.io/ingress-dns]
IngressController [ingresscontroller.operator.openshift.io/finalizer-ingresscontroller]
KubeStorageVersionMigrator [cluster-kube-storage-version-migrator-operator.operator.openshift.io/KubeStorageVersionMigrator]
NooBaa [noobaa.io/graceful_finalizer]
PersistentVolumeClaim [kubernetes.io/pvc-protection]
PersistentVolume [external-provisioner.volume.kubernetes.io/finalizer kubernetes.io/pv-protection]
PersistentVolume [kubernetes.io/pv-protection external-attacher/ebs-csi-aws-com]
Secret [ceph.rook.io/disaster-protection]
Service [service.kubernetes.io/load-balancer-cleanup]
StorageCluster [storagecluster.ocs.openshift.io]
StorageSystem [storagesystem.odf.openshift.io]
VolumeAttachment [external-attacher/ebs-csi-aws-com]