OpenShift を含む Kubernetes の環境で様々なリソースを削除する場合、Deletion is in progress のようなメッセージと共に処理が滞っているように見える事があります。この原因として考えられるのは Finalizer の処理における問題であるというのが、最も一般的な見解です。

"Deletion is in progress" sample

Deletion is in progress のサンプルとして OpenShift Data Foundation (ODF) の StorageSystem CRD を削除する場合を見てみます。

$ oc delete StorageSystem my-flashsystemcluster-storagesystem -n openshift-storage
storagesystem.odf.openshift.io "my-flashsystemcluster-storagesystem" deleted

$ oc delete --grace-period=0 --force StorageSystem my-flashsystemcluster-storagesystem -n openshift-storage
storagesystem.odf.openshift.io "my-flashsystemcluster-storagesystem" deleted

上記のいずれを実行してもリソースは削除されず、詳細を確認すると .status.conditions[].message に Deletion is in progress があり、.metadata.finalizers が設定されている事が分かります。

$ oc -o yaml get storsys my-flashsystemcluster-storagesystem -n openshift-storage
apiVersion: odf.openshift.io/v1alpha1
kind: StorageSystem
metadata:
  creationTimestamp: "2024-10-23T07:07:24Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2024-10-23T07:08:38Z"
  finalizers:
  - storagesystem.odf.openshift.io
  generation: 2
  name: my-flashsystemcluster-storagesystem
  namespace: openshift-storage
  resourceVersion: "4304211"
  uid: 149b7f69-37ea-4636-95c2-0d423c2c6795
spec:
  kind: flashsystemcluster.odf.ibm.com/v1alpha1
  name: my-flashsystemcluster
  namespace: openshift-storage
status:
  conditions:
  - lastHeartbeatTime: "2024-10-24T23:44:35Z"
    lastTransitionTime: "2024-10-23T07:07:24Z"
    message: Deletion is in progress
    reason: Deleting
    status: "False"
    type: Available
  - lastHeartbeatTime: "2024-10-24T23:44:35Z"
    lastTransitionTime: "2024-10-23T07:07:24Z"
    message: Deletion is in progress
    reason: Deleting
    status: "True"
    type: Progressing

.metadata.finalizers は Kubernetes 共通であるため、どのリソースでも同じ内容である事が確認できます。

$ oc explain StorageSystem.metadata.finalizers
GROUP:      odf.openshift.io
KIND:       StorageSystem
VERSION:    v1alpha1

FIELD: finalizers <[]string>

DESCRIPTION:
    Must be empty before the object is deleted from the registry. Each entry is
    an identifier for the responsible component that will remove the entry from
    the list. If the deletionTimestamp of the object is non-nil, entries in this
    list can only be removed. Finalizers may be processed and removed in any
    order.  Order is NOT enforced because it introduces significant risk of
    stuck finalizers. finalizers is a shared field, any actor with permission
    can reorder it. If the finalizer list is processed in order, then this can
    lead to a situation in which the component responsible for the first
    finalizer in the list is waiting for a signal (field value, external system,
    or other) produced by a component responsible for a finalizer later in the
    list, resulting in a deadlock. Without enforced ordering finalizers are free
    to order amongst themselves and are not vulnerable to ordering changes in
    the list.

oc explain Pod.metadata.finalizers
KIND:       Pod
VERSION:    v1

FIELD: finalizers <[]string>

DESCRIPTION:
    Must be empty before the object is deleted from the registry. Each entry is
    an identifier for the responsible component that will remove the entry from
    the list. If the deletionTimestamp of the object is non-nil, entries in this
    list can only be removed. Finalizers may be processed and removed in any
    order.  Order is NOT enforced because it introduces significant risk of
    stuck finalizers. finalizers is a shared field, any actor with permission
    can reorder it. If the finalizer list is processed in order, then this can
    lead to a situation in which the component responsible for the first
    finalizer in the list is waiting for a signal (field value, external system,
    or other) produced by a component responsible for a finalizer later in the
    list, resulting in a deadlock. Without enforced ordering finalizers are free
    to order amongst themselves and are not vulnerable to ordering changes in
    the list.

このような状態のリソースを削除する方法の1つとして、oc edit や oc patch による .metadata.finalizers の設定解除があります。

$ oc edit StorageSystem my-flashsystemcluster-storagesystem -n openshift-storage

$ oc patch StorageSystem my-flashsystemcluster-storagesystem -n openshift-storage -p '{"metadata":{"finalizers":null}}' --type=merge

.metadata.finalizers sample

.metadata.finalizers の設定状況は環境によって異なります。ここでは2つの環境の例をご紹介します。

Cluster sample #1

KIND                              FINALIZER
ClusterQueue                      [kueue.x-k8s.io/resource-in-use]
ClusterRole                       [tigera.io/cni-protector]
ClusterRoleBinding                [tigera.io/cni-protector]
ClusterSecret                     [kopf.zalando.org/KopfFinalizerMarker]
Config                            [imageregistry.operator.openshift.io/finalizer]
Config                            [samples.operator.openshift.io/finalizer]
DNS                               [dns.operator.openshift.io/dns-controller]
DNSRecord                         [operator.openshift.io/ingress-dns]
DSCInitialization                 [dscinitialization.opendatahub.io/finalizer]
DataScienceCluster                [datasciencecluster.opendatahub.io/finalizer]
DataSciencePipelinesApplication   [datasciencepipelinesapplications.opendatahub.io/finalizer]
InferenceService                  [inferenceservice.finalizers]
Ingress                           [ingresses.networking.internal.knative.dev ocp-ingress]
IngressController                 [ingresscontroller.operator.openshift.io/finalizer-ingresscontroller]
Installation                      [tigera.io/operator-cleanup]
KnativeServing                    [knativeservings.operator.knative.dev knative-serving-openshift]
KubeStorageVersionMigrator        [cluster-kube-storage-version-migrator-operator.operator.openshift.io/KubeStorageVersionMigrator]
NodeFeatureDiscovery              [foreground-deletion]
PersistentVolume                  [kubernetes.io/pv-protection external-attacher/vpc-block-csi-ibm-io]
PersistentVolume                  [kubernetes.io/pv-protection]
PersistentVolumeClaim             [kubernetes.io/pvc-protection]
ResourceFlavor                    [kueue.x-k8s.io/resource-in-use]
Route                             [routes.serving.knative.dev]
Service                           [service.kubernetes.io/load-balancer-cleanup]
ServiceAccount                    [tigera.io/cni-protector]
ServiceMeshControlPlane           [maistra.io/istio-operator]
ServiceMeshMember                 [maistra.io/istio-operator]
ServiceMeshMemberRoll             [maistra.io/istio-operator]
VolumeAttachment                  [external-attacher/vpc-block-csi-ibm-io]

Cluster sample #2

KIND                             FINALIZER
BackingStore                     [noobaa.io/finalizer]
BucketClass                      [noobaa.io/finalizer]
CephBlockPool                    [cephblockpool.ceph.rook.io]
CephCluster                      [cephcluster.ceph.rook.io]
CephFilesystem                   [cephfilesystem.ceph.rook.io]
CephFilesystemSubVolumeGroup     [cephfilesystemsubvolumegroup.ceph.rook.io]
ClusterServiceVersion            [operators.coreos.com/csv-cleanup]
Config                           [imageregistry.operator.openshift.io/finalizer]
ConfigMap                        [ceph.rook.io/disaster-protection]
Config                           [samples.operator.openshift.io/finalizer]
CredentialsRequest               [cloudcredential.openshift.io/deprovision]
CSIAddonsNode                    [csiaddons.openshift.io/csiaddonsnode]
DNS                              [dns.operator.openshift.io/dns-controller]
DNSRecord                        [operator.openshift.io/ingress-dns]
IngressController                [ingresscontroller.operator.openshift.io/finalizer-ingresscontroller]
KubeStorageVersionMigrator       [cluster-kube-storage-version-migrator-operator.operator.openshift.io/KubeStorageVersionMigrator]
NooBaa                           [noobaa.io/graceful_finalizer]
PersistentVolumeClaim            [kubernetes.io/pvc-protection]
PersistentVolume                 [external-provisioner.volume.kubernetes.io/finalizer kubernetes.io/pv-protection]
PersistentVolume                 [kubernetes.io/pv-protection external-attacher/ebs-csi-aws-com]
Secret                           [ceph.rook.io/disaster-protection]
Service                          [service.kubernetes.io/load-balancer-cleanup]
StorageCluster                   [storagecluster.ocs.openshift.io]
StorageSystem                    [storagesystem.odf.openshift.io]
VolumeAttachment                 [external-attacher/ebs-csi-aws-com]