2
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

解决docker删除container出现device or resource busy

Posted at

问题:

基本环境是CentOS7.3 kernel version:3.10.0-514.el7.x86_64

docker info:
Containers: 10
Running: 10
Paused: 0
Stopped: 0
Images: 31
Server Version: 17.12.0-ce
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 40
Total Memory: 31.15GiB
Name: web-130.fof.lczq.com
ID: JBM5:LNHC:TWFW:XJHW:P7SA:6KCE:HT23:HAFE:EWC3:TWBJ:XOZK:PNG5
Docker Root Dir: /data1/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://wmsl2uyx.mirror.aliyuncs.com/
Live Restore Enabled: false

服务器因为电源问题自动关机,重启以后发现有的container不能被删除。

docker-compose down

报错:
Error response from daemon: driver "overlay" failed to remove root filesystem for <some-id>: 
remove /data1/docker/overlay/<some-id>/merged: device or resource busy

排查

经过排查发现是:/data1/docker/overlay//merged 是空的,但是无法删除。手动rm也会报"device or resouce busy"

尝试过重启docker,但是问题依旧。即使将docker stop,也无法删除目录。

查看mount信息,也没有发现任何挂载信息。

在docker的service里增加了

[Service]
# MountFlags "slave" helps to prevent "device busy" errors on RHEL/CentOS 7.3 kernels
MountFlags=slave

也没有解决

解决

最后通过搜索,发现可以通过查看进程的mountinfo来查找占用目录的进程。

grep <some-id> /proc/*/mountinfo

一般通过查找可以找到PID。

根据这个PID发现是一个/usr/libexec/colord的程序在使用。

查了一下程序的信息,发现确实是将之前删除不了的目录写在了mountinfo中。

找到进程就好办了。直接将这个进程kill掉。

重新执行docker-compose down,提示删除成功。

分析

估计是重启之前container的pid被这个colord占用了。导致container无法启动,同时删除的时候因为将目录写入了mountinfo中,导致无法删除。

2
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
2
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?