LoginSignup
9
5

More than 3 years have passed since last update.

@stwind

bpftraceで Docker の仕組み調べてみる

Last updated at Posted at 2020-07-13

概要

Dockerはよくdocker runとかdocker execのコマンドで Black Box みたいに使われて完結ですが、その中身をもっと詳しく調査してみたい。そのための有力ツールの一つはbpftraceです。

作業環境

  • Ubuntu 20.04

Dockerの構成

Dockerはおよそ以下のパーツに構成されている

  • docker-cli：docker runとdocker execなどのコマンドを提供するDockerのクライアント側
  • dockcerd：Docker のサーバー側、Docker Engine のデーモンプロセス、docker-cliなどクライアントからのリクエストを処理する場所
  • containerd：Container Runtime の一つ、いろんな Container の管理を機能しているデーモンプログラム
  • runc：いろんなOS機能を利用して、単独のContainerの実行を成せているツール

Dockerがやっていること

Containerは Namespace、Cgroup、OverlayFS、Virtual Network などのOS機能利用して可能になるのがよくご存知ですが、Docker はこちらの機能どう利用したますか？

まずは、Container が実行した結果を確認していきます。この一番簡単の Container 起動します

$ docker run -ti --rm --name test alpine sh

これからの利用のために pid と Container ID をに保存しておきます

$ export CPID=$(docker inspect test -f '{{ .State.Pid }}')
$ echo $CPID
230973
$ export CID=$(docker inspect test -f '{{ .Id }}')
$ echo $CID
b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0

containerdとの親子関係

$ pstree -s $CPID -STUpau
systemd,1
  └─containerd,80405
      └─containerd-shim,230955 -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd ...
          └─sh,230973,ipc,mnt,net,pid,uts

containerdはcontainerd-shimを実行して、そしてまた Container プロセスを実行する。また、このContainer プロセスにはipcとかmntなどの namespace がついている。

runcの設定

containerdがruncを利用してるので、その設定も確認します

$ jq -r '.process' < /var/run/containerd/io.containerd.runtime.v1.linux/moby/$CID/config.json
{
  "terminal": true,
  "user": {
    "uid": 0,
    "gid": 0,
　　...
  },
  "args": [
    "sh"
  ],
  "env": [
    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "HOSTNAME=b6431f14cf40",
    "TERM=xterm"
  ],
  "cwd": "/",
  "capabilities": {
    "bounding": [
      "CAP_CHOWN",
      "CAP_DAC_OVERRIDE",
      "CAP_FSETID",
      "CAP_FOWNER",
      "CAP_MKNOD",
      "CAP_NET_RAW",
      "CAP_SETGID",
      "CAP_SETUID",
      "CAP_SETFCAP",
      "CAP_SETPCAP",
      "CAP_NET_BIND_SERVICE",
      "CAP_SYS_CHROOT",
      "CAP_KILL",
      "CAP_AUDIT_WRITE"
    ],
    ...
  },
  "apparmorProfile": "docker-default",
  "oomScoreAdj": 0
}

Namespaces

Container プロセスについている namespace を具体的に見に行きましょう

$ lsns -p $CPID --output-all
        NS TYPE   PATH                NPROCS    PID   PPID COMMAND    UID USER NETNSID NSFS
4026531835 cgroup /proc/1/ns/cgroup      120      1      0 /sbin/init   0 root
4026531837 user   /proc/1/ns/user        120      1      0 /sbin/init   0 root
4026532186 mnt    /proc/230973/ns/mnt      1 230973 230955 sh           0 root
4026532187 uts    /proc/230973/ns/uts      1 230973 230955 sh           0 root
4026532188 ipc    /proc/230973/ns/ipc      1 230973 230955 sh           0 root
4026532189 pid    /proc/230973/ns/pid      1 230973 230955 sh           0 root
4026532191 net    /proc/230973/ns/net      1 230973 230955 sh           0 root       0 /run/docker/netns/904e1ae9696c

Cgroups

所属している Cgroup はこちら

$ cat /proc/$CPID/cgroup
12:freezer:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
11:devices:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
10:blkio:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
9:rdma:/
8:pids:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
7:memory:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
6:hugetlb:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
5:net_cls,net_prio:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
4:cpu,cpuacct:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
3:cpuset:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
2:perf_event:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
1:name=systemd:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
0::/system.slice/containerd.service

そして Cgroup の内容もちょっと見に行きましよう

$ cgget -g pids:/docker/$CID
/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0:
pids.current: 1
pids.events: max 0
pids.max: max
$ cgget -g net_cls,net_prio:/docker/$CID
/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0:
net_cls.classid: 0
net_prio.prioidx: 3
net_prio.ifpriomap: lo 0
    enp0s3 0
    docker0 0
    veth18981d6 0

Mounts

このContainer プロセスがマウントしたディレクトリはこちら

$ cat /proc/$CPID/mounts
overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/JVWDDYIF5YQFGNGL3PDNWJ4A4M:/var/lib/docker/overlay2/l/TQBVPLJN6SEMDRW7VD5FBARG4F,upperdir=/var/lib/docker/overlay2/ae3eab82db4efee497c2d69a4ad18a8cfc816ce0d61296fee7b6f611d7f6ebb3/diff,workdir=/var/lib/docker/overlay2/ae3eab82db4efee497c2d69a4ad18a8cfc816ce0d61296fee7b6f611d7f6ebb3/work,xino=off 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev tmpfs rw,nosuid,size=65536k,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
sysfs /sys sysfs ro,nosuid,nodev,noexec,relatime 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup ro,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
cgroup /sys/fs/cgroup/perf_event cgroup ro,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/cpuset cgroup ro,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup ro,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup ro,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup ro,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/memory cgroup ro,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/pids cgroup ro,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/rdma cgroup ro,nosuid,nodev,noexec,relatime,rdma 0 0
cgroup /sys/fs/cgroup/blkio cgroup ro,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/devices cgroup ro,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup ro,nosuid,nodev,noexec,relatime,freezer 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
/dev/sda1 /etc/resolv.conf ext4 rw,relatime 0 0
/dev/sda1 /etc/hostname ext4 rw,relatime 0 0
/dev/sda1 /etc/hosts ext4 rw,relatime 0 0
devpts /dev/console devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
proc /proc/bus proc ro,relatime 0 0
proc /proc/fs proc ro,relatime 0 0
proc /proc/irq proc ro,relatime 0 0
proc /proc/sys proc ro,relatime 0 0
proc /proc/sysrq-trigger proc ro,relatime 0 0
tmpfs /proc/acpi tmpfs ro,relatime 0 0
tmpfs /proc/kcore tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/keys tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/timer_list tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/sched_debug tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/scsi tmpfs ro,relatime 0 0
tmpfs /sys/firmware tmpfs ro,relatime 0 0

ここで注目するのは overlay の情報

overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/JVWDDYIF5YQFGNGL3PDNWJ4A4M:/var/lib/docker/overlay2/l/TQBVPLJN6SEMDRW7VD5FBARG4F,upperdir=/var/lib/docker/overlay2/ae3eab82db4efee497c2d69a4ad18a8cfc816ce0d61296fee7b6f611d7f6ebb3/diff,workdir=/var/lib/docker/overlay2/ae3eab82db4efee497c2d69a4ad18a8cfc816ce0d61296fee7b6f611d7f6ebb3/work,xino=off 0 0

ここでマウントした/var/lib/docker/overlay2/l/TQBVPLJN6SEMDRW7VD5FBARG4Fは実は alpine イメージの内容です

$ cat $(docker image inspect alpine -f '{{ .GraphDriver.Data.UpperDir }}')/../link
TQBVPLJN6SEMDRW7VD5FBARG4F

ネットワーク

最後にネットワークの設定も確認しましよう。Container プロセスは独自の network namespace に区別しているから、host mahcineと通信のためには、vethを利用します。

まずは、Container側のネットワーク設定

$ nsenter -t $CPID -n ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.2  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:ac:11:00:02  txqueuelen 0  (Ethernet)
        RX packets 17  bytes 1366 (1.3 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
$ nsenter -t $CPID -n ip link show type veth
280: eth0@if281: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
$ nsenter -t $CPID -n ip route show dev eth0
default via 172.17.0.1
172.17.0.0/16 proto kernel scope link src 172.17.0.2

host machine 側

$ ifconfig docker0
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:ffff:fe14:4bd6  prefixlen 64  scopeid 0x20<link>
        ether 02:42:ff:14:4b:d6  txqueuelen 0  (Ethernet)
        RX packets 8  bytes 433 (433.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 283  bytes 30567 (30.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
$ ip addr show type veth
281: veth18981d6@if280: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether fa:eb:12:ad:57:d1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::f8eb:12ff:fead:57d1/64 scope link
       valid_lft forever preferred_lft forever
$ ip route show dev docker0
172.17.0.0/16 proto kernel scope link src 172.17.0.1

Container のアドレスは172.17.0.2で、hostの方は172.17.0.1、veth18981d6という veth で２つの interface を繋がている。そしてそれぞれの route table に転送経路のルールが載せている。

bpftraceでDocker実行順番を明白する

さて、いろいろの結果を確認してきたから、具体的に Docker はどうやってその結果のたどり着いたか、例えばいつにどんな関数をどんな変数で呼びましたか。ここではbpftraceが活躍するところです。

bpftrace っていうのは Linux Kenel の eBPF 機能を awk みたい記述で簡単に利用できるツールです。具体的な使用方法はここでは省きますが、Docに参照してください。

bpftraceのセットアップ

# bcc
$ apt-get install -y linux-headers-$(uname -r) bison build-essential cmake flex g++ git libelf-dev zlib1g-dev libfl-dev systemtap-sdt-dev binutils-dev llvm-8-dev llvm-8-runtime libclang-8-dev clang-8 arping netperf iperf3 python3-distutils
$ git clone --recurse-submodules https://github.com/iovisor/bcc.git
$ mkdir bcc/build; cd bcc/build
$ cmake -DPYTHON_CMD=python3 ..
$ make -j8 && make install && ldconfig

$ cd ../..

# bpftrace
$ git clone https://github.com/iovisor/bpftrace.git
$ mkdir bpftrace/build; cd bpftrace/build
$ cmake -DHAVE_BCC_PROG_LOAD=ON -DHAVE_BCC_CREATE_MAP=ON -DBUILD_TESTING=OFF ..
$ make -j8 && make install

Dockerをトレース

bpftraceで docker と OS のやり取りがトレースすることができる。まずはこちらの内容でdocker.btのファイル作成する

docker.bt
#!/usr/bin/env bpftrace

#include <linux/nsproxy.h>
#include <linux/ns_common.h>
#include <linux/utsname.h>
#include <linux/pid_namespace.h>
#include <linux/ipc_namespace.h>
#include <linux/cgroup.h>
#include <net/net_namespace.h>
#include <linux/netdevice.h>

BEGIN { 
    printf("%-12s %-15s %-8s %-10s %s\n", "TIME", "COMMAND", "PID", "ACTION", "CONTENT"); 
}

tracepoint:syscalls:sys_enter_execve { 
    printf("%-12ld %-15s %-8d %-10s ", elapsed , comm, pid, "execve");
    join(args->argv); 
}

tracepoint:syscalls:sys_enter_mount { 
    printf("%-12ld %-15s %-8d %-10s ", elapsed, comm, pid, "mount");
    printf("type=%s, dev=%s, dir=%s\n", str(args->type), str(args->dev_name), str(args->dir_name))
}

kretprobe:create_new_namespaces /comm=="runc:[1:CHILD]"/ { 
    printf("%-12ld %-15s %-8d %-10s ", elapsed, comm, pid, "namespace");
    $nsp = (struct nsproxy *)retval;
    printf("uts=%ld, ipc=%ld, cgroup=%ld, net=%ld, pid=%ld\n", 
        ((struct uts_namespace *)$nsp->uts_ns)->ns.inum,
        ((struct ipc_namespace *)$nsp->ipc_ns)->ns.inum,
        ((struct cgroup_namespace *)$nsp->cgroup_ns)->ns.inum,
        ((struct net *)$nsp->net_ns)->ns.inum,
        ((struct pid_namespace *)$nsp->pid_ns_for_children)->ns.inum);
}

kprobe:veth_newlink {
    printf("%-12ld %-15s %-8d %-10s ", elapsed, comm, pid, "veth");
    printf("name=%s, netns=%ld\n", ((struct net_device *)arg1)->name, ((struct net *)arg0)->ns.inum);
}

そして実行する

$ ./docker.bt
Attaching 5 probes...
TIME         COMMAND         PID      ACTION     CONTENT

もう一度 Container の起動する（別のターミナル）

$ docker run -ti --rm --name test alpine echo hi

docker.btに戻せば、こちらの内容出力されます

TIME         COMMAND         PID      ACTION     CONTENT
1827725094   bash            235026   execve     docker run -ti --rm --name test alpine echo hi
1874368200   dockerd         81563    mount      type=overlay, dev=overlay, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
1892118900   dockerd         81563    mount      type=overlay, dev=overlay, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
1904728454   dockerd         81563    mount      type=overlay, dev=overlay, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
1905395722   dockerd         81563    veth       name=veth3d71f6e, netns=4026531992
1912608987   networkd-dispat 235038   execve     /usr/bin/networkctl list --no-pager --no-legend
1913900866   (spawn)         235036   execve     /lib/udev/bridge-network-interface
1917882282   (spawn)         235039   execve     /lib/open-iscsi/net-interface-handler start
1922126799   (spawn)         235040   execve     /lib/udev/bridge-network-interface
1923736757   (spawn)         235041   execve     /lib/systemd/systemd-sysctl --prefix=/net/ipv4/conf/veth3d71f6e --prefix=/net/ipv4/neigh/veth3d71f6e --prefix=/net/ipv6/conf/veth3d71f6e --prefix=/net/ipv6/neigh/veth3d71f6e
1925374081   (spawn)         235042   execve     /lib/open-iscsi/net-interface-handler start
1935733149   containerd      235044   execve     containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
1940074927   containerd-shim 235053   execve     runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb/log.json --log-format json create --bundle /run/containerd/io.containerd.runtime.v1.linux/moby/6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb --pid-file /run/containerd/io.containerd.runtime.v1.linux/moby/6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb/init.pid --console-socket /tmp/pty046813817/pty.sock 6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb
1940243736   (spawn)         235043   execve     /lib/systemd/systemd-sysctl --prefix=/net/ipv4/conf/vethc30992d --prefix=/net/ipv4/neigh/vethc30992d --prefix=/net/ipv6/conf/vethc30992d --prefix=/net/ipv6/neigh/vethc30992d
1955997514   runc            235060   execve     runc init
1959669403   exe             235060   mount      type=, dev=/proc/self/exe, dir=/var/run/docker/runtime-runc/moby/6975e3344ad32b92a6a09066f3113
1962524451   exe             235060   mount      type=, dev=, dir=/var/run/docker/runtime-runc/moby/6975e3344ad32b92a6a09066f3113
1977110103   runc:[1:CHILD]  235061   namespace  uts=4026532187, ipc=4026532188, cgroup=4026531835, net=4026532191, pid=4026532189
2002233022   runc:[2:INIT]   235062   mount      type=, dev=, dir=/
2002647666   runc:[2:INIT]   235062   mount      type=bind, dev=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2002787106   runc:[2:INIT]   235062   mount      type=proc, dev=proc, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2002848509   runc:[2:INIT]   235062   mount      type=tmpfs, dev=tmpfs, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2003072500   runc:[2:INIT]   235062   mount      type=devpts, dev=devpts, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2006987497   runc:[2:INIT]   235062   mount      type=sysfs, dev=sysfs, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2007808775   runc:[2:INIT]   235062   mount      type=tmpfs, dev=tmpfs, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008001937   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/systemd/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008021032   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/systemd/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008102923   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/perf_event/docker/6975e3344ad32b92a6a09066f31132, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008116742   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/perf_event/docker/6975e3344ad32b92a6a09066f31132, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008189722   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/cpuset/docker/6975e3344ad32b92a6a09066f31132f9b4, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008203301   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/cpuset/docker/6975e3344ad32b92a6a09066f31132f9b4, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008284557   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/cpu,cpuacct/docker/6975e3344ad32b92a6a09066f3113, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008300251   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/cpu,cpuacct/docker/6975e3344ad32b92a6a09066f3113, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008365239   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/net_cls,net_prio/docker/6975e3344ad32b92a6a09066, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008397968   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/net_cls,net_prio/docker/6975e3344ad32b92a6a09066, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008471763   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/hugetlb/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008489627   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/hugetlb/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008575004   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/memory/docker/6975e3344ad32b92a6a09066f31132f9b4, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008588697   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/memory/docker/6975e3344ad32b92a6a09066f31132f9b4, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008655100   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/pids/docker/6975e3344ad32b92a6a09066f31132f9b4e3, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008667811   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/pids/docker/6975e3344ad32b92a6a09066f31132f9b4e3, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008731445   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/rdma, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008744138   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/rdma, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008792554   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/blkio/docker/6975e3344ad32b92a6a09066f31132f9b4e, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008805034   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/blkio/docker/6975e3344ad32b92a6a09066f31132f9b4e, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008855551   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/devices/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008868073   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/devices/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008919459   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/freezer/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008932321   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/freezer/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008958905   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008977832   runc:[2:INIT]   235062   mount      type=mqueue, dev=mqueue, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009006418   runc:[2:INIT]   235062   mount      type=tmpfs, dev=shm, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009102662   runc:[2:INIT]   235062   mount      type=bind, dev=/var/lib/docker/containers/6975e3344ad32b92a6a09066f31132f9b4e3, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009116671   runc:[2:INIT]   235062   mount      type=, dev=, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009152544   runc:[2:INIT]   235062   mount      type=bind, dev=/var/lib/docker/containers/6975e3344ad32b92a6a09066f31132f9b4e3, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009184117   runc:[2:INIT]   235062   mount      type=, dev=, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009256053   runc:[2:INIT]   235062   mount      type=bind, dev=/var/lib/docker/containers/6975e3344ad32b92a6a09066f31132f9b4e3, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009268672   runc:[2:INIT]   235062   mount      type=, dev=, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2012566089   runc            235085   execve     libnetwork-setkey -exec-root=/var/run/docker 6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb ee4af79b8952
2055580696   dockerd         81563    mount      type=bind, dev=/proc/235062/ns/net, dir=/var/run/docker/netns/53666134ee2d
2058245033   dockerd         235093   execve     set-ipv6 /var/run/docker/netns/53666134ee2d all false
2117481953   (spawn)         235100   execve     /lib/open-iscsi/net-interface-handler stop
2135647000   runc:[2:INIT]   235062   mount      type=, dev=, dir=.
2135881831   runc:[2:INIT]   235062   mount      type=bind, dev=/dev/pts/0, dir=/dev/console
2135959898   runc:[2:INIT]   235062   mount      type=, dev=/proc/bus, dir=/proc/bus
2135974823   runc:[2:INIT]   235062   mount      type=, dev=/proc/bus, dir=/proc/bus
2135984627   runc:[2:INIT]   235062   mount      type=, dev=/proc/fs, dir=/proc/fs
2135995698   runc:[2:INIT]   235062   mount      type=, dev=/proc/fs, dir=/proc/fs
2136004311   runc:[2:INIT]   235062   mount      type=, dev=/proc/irq, dir=/proc/irq
2136017445   runc:[2:INIT]   235062   mount      type=, dev=/proc/irq, dir=/proc/irq
2136025919   runc:[2:INIT]   235062   mount      type=, dev=/proc/sys, dir=/proc/sys
2136036802   runc:[2:INIT]   235062   mount      type=, dev=/proc/sys, dir=/proc/sys
2136045583   runc:[2:INIT]   235062   mount      type=, dev=/proc/sysrq-trigger, dir=/proc/sysrq-trigger
2136057589   runc:[2:INIT]   235062   mount      type=, dev=/proc/sysrq-trigger, dir=/proc/sysrq-trigger
2136066994   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/asound
2136076692   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/acpi
2136145362   runc:[2:INIT]   235062   mount      type=tmpfs, dev=tmpfs, dir=/proc/acpi
2136198403   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/kcore
2136211095   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/keys
2136271650   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/latency_stats
2136281142   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/timer_list
2136292991   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/timer_stats
2136302265   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/sched_debug
2136312542   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/scsi
2136411156   runc:[2:INIT]   235062   mount      type=tmpfs, dev=tmpfs, dir=/proc/scsi
2136458345   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/sys/firmware
2136537743   runc:[2:INIT]   235062   mount      type=tmpfs, dev=tmpfs, dir=/sys/firmware
2325058622   containerd-shim 235102   execve     runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb/log.json --log-format json start 6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb
2334946680   runc:[2:INIT]   235062   execve     echo hi
2357752100   containerd-shim 235108   execve     /usr/bin/containerd --address /run/containerd/containerd.sock publish --topic /tasks/exit --namespace moby
2389193617   containerd-shim 235116   execve     runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb/log.json --log-format json delete 6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb
2438983541   (spawn)         235129   execve     /lib/udev/bridge-network-interface
2441125682   networkd-dispat 235130   execve     /usr/bin/networkctl list --no-pager --no-legend
2441917196   (spawn)         235131   execve     /lib/open-iscsi/net-interface-handler start
2445708579   (spawn)         235132   execve     /lib/systemd/systemd-sysctl --prefix=/net/ipv4/conf/vethc30992d --prefix=/net/ipv4/neigh/vethc30992d --prefix=/net/ipv6/conf/vethc30992d --prefix=/net/ipv6/neigh/vethc30992d
2460630127   networkd-dispat 235133   execve     /usr/bin/networkctl list --no-pager --no-legend
2469750858   (spawn)         235134   execve     /lib/open-iscsi/net-interface-handler stop
2470070134   (spawn)         235135   execve     /lib/open-iscsi/net-interface-handler stop

これは Docker がこの Container を実行するためにやったことは一目瞭然です。簡単にまとめると、順番的にはこちらです

  • イメージの overlay をマウントする
  • veth などのネットワーク設定
  • containerd-shimを実行
  • runc init
  • namespaces 作り
  • cgroup などにいろんなマウント
  • runc start
  • ユーザーコマンド echo hi
  • runc delete
  • ネットワークの cleanup

参考資料

9
5
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
9
5