問題
- PaspberryPi5上のAlmaLinux9.4でMicroShift（4.8）を立ち上げるとflannelのPodが作られない
$ oc get pods -n kube-system | grep flannel kube-flannel-ds-24dll 0/1 CreateContainerError 0 5m43s
解決策
- cgroupv2のメモリコントローラを有効化する
$ sudo sed -i -e "1 s/$/ cgroup_enable=memory/" /boot/cmdline.txt $ sudo reboot
解決までの経緯
- flannelの状態を確認
$ oc get pods -n kube-system | grep flannel kube-flannel-ds-24dll 0/1 CreateContainerError 0 5m43s -
kube-flannel-ds-24dllの詳細を確認$ oc describe pods -n kube-system kube-flannel-ds-24dll Name: kube-flannel-ds-24dll Namespace: kube-system ...（中略）... Normal Pulled 5m13s kubelet Successfully pulled image "quay.io/microshift/flannel:4.8.0-0.okd-2021-10-10-030117" in 48.989413278s Normal Created 5m13s kubelet Created container install-cni Normal Started 5m12s kubelet Started container install-cni Warning Failed 5m11s kubelet Error: container create failed: time="2024-09-03T07:48:21Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-20e7bcad2a1c0b50e78a7dd5e3c9ba6b49a0b6f944fe70679856282e296cda2b.scope/memory.events: no such file or directory" time="2024-09-03T07:48:21Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-20e7bcad2a1c0b50e78a7dd5e3c9ba6b49a0b6f944fe70679856282e296cda2b.scope/memory.max: no such file or directory" Warning Failed 5m10s kubelet Error: container create failed: time="2024-09-03T07:48:22Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-07e79f778e1a0368459d234e3a201be6565c77aaa696778120de03ae1e592658.scope/memory.events: no such file or directory" time="2024-09-03T07:48:22Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-07e79f778e1a0368459d234e3a201be6565c77aaa696778120de03ae1e592658.scope/memory.max: no such file or directory" Warning Failed 4m55s kubelet Error: container create failed: time="2024-09-03T07:48:37Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-2c2894680182139a1967ee1fd01e8e2a764f09dea7aa346f90cad5abc99ac691.scope/memory.events: no such file or directory" time="2024-09-03T07:48:37Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-2c2894680182139a1967ee1fd01e8e2a764f09dea7aa346f90cad5abc99ac691.scope/memory.max: no such file or directory" Warning Failed 4m33s kubelet Error: container create failed: time="2024-09-03T07:48:59Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-60877050a3d72ca863da01d0adbca82c36960c2424abb2fede98463527773847.scope/memory.events: no such file or directory" time="2024-09-03T07:48:59Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-60877050a3d72ca863da01d0adbca82c36960c2424abb2fede98463527773847.scope/memory.max: no such file or directory" Warning Failed 4m19s kubelet Error: container create failed: time="2024-09-03T07:49:13Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-036db4110351bb9e944850917b4d08354ba2cf09e85ace3ab6de1cc56bbd0a89.scope/memory.events: no such file or directory" time="2024-09-03T07:49:13Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-036db4110351bb9e944850917b4d08354ba2cf09e85ace3ab6de1cc56bbd0a89.scope/memory.max: no such file or directory" Warning Failed 4m3s kubelet Error: container create failed: time="2024-09-03T07:49:29Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-6716142d9b83c6891318fa461b26bc46e5535b85cab3f86ffffa8fbb4fedca99.scope/memory.events: no such file or directory" time="2024-09-03T07:49:29Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-6716142d9b83c6891318fa461b26bc46e5535b85cab3f86ffffa8fbb4fedca99.scope/memory.max: no such file or directory" Warning Failed 3m52s kubelet Error: container create failed: time="2024-09-03T07:49:40Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-77663b47bcb91fac2b2e28c054974f31cd808d004ac43906551472c7ba87fba2.scope/memory.events: no such file or directory" time="2024-09-03T07:49:40Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-77663b47bcb91fac2b2e28c054974f31cd808d004ac43906551472c7ba87fba2.scope/memory.max: no such file or directory" Warning Failed 3m40s kubelet Error: container create failed: time="2024-09-03T07:49:52Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-8078e233d83a6185a68c9c325b43adb090f1fe9dfb4c3ab886eaf4b4195733d9.scope/memory.events: no such file or directory" time="2024-09-03T07:49:52Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-8078e233d83a6185a68c9c325b43adb090f1fe9dfb4c3ab886eaf4b4195733d9.scope/memory.max: no such file or directory" Normal Pulled 78s (x19 over 5m12s) kubelet Container image "quay.io/microshift/flannel:4.8.0-0.okd-2021-10-10-030117" already present on machine - cgroup関連（memory?）の問題っぽいので、cgroupv2が有効になっているか確認
$ mount | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot) - 利用可能なcgroupのコントローラを確認
$ cat /sys/fs/cgroup/cgroup.controllers cpuset cpu io pids -
memoryの記述がないので、カーネルリングバッファのcgroup関連のメッセージを検索$ dmesg | grep -i cgroup [ 0.000811] cgroup: Disabling memory control group subsystem [ 3.568089] SELinux: policy capability cgroup_seclabel=1 [ 3.868919] systemd[1]: Relabelled /dev, /dev/shm, /run, /sys/fs/cgroup in 19.206ms. -
Disabling memory control group subsystemとあるので、メモリコントローラを有効化したい -
/boot/cmdline.txtというbootプロセス中にLinuxカーネルに渡されるカーネルパラメーターを含む、Raspberry Piシステム上のコンフィグファイルがあるらしいのでこれを編集$ cat /boot/cmdline.txt console=serial0,115200 console=tty1 root=PARTUUID=1f4a55c6-02 rootfstype=ext4 rootwait $ sudo sed -i -e "1 s/$/ cgroup_enable=memory/" /boot/cmdline.txt $ cat /boot/cmdline.txt console=serial0,115200 console=tty1 root=PARTUUID=1f4a55c6-02 rootfstype=ext4 rootwait cgroup_enable=memory - 再起動
$ sudo reboot - 再起動後、カーネルリングバッファを確認
$ dmesg | grep -i "cgroup\|memory" [ 0.000000] Reserved memory: created CMA memory pool at 0x0000000001c00000, size 320 MiB [ 0.000000] Early memory node ranges [ 0.000000] Kernel command line: reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe smsc95xx.macaddr=2C:CF:67:44:93:06 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000 console=ttyAMA10,115200 console=tty1 root=PARTUUID=1f4a55c6-02 rootfstype=ext4 rootwait cgroup_enable=memory [ 0.000000] cgroup: Enabling memory control group subsystem [ 0.000000] Memory: 7799420K/8380416K available (13120K kernel code, 2206K rwdata, 4284K rodata, 4928K init, 1102K bss, 253316K reserved, 327680K cma-reserved) [ 2.654119] Freeing unused kernel memory: 4928K [ 3.594043] SELinux: policy capability cgroup_seclabel=1 [ 3.888098] systemd[1]: Relabelled /dev, /dev/shm, /run, /sys/fs/cgroup in 19.263ms. - cgroupコントローラの確認
$ cat /sys/fs/cgroup/cgroup.controllers cpuset cpu io memory pids - flannelのPodが正常起動していることを確認
$ oc get pods -n kube-system | grep flannel kube-flannel-ds-vm2q4 1/1 Running 0 13m