目的
RedHat OpenShift on IBM Cloud（ROKS）では、ワーカーノード上のネットワークトラブル調査のためにパケットキャプチャが必要になる場面がある。
ROKSでは、"oc debug node"コマンドを使えば、ノード上でtcpdumpを実行することが可能
本記事では、ROKS環境で"oc debug node"を使ってパケットキャプチャを行う手順を記載する
前提
OpenShift クラスタに oc CLI でアクセス可能
oc debug コマンドが使える
権限として cluster-admin に準ずる権限が必要
事前確認
パケット通信確認の対象として、corednsの通信を確認してみる
corednsのPODを確認
kubectl get pods -n openshift-dns -o wide
% kubectl get pods -n openshift-dns -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
dns-default-lhngm 2/2 Running 0 22d 172.17.137.57 10.244.64.21 <none> <none>
dns-default-n92dv 2/2 Running 0 21d 172.17.173.4 10.244.0.4 <none> <none>
node-resolver-cdv7l 1/1 Running 0 22d 10.244.64.21 10.244.64.21 <none> <none>
node-resolver-ltr4f 1/1 Running 0 21d 10.244.0.4 10.244.0.4 <none> <none>
パケットキャプチャ手順
1.ノード名を確認
oc get node
% oc get node
NAME STATUS ROLES AGE VERSION
10.244.0.4 Ready master,worker 21d v1.27.16+03a907c
10.244.64.21 Ready master,worker 120d v1.27.16+03a907c
2.oc debugでノードにアクセス
oc debug node/<ノード名>
% oc debug node/10.244.0.4
Starting pod/1024404-debug-5t2qn ...
To use host binaries, run `chroot /host`
Pod IP: 10.244.0.4
If you don't see a command prompt, try pressing enter.
sh-4.4#
3.ノードの実OS環境に入る
chroot /host
sh-4.4# chroot /host
sh-4.4#
4.tcpdumpを導入
RHELノードの場合は以下の通り
yum -y install tcpdump
sh-4.4# which tcpdump
which: no tcpdump in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
sh-4.4# yum -y install tcpdump
Updating Subscription Management repositories.
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) 101 kB/s | 4.1 kB 00:00
Red Hat Enterprise Linux 8 for x86_64 - Supplementary (RPMs) 100 kB/s | 3.8 kB 00:00
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) 113 kB/s | 4.5 kB 00:00
Red Hat Ansible Engine 2 for RHEL 8 x86_64 (RPMs) 96 kB/s | 4.0 kB 00:00
Dependencies resolved.
======================================================================================================================================================================================================================
Package Architecture Version Repository Size
======================================================================================================================================================================================================================
Installing:
tcpdump x86_64 14:4.9.3-5.el8 rhel-8-for-x86_64-appstream-rpms 453 k
Transaction Summary
======================================================================================================================================================================================================================
Install 1 Package
Total download size: 453 k
Installed size: 1.1 M
Downloading Packages:
tcpdump-4.9.3-5.el8.x86_64.rpm 1.5 MB/s | 453 kB 00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 1.5 MB/s | 453 kB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: tcpdump-14:4.9.3-5.el8.x86_64 1/1
Installing : tcpdump-14:4.9.3-5.el8.x86_64 1/1
Running scriptlet: tcpdump-14:4.9.3-5.el8.x86_64 1/1
Verifying : tcpdump-14:4.9.3-5.el8.x86_64 1/1
Installed products updated.
Installed:
tcpdump-14:4.9.3-5.el8.x86_64
Complete!
バージョン確認
# tcpdump --version
tcpdump version 4.9.3
libpcap version 1.9.1 (with TPACKET_V3)
OpenSSL 1.1.1k FIPS 25 Mar 2021
5.パケットキャプチャの実施
tcpdumpコマンドでパケットをキャプチャ可能
実際に通信が発生した場合にパケットログが流れる
サンプル:[UDP Port53に絞ってキャプチャ]
tcpdump -lnei eth0 udp and port 53
# tcpdump -lnei eth0 udp and port 53
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:12:40.256384 02:00:03:60:57:a4 > 02:00:00:60:57:a4, ethertype IPv4 (0x0800), length 109: 10.244.0.4.64283 > 161.26.0.7.domain: 12748+ [1au] A? c115-3.jp-tok.containers.cloud.ibm.com. (67)
19:12:40.256463 02:00:03:60:57:a4 > 02:00:00:60:57:a4, ethertype IPv4 (0x0800), length 109: 10.244.0.4.64283 > 161.26.0.7.domain: 34479+ [1au] A? c115-1.jp-tok.containers.cloud.ibm.com. (67)
19:12:40.256487 02:00:03:60:57:a4 > 02:00:00:60:57:a4, ethertype IPv4 (0x0800), length 109: 10.244.0.4.64283 > 161.26.0.7.domain: 48738+ [1au] A? c115-2.jp-tok.containers.cloud.ibm.com. (67)
19:12:40.256506 02:00:03:60:57:a4 > 02:00:00:60:57:a4, ethertype IPv4 (0x0800), length 135: 10.244.0.4.64283 > 161.26.0.7.domain: 64414+ [1au] A? cq29stct0hgp9gqe8hsg.vpe.private.jp-tok.containers.cloud.ibm.com. (93)
10.244.0.4のワーカーノード上のcorednsを経由し、IBMCloudのDNSサーバにフォワードしていることがわかる(161.26.0.7)
別のPODから「www.ibm.com」にアクセスすると、corednsを経由して問い合わせていることが確認できる
/ # curl www.ibm.com
19:16:07.960635 02:00:03:60:57:a4 > 02:00:00:60:57:a4, ethertype IPv4 (0x0800), length 71: 10.244.0.4.60156 > 161.26.0.7.domain: 36566+ AAAA? www.ibm.com. (29)
19:16:07.960760 02:00:03:60:57:a4 > 02:00:00:60:57:a4, ethertype IPv4 (0x0800), length 71: 10.244.0.4.52988 > 161.26.0.7.domain: 20412+ A? www.ibm.com. (29)
19:16:07.963025 02:00:00:60:57:a4 > 02:00:03:60:57:a4, ethertype IPv4 (0x0800), length 260: 161.26.0.7.domain > 10.244.0.4.52988: 20412 3/0/0 CNAME outer-global-dual.ibmcom-tls12.edgekey.net., CNAME e7817.dscx.akamaiedge.net., A 23.34.107.97 (218)
19:16:07.964595 02:00:00:60:57:a4 > 02:00:03:60:57:a4, ethertype IPv4 (0x0800), length 325: 161.26.0.7.domain > 10.244.0.4.60156: 36566 4/0/0 CNAME outer-global-dual.ibmcom-tls12.edgekey.net., CNAME e7817.dscx.akamaiedge.net., AAAA 2600:140b:a800:799::1e89, AAAA 2600:140b:a800:7ad::1e89 (283)
ファイルに保存する場合
tcpdump -lnei eth0 udp and port 53 -w /tmp/capture.pcap
# tcpdump -lnei eth0 udp and port 53 -w /tmp/capture.pcap
dropped privs to tcpdump
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C8 packets captured
17 packets received by filter
0 packets dropped by kernel
pcapファイルをダウンロードする
oc debug node/ -- chroot /host cat /tmp/capture.pcap > capture.pcap
% oc debug node/10.244.0.4 -- chroot /host cat /tmp/capture.pcap > capture.pcap
Starting pod/1024404-debug-s49w8 ...
To use host binaries, run `chroot /host`
Removing debug pod ...
pcap % ls
capture.pcap
Macでpcapファイルを確認する
tcpdump -nn -r capture.pcap
% tcpdump -nn -r capture.pcap
reading from file capture.pcap, link-type EN10MB (Ethernet)
09:18:47.707687 IP 10.244.0.4.46900 > 161.26.0.7.53: 531+ AAAA? www.ibm.com. (29)
09:18:47.707809 IP 10.244.0.4.33619 > 161.26.0.7.53: 48994+ A? www.ibm.com. (29)
09:18:47.709646 IP 161.26.0.7.53 > 10.244.0.4.46900: 531 4/0/0 CNAME outer-global-dual.ibmcom-tls12.edgekey.net., CNAME e7817.dscx.akamaiedge.net., AAAA 2600:140b:a800:799::1e89, AAAA 2600:140b:a800:7ad::1e89 (283)
09:18:47.709774 IP 161.26.0.7.53 > 10.244.0.4.33619: 48994 3/0/0 CNAME outer-global-dual.ibmcom-tls12.edgekey.net., CNAME e7817.dscx.akamaiedge.net., A 23.34.107.97 (218)
09:18:51.204991 IP 10.244.0.4.33619 > 161.26.0.7.53: 45355+ A? logs.private.jp-tok.logging.cloud.ibm.com. (59)
09:18:51.207007 IP 161.26.0.7.53 > 10.244.0.4.33619: 45355 NXDomain 0/1/0 (150)
09:18:54.936788 IP 10.244.0.4.33619 > 161.26.0.7.53: 52507+ AAAA? logs.private.jp-tok.logging.cloud.ibm.com. (59)
09:18:54.939596 IP 161.26.0.7.53 > 10.244.0.4.33619: 52507 NXDomain 0/1/0 (150)