Go to Qiita Advent Calendar 2024 Top
LoginSignup
1
2

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

@aandby(Anjack AandBy)

Calico と CoreDNS で Kubernetes Cluster 外から サービス名でアクセス

Posted at

#####● 何をしたいのか。

  1. Calico で Pod と同じように Service を扱いたい。
  2. CoreDNS の名前解決を外部から使いたい。

#####● 事前準備
● Kubernetes 環境
KVM環境下に Master 1 , Node 2 で構築
CNI は、 Calico

$ kubectl get nodes
NAME                 STATUS   ROLES    AGE    VERSION
070master.testk.jp   Ready    master   5d2h   v1.16.1
071node01.testk.jp   Ready    <none>   5d1h   v1.16.1
072node02.testk.jp   Ready    <none>   5d1h   v1.16.1

● Nginx を Deployment で配置し、サービスを登録

$ kubectl get deployment -n apptest
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
deployment01   2/2     2            2           43h
$ kubectl get pod -n apptest
NAME                            READY   STATUS    RESTARTS   AGE
deployment01-7b4b9477fb-9z8ws   1/1     Running   2          43h
deployment01-7b4b9477fb-ff652   1/1     Running   2          43h
$ kubectl get svc -n apptest
NAME        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service01   ClusterIP   10.20.71.66     <none>        80/TCP    43h

● kube-dns を確認

$ kubectl get svc -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.20.0.10   <none>        53/UDP,53/TCP,9153/TCP   5d3h

● Advertising Kubernetes services over BGP
下記のページを参照し、設定
Advertising Kubernetes services over BGP
kubeadm init 実行時のオプション
--service-cidr=10.20.0.0/16

$ kubectl describe ds/calico-node -n kube-system | grep CALICO_ADVERTISE_CLUSTER_IPS
      CALICO_ADVERTISE_CLUSTER_IPS:       10.20.0.0/16

● BGP ルータ 2 を用意

router1
# birdc show status
BIRD 2.0.4 ready.
BIRD 2.0.4
Router ID is 10.0.0.51
Current server time is 2019-10-19 17:48:01.962
Last reboot on 2019-10-19 12:07:21.365
Last reconfiguration on 2019-10-19 12:07:21.365
Daemon is up and running

router2
# birdc show status
BIRD 2.0.4 ready.
BIRD 2.0.4
Router ID is 10.0.0.52
Current server time is 2019-10-19 17:48:19.236
Last reboot on 2019-10-19 12:08:14.849
Last reconfiguration on 2019-10-19 12:08:14.849
Daemon is up and running

● /etc/resolv.conf に nameserver の追加
router2 は、router1 を指定

router1
# cat /etc/resolv.conf

nameserver 10.20.0.10
nameserver 192.168.122.1

router2
# cat /etc/resolv.conf

nameserver 10.0.0.51
nameserver 192.168.122.1

● router1 に Dnsmasq をインストール

# systemctl status dnsmasq
● dnsmasq.service - DNS caching server.
   Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; disabled; vendor preset: disabled)
   Active: active (running) since 土 2019-10-19 12:52:09 JST; 4h 59min ago
 Main PID: 1286 (dnsmasq)
   CGroup: /system.slice/dnsmasq.service
           └─1286 /usr/sbin/dnsmasq -k --auth-peer=10.20.0.10

● BGP Peer の確認

Kubernetes側
$ calicoctl get BGPPeer -o wide
NAME        PEERIP      NODE                 ASN
bgppeer00   10.0.0.51   070master.testk.jp   65051
bgppeer01   10.0.0.70   071node01.testk.jp   65400
bgppeer02   10.0.0.70   072node02.testk.jp   65400

router1 kube が Kubernetes側との接続を示す。
# birdc show protocol
BIRD 2.0.4 ready.
Name       Proto      Table      State  Since         Info
kube       BGP        ---        up     12:07:21.852  Established
master     BGP        ---        up     12:08:20.255  Established

router2
# birdc show protocol
BIRD 2.0.4 ready.
Name       Proto      Table      State  Since         Info
master     BGP        ---        up     12:08:19.359  Established

これで事前準備は終わり

#####● 調査開始
調査項目
router1 と router2 で下記を確認

  1. service01 の CLUSTER-IP からサービス名を取得
  2. service01 の サービス名から CLUSTER-IP を取得
  3. service01 の CLUSTER-IP へ curl で HTTPリクエスト
  4. service01 の サービス名 へ curl で HTTPリクエスト
router1
1.
# dig -4 -x 10.20.71.66

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> -4 -x 10.20.71.66
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6203
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;66.71.20.10.in-addr.arpa.	IN	PTR

;; ANSWER SECTION:
66.71.20.10.in-addr.arpa. 30	IN	PTR	service01.apptest.svc.cluster.local.

;; Query time: 1 msec
;; SERVER: 10.20.0.10#53(10.20.0.10)
;; WHEN: 土 10月 19 18:25:44 JST 2019
;; MSG SIZE  rcvd: 126

2.
# dig service01.apptest.svc.cluster.local

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> service01.apptest.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40918
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;service01.apptest.svc.cluster.local. IN	A

;; ANSWER SECTION:
service01.apptest.svc.cluster.local. 30	IN A	10.20.71.66

;; Query time: 0 msec
;; SERVER: 10.20.0.10#53(10.20.0.10)
;; WHEN: 土 10月 19 18:26:22 JST 2019
;; MSG SIZE  rcvd: 115

3.
# curl -I 10.20.71.66
HTTP/1.1 200 OK
Server: nginx/1.15.12
Date: Sat, 19 Oct 2019 09:26:52 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 16 Apr 2019 13:08:19 GMT
Connection: keep-alive
ETag: "5cb5d3c3-264"
Accept-Ranges: bytes

4.
# curl -I service01.apptest.svc.cluster.local
HTTP/1.1 200 OK
Server: nginx/1.15.12
Date: Sat, 19 Oct 2019 09:27:49 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 16 Apr 2019 13:08:19 GMT
Connection: keep-alive
ETag: "5cb5d3c3-264"
Accept-Ranges: bytes

router1 に間してはすべてできました。

router2
1.
# dig -4 -x 10.20.71.66

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> -4 -x 10.20.71.66
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 23998
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;66.71.20.10.in-addr.arpa.	IN	PTR

;; Query time: 0 msec
;; SERVER: 10.0.0.51#53(10.0.0.51)
;; WHEN: 土 10月 19 18:25:29 JST 2019
;; MSG SIZE  rcvd: 53

2.
# dig service01.apptest.svc.cluster.local

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> service01.apptest.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63847
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;service01.apptest.svc.cluster.local. IN	A

;; ANSWER SECTION:
service01.apptest.svc.cluster.local. 30	IN A	10.20.71.66

;; Query time: 0 msec
;; SERVER: 10.0.0.51#53(10.0.0.51)
;; WHEN: 土 10月 19 18:29:33 JST 2019
;; MSG SIZE  rcvd: 115

3.
# curl -I 10.20.71.66
HTTP/1.1 200 OK
Server: nginx/1.15.12
Date: Sat, 19 Oct 2019 09:30:08 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 16 Apr 2019 13:08:19 GMT
Connection: keep-alive
ETag: "5cb5d3c3-264"
Accept-Ranges: bytes

4.
# curl -I service01.apptest.svc.cluster.local
HTTP/1.1 200 OK
Server: nginx/1.15.12
Date: Sat, 19 Oct 2019 09:30:38 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 16 Apr 2019 13:08:19 GMT
Connection: keep-alive
ETag: "5cb5d3c3-264"
Accept-Ranges: bytes

router2 では、1. service01 の CLUSTER-IP からサービス名を取得 でうまくいきません。
しかし、やりたかったことは出来たようです。
router1 に Envoy を追加すれば、もっと面白いかな。

1
2
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
1
2

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?