Help us understand the problem. What is going on with this article?

Kubernetes1.11 + Calico3.1 BGPモード 構築 (kubeadm)

More than 1 year has passed since last update.

はじめに

KubernetesのPod間ネットワークには、様々な通信方法が存在します。これは、KubernetesのCNI Plugin で何を選択するかによって構成が大きく変わります。
今回は比較的有名(?)な CNI Plugin の Calico をオンプレミスで構築するための手順を確認します。

Calicoには、2つのモードが存在します

  • IPIPトンネリングモード
  • BGPモード

IPIPトンネリングモードは、IPパケットをIPパケットでカプセル化するため、通信のオーバーヘッドが気になります。性能の観点からBGPモードでの構成を見ていきます。
BGPモードとという名称ですが、PodNetworkを直接BGPで外部に公開しない場合、かつ、全てのKubernetesクラスタがL2で通信が出来る場合、外部BGPルータは必要ありません。
Service NodePort などPodを公開する方法と併用することで、サーバ側だけでCalicoを構成出来ます。

なお、全クラスタのL2接続については以下を参照ください
https://www.slideshare.net/techblogyahoo/yahoo-japan-meetup-8-71847314

外部BGPルータが必要となる場合は、以下の使用用途となると考えられます

  • 全てのKubernetesNodeがL2で通信出来ない場合、かつ、性能を考慮しIPIPトンネリングを実施したくない場合
  • 外部のNWから直接Podにアクセスする特殊な要件が有る場合 (Serviceを使用してアクセスするのが普通)

つまり、KuberneretesクラスタをL2で接続さえしてあげれば、外部のBGPルータは必要ではなさそうに思います

構成

  • Master1台
  • Node2台

001.png

事前設定

kubeadmを実行する際に、swapが有効だとinstallにFailしてしますので、

swapの無効化を実施します

swapoff -a

編集 (swapがあれば)

vim /etc/fstab

kubeadmインストール 全てのサーバで実施

Dockerをinstall

yum install -y docker
systemctl enable docker && systemctl start docker

依存関係メモ

===============================================================================================================================================================================================================
 Package                                                     Arch                                 Version                                                           Repository                            Size
===============================================================================================================================================================================================================
Installing:
 docker                                                      x86_64                               2:1.13.1-63.git94f4240.el7.centos                                 extras                                16 M
Installing for dependencies:
 container-selinux                                           noarch                               2:2.55-1.el7                                                      extras                                34 k
 container-storage-setup                                     noarch                               0.9.0-1.rhel75.gite0997c3.el7                                     extras                                33 k
 device-mapper-event                                         x86_64                               7:1.02.146-4.el7                                                  base                                 185 k
 device-mapper-event-libs                                    x86_64                               7:1.02.146-4.el7                                                  base                                 184 k
 device-mapper-persistent-data                               x86_64                               0.7.3-3.el7                                                       base                                 405 k
 docker-client                                               x86_64                               2:1.13.1-63.git94f4240.el7.centos                                 extras                               3.8 M
 docker-common                                               x86_64                               2:1.13.1-63.git94f4240.el7.centos                                 extras                                88 k
 lvm2                                                        x86_64                               7:2.02.177-4.el7                                                  base                                 1.3 M
 lvm2-libs                                                   x86_64                               7:2.02.177-4.el7                                                  base                                 1.0 M
 oci-register-machine                                        x86_64                               1:0-6.git2b44233.el7                                              extras                               1.1 M
 oci-systemd-hook                                            x86_64                               1:0.1.15-2.gitc04483d.el7                                         extras                                33 k
 oci-umount                                                  x86_64                               2:2.3.3-3.gite3c9055.el7                                          extras                                32 k
 skopeo-containers                                           x86_64                               1:0.1.29-3.dev.git7add6fc.el7.0                                   extras                                15 k
 yajl                                                        x86_64                               2.0.4-4.el7                                                       base                                  39 k

Transaction Summary
===============================================================================================================================================================================================================

repository設定

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

kubelet, kubeadm, kubectlをinstall

yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet

依存関係メモ

===============================================================================================================================================================================================================
 Package                                              Arch                                         Version                                              Repository                                        Size
===============================================================================================================================================================================================================
Installing:
 kubeadm                                              x86_64                                       1.11.0-0                                             kubernetes                                       7.5 M
 kubectl                                              x86_64                                       1.11.0-0                                             kubernetes                                       7.5 M
 kubelet                                              x86_64                                       1.11.0-0                                             kubernetes                                        18 M
Installing for dependencies:
 cri-tools                                            x86_64                                       1.11.0-0                                             kubernetes                                       4.2 M
 ebtables                                             x86_64                                       2.0.10-16.el7                                        base                                             123 k
 kubernetes-cni                                       x86_64                                       0.6.0-0                                              kubernetes                                       8.6 M
 socat                                                x86_64                                       1.7.3.2-2.el7                                        base                                             290 k

Transaction Summary
===============================================================================================================================================================================================================

iptablesがバイパスしているため、正しくルーティングされない問題が、いくつか報告されているらしい。
以下のsysctlで回避

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

実行例

[root@ntw-k8s-master01 ~]# sysctl --system
* Applying /usr/lib/sysctl.d/00-system.conf ...
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /usr/lib/sysctl.d/99-docker.conf ...
fs.may_detach_mounts = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
* Applying /etc/sysctl.conf ...

dockerが使用している cgourp driver と kubeletが認識している cgoup driver が一致しているか確認する

Dockerは、 systemd を使用している

[root@ntw-k8s-master01 ~]# docker info | grep -i cgroup
  WARNING: You're not using the default seccomp profile
Cgroup Driver: systemd

なお、この時点で kubelet は起動していないが問題はない。kubeadm init 後に起動する。

Snapshot取得

念のためKVMなどでSnapshotを取得しておく

virsh snapshot-create-as calico-k8s-master01 001_before_kubeadm
virsh snapshot-create-as calico-k8s-node01 001_before_kubeadm
virsh snapshot-create-as calico-k8s-node02 001_before_kubeadm

Master で kubeadm を構成

mkdir /root/kubeadm/
cat <<'EOF' > /root/kubeadm/config.yaml
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
networking:
  serviceSubnet: '10.1.0.0/22'
  podSubnet: '10.1.4.0/22'
tokenTTL: '0'
EOF

実行時間は約2分くらい

kubeadm init --config /root/kubeadm/config.yaml

実行例

[root@calico-k8s-master01 ~]# kubeadm init --config /root/kubeadm/config.yaml
I0714 11:43:56.011986    1529 feature_gate.go:230] feature gates: &{map[]}
[init] using Kubernetes version: v1.11.0
[preflight] running pre-flight checks
I0714 11:43:56.040648    1529 kernel_validator.go:81] Validating kernel version
I0714 11:43:56.041097    1529 kernel_validator.go:96] Validating kernel config
        [WARNING Hostname]: hostname "calico-k8s-master01.maas" could not be reached
        [WARNING Hostname]: hostname "calico-k8s-master01.maas" lookup calico-k8s-master01.maas on 8.8.8.8:53: no such host
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [calico-k8s-master01.maas kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.1.0.1 10.44.194.67]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [calico-k8s-master01.maas localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [calico-k8s-master01.maas localhost] and IPs [10.44.194.67 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests" 
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 41.001936 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node calico-k8s-master01.maas as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node calico-k8s-master01.maas as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "calico-k8s-master01.maas" as an annotation
[bootstraptoken] using token: g548mi.nt8u59mgdg3ro7ve
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 10.44.194.67:6443 --token g7ahjt.l7lcgd9i1pfxcbmk --discovery-token-ca-cert-hash sha256:73fda0cd56b925617ec18f0bd3933594afee2398e2b6811fdfe60b3dc5192826

[root@calico-k8s-master01 ~]# 

kubectl の実行環境設定

環境変数設定

rootユーザでkubectlを実行するために環境変数を指定します

export KUBECONFIG=/etc/kubernetes/admin.conf

KubernetesクラスタのNode状態確認

[root@calico-k8s-master01 kubeadm(default kubernetes-admin)]# kubectl get nodes -o wide
NAME                       STATUS     ROLES     AGE       VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
calico-k8s-master01.maas   NotReady   master    36m       v1.11.0   192.168.204.100   <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://1.13.1

master に join するための kubeadm の トークン確認
Defaultでは、Expire が 24時間制限となっているが、無制限としている。

[root@calico-k8s-master01 ~]# kubeadm token list
TOKEN                     TTL         EXPIRES   USAGES                   DESCRIPTION   EXTRA GROUPS
g7ahjt.l7lcgd9i1pfxcbmk   <forever>   <never>   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token

bash_completionの有効化

kubectlのサブコマンドの自動補完を有効にするために、以下のパッケージを導入する

yum install -y bash-completion

依存関係メモ

===============================================================================================================================================================================================================
 Package                                                 Arch                                           Version                                             Repository                                    Size
===============================================================================================================================================================================================================
Installing:
 bash-completion                                         noarch                                         1:2.1-6.el7                                         base                                          85 k

Transaction Summary
===============================================================================================================================================================================================================

bashrcに以下の行を追記

echo "source <(kubectl completion bash)" >> ~/.bashrc

一度ターミナルをexitし、再度ログインすると、kubectlのcompletionが有効になる

kubectx と kubens のインストール

Kubernetesクラスタを管理する際に便利な物を導入する。

  • kubextc : kubectlの実行クラスタを変更する際に、楽に変更可能とするためのもの
  • kubens : kubectlで namespace を変更する際に、楽に変更可能とするためのもの
sudo git clone https://github.com/ahmetb/kubectx /opt/kubectx
sudo ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx
sudo ln -s /opt/kubectx/kubens /usr/local/bin/kubens

kube-prompt-bash を install

bash-promt に kubernetes クラスタや namespace を表示するツールを導入

cd ~
git clone https://github.com/Sugi275/kube-prompt-bash.git
echo "source ~/kube-prompt-bash/kube-prompt-bash.sh" >> ~/.bashrc
echo 'export PS1='\''[\u@\h \W($(kube_prompt))]\$ '\' >> ~/.bashrc

kubectlのconfigをhomedirに作成

kubeadmで自動作成された config を ホームディレクトリにコピーします

mkdir ~/.kube
cp -p /etc/kubernetes/admin.conf ~/.kube/config

環境変数に、KUBECONFIG=/etc/kubernetes/admin.conf と指定されており、これを変更する

export KUBECONFIG=$HOME/.kube/config

.bash_profileに環境変数を指定する

cat <<'EOF' > /root/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# add for kubernetes
export KUBECONFIG=$HOME/.kube/config
EOF

コピーしてきたconfigにnamespaceの指定を追加する

vim ~/.kube/config

- context:
    cluster: kubernetes
    namespace: default  <-------------追加
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes

kubectl config で NAMESPACE に default が表示されたことを確認

[root@calico-k8s-master01 kubeadm(default kubernetes-admin)]# kubectl config get-contexts 
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin   default

memo この段階では、DNSコンテナはpendingとなっているが正しいです

[root@calico-k8s-master01 kubeadm(default kubernetes-admin)]# kubectl get pods -o wide --all-namespaces
NAMESPACE     NAME                                               READY     STATUS    RESTARTS   AGE       IP                NODE
kube-system   coredns-78fcdf6894-6r6h6                           0/1       Pending   0          35m       <none>            <none>
kube-system   coredns-78fcdf6894-fvlzv                           0/1       Pending   0          35m       <none>            <none>
kube-system   etcd-calico-k8s-master01.maas                      1/1       Running   0          34m       192.168.204.100   calico-k8s-master01.maas
kube-system   kube-apiserver-calico-k8s-master01.maas            1/1       Running   0          34m       192.168.204.100   calico-k8s-master01.maas
kube-system   kube-controller-manager-calico-k8s-master01.maas   1/1       Running   0          34m       192.168.204.100   calico-k8s-master01.maas
kube-system   kube-proxy-jn546                                   1/1       Running   0          35m       192.168.204.100   calico-k8s-master01.maas
kube-system   kube-scheduler-calico-k8s-master01.maas            1/1       Running   0          34m       192.168.204.100   calico-k8s-master01.maas

Calicoのinstall

wget で git 上に公開されている Calico の manifest ファイルをダウンロードします

cd /root/kubeadm
wget https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubeadm/1.7/calico.yaml

以下のようにファイルを編集します

cp -p calico.yaml{,.org}
vim calico.yaml

編集前

kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # The location of your etcd cluster.  This uses the Service clusterIP defined below.
  etcd_endpoints: "http://10.96.232.136:6666"

  # Configure the Calico backend to use.
  calico_backend: "bird"

  # The CNI network configuration to install on each node.
  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.0",
      "plugins": [
        {
          "type": "calico",
          "etcd_endpoints": "__ETCD_ENDPOINTS__",
          "log_level": "info",
          "mtu": 1500,
          "ipam": {
              "type": "calico-ipam"
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        }
      ]
    }


------ snip ------
            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            - name: CALICO_IPV4POOL_CIDR
              value: "192.168.0.0/16"
            - name: CALICO_IPV4POOL_IPIP
              value: "Always"
------ snip ------

# This manifest installs the Service which gets traffic to the Calico
# etcd.
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: calico-etcd
  name: calico-etcd
  namespace: kube-system
spec:
  # Select the calico-etcd pod running on the master.
  selector:
    k8s-app: calico-etcd
  # This ClusterIP needs to be known in advance, since we cannot rely
  # on DNS to get access to etcd.
  clusterIP: 10.96.232.136
  ports:
    - port: 6666

編集後

  • PodNetworkの変更 (動作検証のため短めなRangeにしてみる。本番環境では/16といった大きな範囲にするべき)
  • Calicoで必要としているEtcdEndpointのClusterIPを、上記の変更に合わせて変更
  • IPIPモードをOFFにする
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # The location of your etcd cluster.  This uses the Service clusterIP defined below.
  etcd_endpoints: "http://10.1.3.250:6666"              <---------------- here

  # Configure the Calico backend to use.
  calico_backend: "bird"

  # The CNI network configuration to install on each node.
  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.0",
      "plugins": [
        {
          "type": "calico",
          "etcd_endpoints": "__ETCD_ENDPOINTS__",
          "log_level": "info",
          "mtu": 1500,
          "ipam": {
              "type": "calico-ipam"
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        }
      ]
    }


------ snip ------
            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            - name: CALICO_IPV4POOL_CIDR
              value: "10.1.4.0/22"             <---------------- here
            - name: CALICO_IPV4POOL_IPIP
              value: "off"             <---------------- here

------ snip ------

# This manifest installs the Service which gets traffic to the Calico
# etcd.
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: calico-etcd
  name: calico-etcd
  namespace: kube-system
spec:
  # Select the calico-etcd pod running on the master.
  selector:
    k8s-app: calico-etcd
  # This ClusterIP needs to be known in advance, since we cannot rely
  # on DNS to get access to etcd.
  clusterIP: 10.1.3.250             <---------------- here
  ports:
    - port: 6666

差分を確認します

[root@calico-k8s-master01 kubeadm(default kubernetes-admin)]# diff -u calico.yaml.org calico.yaml
--- calico.yaml.org     2018-07-12 09:14:13.000000000 +0900
+++ calico.yaml 2018-07-14 12:28:20.865564746 +0900
@@ -13,7 +13,7 @@
   namespace: kube-system
 data:
   # The location of your etcd cluster.  This uses the Service clusterIP defined below.
-  etcd_endpoints: "http://10.96.232.136:6666"
+  etcd_endpoints: "http://10.1.3.250:6666"

   # Configure the Calico backend to use.
   calico_backend: "bird"
@@ -129,7 +129,7 @@
     k8s-app: calico-etcd
   # This ClusterIP needs to be known in advance, since we cannot rely
   # on DNS to get access to etcd.
-  clusterIP: 10.96.232.136
+  clusterIP: 10.1.3.250
   ports:
     - port: 6666

@@ -214,9 +214,9 @@
             # chosen from this range. Changing this value after installation will have
             # no effect. This should fall within `--cluster-cidr`.
             - name: CALICO_IPV4POOL_CIDR
-              value: "192.168.0.0/16"
+              value: "10.1.4.0/22"
             - name: CALICO_IPV4POOL_IPIP
-              value: "Always"
+              value: "off"
             # Disable IPv6 on Kubernetes.
             - name: FELIX_IPV6SUPPORT
               value: "false"

manifest ファイルから apply を実行します

kubectl apply -f /root/kubeadm/calico.yaml

kube-dnsやCalico用のetcdPodが作成されています

[root@calico-k8s-master01 kubeadm(default kubernetes-admin)]# kubectl get svc --all-namespaces
NAMESPACE     NAME          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
default       kubernetes    ClusterIP   10.1.0.1     <none>        443/TCP         44m
kube-system   calico-etcd   ClusterIP   10.1.3.250   <none>        6666/TCP        11s
kube-system   kube-dns      ClusterIP   10.1.0.10    <none>        53/UDP,53/TCP   44m

Podも正常に作成されています

[root@calico-k8s-master01 kubeadm]# kubectl get pods -o wide --all-namespaces
NAMESPACE     NAME                                               READY     STATUS    RESTARTS   AGE       IP                NODE
kube-system   calico-etcd-jc5st                                  1/1       Running   0          1m        192.168.204.100   calico-k8s-master01.maas
kube-system   calico-kube-controllers-84fd4db7cd-pxndj           1/1       Running   0          1m        192.168.204.100   calico-k8s-master01.maas
kube-system   calico-node-cqcvb                                  2/2       Running   0          1m        192.168.204.100   calico-k8s-master01.maas
kube-system   coredns-78fcdf6894-5m2qn                           1/1       Running   0          5m        10.1.6.64         calico-k8s-master01.maas
kube-system   coredns-78fcdf6894-k5tf4                           1/1       Running   0          5m        10.1.6.65         calico-k8s-master01.maas
kube-system   etcd-calico-k8s-master01.maas                      1/1       Running   0          4m        192.168.204.100   calico-k8s-master01.maas
kube-system   kube-apiserver-calico-k8s-master01.maas            1/1       Running   0          4m        192.168.204.100   calico-k8s-master01.maas
kube-system   kube-controller-manager-calico-k8s-master01.maas   1/1       Running   0          4m        192.168.204.100   calico-k8s-master01.maas
kube-system   kube-proxy-pvnk8                                   1/1       Running   0          5m        192.168.204.100   calico-k8s-master01.maas
kube-system   kube-scheduler-calico-k8s-master01.maas            1/1       Running   0          4m        192.168.204.100   calico-k8s-master01.maas

Node で kubeadm を構成

Masterに参加するように、kubeadmコマンドを実行する

Master側でkubeadmを実行した際に出力された最後の文字列から引用

kubeadm join 10.44.194.67:6443 --token g7ahjt.l7lcgd9i1pfxcbmk --discovery-token-ca-cert-hash sha256:73fda0cd56b925617ec18f0bd3933594afee2398e2b6811fdfe60b3dc5192826

実行例

2秒ほどで終了する

[root@calico-k8s-node01 ~]# kubeadm join 10.44.194.67:6443 --token g7ahjt.l7lcgd9i1pfxcbmk --discovery-token-ca-cert-hash sha256:73fda0cd56b925617ec18f0bd3933594afee2398e2b6811fdfe60b3dc5192826
[preflight] running pre-flight checks
        [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{} ip_vs_rr:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

I0714 12:45:06.349270    5362 kernel_validator.go:81] Validating kernel version
I0714 12:45:06.349574    5362 kernel_validator.go:96] Validating kernel config
        [WARNING Hostname]: hostname "calico-k8s-node01.maas" could not be reached
        [WARNING Hostname]: hostname "calico-k8s-node01.maas" lookup calico-k8s-node01.maas on 8.8.8.8:53: no such host
[discovery] Trying to connect to API Server "10.44.194.67:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.44.194.67:6443"
[discovery] Requesting info from "https://10.44.194.67:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.44.194.67:6443"
[discovery] Successfully established connection with API Server "10.44.194.67:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[preflight] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "calico-k8s-node01.maas" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

nodeサーバでもコンテナが実行されている

[root@calico-k8s-node01 ~]# docker ps
CONTAINER ID        IMAGE                                                                                                 COMMAND                  CREATED              STATUS              PORTS               NAMES
d4eb932f7b5a        quay.io/calico/cni@sha256:ed172c28bc193bb09bce6be6ed7dc6bfc85118d55e61d263cee8bbb0fd464a9d            "/install-cni.sh"        About a minute ago   Up About a minute                       k8s_install-cni_calico-node-4wbfq_kube-system_4d516c63-8718-11e8-ae94-525400139e30_0
0d87c5f45c36        quay.io/calico/node@sha256:a35541153f7695b38afada46843c64a2c546548cd8c171f402621736c6cf3f0b           "start_runit"            About a minute ago   Up About a minute                       k8s_calico-node_calico-node-4wbfq_kube-system_4d516c63-8718-11e8-ae94-525400139e30_0
7ccd9d22a4ff        k8s.gcr.io/kube-proxy-amd64@sha256:3c908257f494b60c0913eae6db3d35fa99825d487b2bcf89eed0a7d8e34c1539   "/usr/local/bin/ku..."   2 minutes ago        Up 2 minutes                            k8s_kube-proxy_kube-proxy-rznnx_kube-system_4d533942-8718-11e8-ae94-525400139e30_0
36544a240048        k8s.gcr.io/pause:3.1                                                                                  "/pause"                 2 minutes ago        Up 2 minutes                            k8s_POD_kube-proxy-rznnx_kube-system_4d533942-8718-11e8-ae94-525400139e30_0
4da65b1c380c        k8s.gcr.io/pause:3.1                                                                                  "/pause"                 2 minutes ago        Up 2 minutes                            k8s_POD_calico-node-4wbfq_kube-system_4d516c63-8718-11e8-ae94-525400139e30_0

ステータス確認

[root@calico-k8s-master01 kubeadm]# kubectl get nodes -o wide
NAME                       STATUS    ROLES     AGE       VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
calico-k8s-master01.maas   Ready     master    11m       v1.11.0   192.168.204.100   <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://1.13.1
calico-k8s-node01.maas     Ready     <none>    2m        v1.11.0   192.168.204.101   <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://1.13.1

pods確認

[root@calico-k8s-master01 kubeadm]# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                               READY     STATUS    RESTARTS   AGE       IP                NODE
kube-system   calico-etcd-jc5st                                  1/1       Running   0          7m        192.168.204.100   calico-k8s-master01.maas
kube-system   calico-kube-controllers-84fd4db7cd-pxndj           1/1       Running   0          7m        192.168.204.100   calico-k8s-master01.maas
kube-system   calico-node-4wbfq                                  2/2       Running   0          1m        192.168.204.101   calico-k8s-node01.maas
kube-system   calico-node-cqcvb                                  2/2       Running   0          7m        192.168.204.100   calico-k8s-master01.maas
kube-system   coredns-78fcdf6894-5m2qn                           1/1       Running   0          10m       10.1.6.64         calico-k8s-master01.maas
kube-system   coredns-78fcdf6894-k5tf4                           1/1       Running   0          10m       10.1.6.65         calico-k8s-master01.maas
kube-system   etcd-calico-k8s-master01.maas                      1/1       Running   0          10m       192.168.204.100   calico-k8s-master01.maas
kube-system   kube-apiserver-calico-k8s-master01.maas            1/1       Running   0          10m       192.168.204.100   calico-k8s-master01.maas
kube-system   kube-controller-manager-calico-k8s-master01.maas   1/1       Running   0          10m       192.168.204.100   calico-k8s-master01.maas
kube-system   kube-proxy-pvnk8                                   1/1       Running   0          10m       192.168.204.100   calico-k8s-master01.maas
kube-system   kube-proxy-rznnx                                   1/1       Running   0          1m        192.168.204.101   calico-k8s-node01.maas
kube-system   kube-scheduler-calico-k8s-master01.maas            1/1       Running   0          10m       192.168.204.100   calico-k8s-master01.maas

他のNodeが有る場合は、同様に追加します。
私の環境ではNodeを2台としているので、以下のようになりました。

[root@calico-k8s-master01 kubeadm]# kubectl get nodes -o wide
NAME                       STATUS    ROLES     AGE       VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
calico-k8s-master01.maas   Ready     master    14m       v1.11.0   192.168.204.100   <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://1.13.1
calico-k8s-node01.maas     Ready     <none>    5m        v1.11.0   192.168.204.101   <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://1.13.1
calico-k8s-node02.maas     Ready     <none>    1m        v1.11.0   192.168.204.102   <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://1.13.1
[root@calico-k8s-master01 kubeadm]# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                               READY     STATUS    RESTARTS   AGE       IP                NODE
kube-system   calico-etcd-jc5st                                  1/1       Running   0          11m       192.168.204.100   calico-k8s-master01.maas
kube-system   calico-kube-controllers-84fd4db7cd-pxndj           1/1       Running   0          11m       192.168.204.100   calico-k8s-master01.maas
kube-system   calico-node-4wbfq                                  2/2       Running   0          6m        192.168.204.101   calico-k8s-node01.maas
kube-system   calico-node-8zwjd                                  2/2       Running   0          2m        192.168.204.102   calico-k8s-node02.maas
kube-system   calico-node-cqcvb                                  2/2       Running   0          11m       192.168.204.100   calico-k8s-master01.maas
kube-system   coredns-78fcdf6894-5m2qn                           1/1       Running   0          15m       10.1.6.64         calico-k8s-master01.maas
kube-system   coredns-78fcdf6894-k5tf4                           1/1       Running   0          15m       10.1.6.65         calico-k8s-master01.maas
kube-system   etcd-calico-k8s-master01.maas                      1/1       Running   0          14m       192.168.204.100   calico-k8s-master01.maas
kube-system   kube-apiserver-calico-k8s-master01.maas            1/1       Running   0          14m       192.168.204.100   calico-k8s-master01.maas
kube-system   kube-controller-manager-calico-k8s-master01.maas   1/1       Running   0          14m       192.168.204.100   calico-k8s-master01.maas
kube-system   kube-proxy-mrj59                                   1/1       Running   0          2m        192.168.204.102   calico-k8s-node02.maas
kube-system   kube-proxy-pvnk8                                   1/1       Running   0          15m       192.168.204.100   calico-k8s-master01.maas
kube-system   kube-proxy-rznnx                                   1/1       Running   0          6m        192.168.204.101   calico-k8s-node01.maas
kube-system   kube-scheduler-calico-k8s-master01.maas            1/1       Running   0          14m       192.168.204.100   calico-k8s-master01.maas
[root@calico-k8s-master01 kubeadm]# kubectl get svc --all-namespaces -o wide
NAMESPACE     NAME          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE       SELECTOR
default       kubernetes    ClusterIP   10.1.0.1     <none>        443/TCP         15m       <none>
kube-system   calico-etcd   ClusterIP   10.1.3.250   <none>        6666/TCP        11m       k8s-app=calico-etcd
kube-system   kube-dns      ClusterIP   10.1.0.10    <none>        53/UDP,53/TCP   15m       k8s-app=kube-dns

calicoctlのInstall

KubernetesのPodとしてDeployする

cd /root/calicoctl
wget https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/calicoctl.yaml
kubectl apply -f calicoctl.yaml

作成したPodでshを立ち上げる

kubectl exec -it -n kube-system calicoctl /bin/sh

calicoctlコマンドで各種確認

~ # calicoctl get nodes -o wide
NAME                       ASN         IPV4               IPV6   
calico-k8s-master01.maas   (unknown)   192.168.217.1/24          
calico-k8s-node01.maas     (unknown)   192.168.217.2/24          
calico-k8s-node02.maas     (unknown)   192.168.217.3/24          

~ # calicoctl get bgpPeer -o wide
NAME   PEERIP   NODE   ASN   

~ # calicoctl get ippool -o wide
NAME                  CIDR          NAT    IPIPMODE   DISABLED   
default-ipv4-ippool   10.1.4.0/22   true   Never      false     

~ # calicoctl get bgpConfiguration -o wide
NAME   LOGSEVERITY   MESHENABLED   ASNUMBER   


動作確認

nginxを作成し、NodePortからアクセス

mkdir /root/manifests/
cat <<'EOF' > /root/manifests/nginx_test.yaml
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-test
spec:
  selector:
    matchLabels:
      app: nginx-test
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx-test
    spec:
      containers:
      - name: master
        image: nginx
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 80
      restartPolicy: Always
EOF
kubectl apply -f /root/manifests/nginx_test.yaml

作成中

[root@calico-k8s-master01 ~(default kubernetes-admin)]# kubectl get pods -o wide
NAME                          READY     STATUS              RESTARTS   AGE       IP        NODE
nginx-test-58586b9f9c-xnk24   0/1       ContainerCreating   0          11s       <none>    calico-k8s-node01.maas
nginx-test-58586b9f9c-xsv9k   0/1       ContainerCreating   0          11s       <none>    calico-k8s-node02.maas

作成完了

[root@calico-k8s-master01 ~(default kubernetes-admin)]# kubectl get pods -o wide
NAME                          READY     STATUS    RESTARTS   AGE       IP         NODE
nginx-test-58586b9f9c-xnk24   1/1       Running   0          2m        10.1.7.0   calico-k8s-node01.maas
nginx-test-58586b9f9c-xsv9k   1/1       Running   0          2m        10.1.5.0   calico-k8s-node02.maas

NodePortサービスを作成

cat <<'EOF' > /root/manifests/nginx_nodeport.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: nginx-test
  name: nginx-test
spec:
  type: NodePort
  ports:
  - port: 8080
    targetPort: 80
    nodePort: 32003
  selector:
    app: nginx-test
EOF
kubectl apply -f /root/manifests/nginx_nodeport.yaml

アクセステスト

http://10.44.194.67:32003/

→ Welcome to nginx! と表示

CentOSPodを作成し、Node間・NameSpace間通信テスト

demo1 namcespaceを作成

kubectl create namespace demo1

切り替え

[root@calico-k8s-master01 ~(default kubernetes-admin)]# kubens demo1
Context "kubernetes-admin@kubernetes" modified.
Active namespace is "demo1".
[root@calico-k8s-master01 ~(demo1 kubernetes-admin)]# 
mkdir /root/manifests/
cat <<'EOF' > /root/manifests/centos_test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: centos-deployment
spec:
  selector:
    matchLabels:
      app: centos
  replicas: 2 # tells deployment to run 2 pods matching the template
  template: # create pods using pod definition in this template
    metadata:
      # unlike pod-nginx.yaml, the name is not included in the meta data as a unique name is
      # generated from the deployment name
      labels:
        app: centos
    spec:
      containers:
      - name: centos
        image: centos:latest
        command: [ "sleep", "360000000" ]
EOF
kubectl apply -f /root/manifests/centos_test.yaml

作成完了

[root@calico-k8s-master01 ~(demo1 kubernetes-admin)]# kubectl get pods -o wide
NAME                                 READY     STATUS    RESTARTS   AGE       IP         NODE
centos-deployment-7d7d7bcb56-td2c5   1/1       Running   0          6s        10.1.5.2   calico-k8s-node02.maas
centos-deployment-7d7d7bcb56-w6pc8   1/1       Running   0          6s        10.1.7.2   calico-k8s-node01.maas

centosでbashを立ち上げる

# kubectl exec -it centos-deployment-7d7d7bcb56-td2c5 bash
[root@centos-deployment-7d7d7bcb56-td2c5 /]#

解析のため以下パッケージを入れる

yum install -y iproute vim tcpdump

別のNodeで稼働しているコンテナへPingが届く

[root@centos-deployment-7d7d7bcb56-td2c5 /]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 6e:21:21:da:49:5a brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.1.5.2/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6c21:21ff:feda:495a/64 scope link 
       valid_lft forever preferred_lft forever
[root@centos-deployment-7d7d7bcb56-td2c5 /]# 
[root@centos-deployment-7d7d7bcb56-td2c5 /]# ping 10.1.7.2
PING 10.1.7.2 (10.1.7.2) 56(84) bytes of data.
64 bytes from 10.1.7.2: icmp_seq=1 ttl=62 time=0.298 ms
64 bytes from 10.1.7.2: icmp_seq=2 ttl=62 time=0.393 ms

また、defaultではNetworkPolicyを設定していないので、別Nacespaceで動作しているPodへアクセス可能

[root@centos-deployment-7d7d7bcb56-td2c5 /]# curl http://10.1.7.0/ 
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

参考URL

https://kubernetes.io/docs/tasks/tools/install-kubeadm/

https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/

https://github.com/kubernetes/kubernetes/blob/release-1.10/pkg/kubelet/apis/kubeletconfig/v1beta1/types.go

https://www.kaitoy.xyz/2018/05/05/kubernetes-kubelet-config-and-pod-sec-policy/

Calico系

https://docs.projectcalico.org/v3.1/getting-started/kubernetes/

https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/calico

https://docs.projectcalico.org/v3.1/usage/configuration/bgp

Calico NW Customize

https://www.n-novice.com/entry/2018/05/06/130151#Calico%E5%AE%9F%E8%A3%85%E8%A6%81%E4%BB%B6

Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
Comments
Sign up for free and join this conversation.
If you already have a Qiita account
Why do not you register as a user and use Qiita more conveniently?
You need to log in to use this function. Qiita can be used more conveniently after logging in.
You seem to be reading articles frequently this month. Qiita can be used more conveniently after logging in.
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away