はじめに
Fedora CoreOSでAWSにKubernetesを立てる。
- Fedora CoreOS(Stable:34.20210518.3.0)
- Kubernetes v1.21.3(1 master、nodeなしの構成。RBAC)
- CRIランタイム:cri-o
- etcd3(podmanで駆動。tls secured)
- コンテナネットワーク:Flannel
- CoreDNS
- EC2インスタンス:t3a.small(スポットインスタンス)
- zram
AWSでインスタンスを作成
あらかじめ、Elastic IP アドレスを1つ割り当てておきます。
リージョンはap-northeast-1を前提としています。
次のyamlを編集
- ssh_authorized_keysを置き換える
variant: fcos
version: 1.3.0
passwd:
users:
- name: core
ssh_authorized_keys:
- "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABA・・・DrOtidE4JlTyG4XOMDLJ core@localhost"
systemd:
units:
- name: podman.service
enabled: true
- name: etcd3.service
enabled: true
contents: |
[Unit]
Description=etcd3
After=network-online.target
Wants=network-online.target
[Service]
ExecStartPre=mkdir -p /var/lib/fcosaws/etcd
ExecStartPre=-/bin/podman kill etcd3
ExecStartPre=-/bin/podman rm etcd3
ExecStart=/bin/podman run \
-p 2379:2379 \
-p 2380:2380 \
-v /var/lib/fcosaws/etcd:/var/lib/fcosaws/etcd:z \
-v /var/lib/fcosaws/pki:/var/lib/fcosaws/pki:z \
--name=etcd3 \
gcr.io/etcd-development/etcd:v3.4.13 \
/usr/local/bin/etcd \
--data-dir /var/lib/fcosaws/etcd \
--name infra1 \
--client-cert-auth \
--trusted-ca-file=/var/lib/fcosaws/pki/ca.crt \
--cert-file=/var/lib/fcosaws/pki/apiserver.crt \
--key-file=/var/lib/fcosaws/pki/apiserver.key \
--peer-client-cert-auth \
--peer-trusted-ca-file=/var/lib/fcosaws/pki/ca.crt \
--peer-cert-file=/var/lib/fcosaws/pki/worker.crt \
--peer-key-file=/var/lib/fcosaws/pki/worker.key \
--advertise-client-urls https://10.10.10.11:2379 \
--listen-client-urls https://0.0.0.0:2379 \
--initial-advertise-peer-urls https://10.10.10.11:2380 \
--listen-peer-urls https://0.0.0.0:2380 \
--initial-cluster-token etcd-cluster-1 \
--initial-cluster infra1=https://10.10.10.11:2380 \
--initial-cluster-state new
ExecStop=/bin/podman stop etcd3
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
- name: kubelet2.service
enabled: true
contents: |
[Unit]
Description=kubelet
[Service]
ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
ExecStartPre=/bin/mkdir -p /var/lib/rook
ExecStart=/root/bin/kubelet \
--config=/etc/kubernetes/kubelet-conf.yaml \
--network-plugin=cni \
--cni-conf-dir=/etc/cni/net.d \
--cni-bin-dir=/opt/cni/bin \
--node-ip=10.10.10.11 \
--container-runtime=remote \
--container-runtime-endpoint=/var/run/crio/crio.sock \
--register-node=true \
--hostname-override=ip-10-10-10-11.ap-northeast-1.compute.internal \
--kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml \
--v=4 \
--cloud-provider=external
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
- name: set_node.service
enabled: true
contents: |
[Unit]
Description=set kubernetes node
After=network-online.target
[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/bash /opt/set_node.sh
[Install]
WantedBy=multi-user.target
WantedBy=network-online.target
- name: fstrim.timer
mask: true
- name: fstrim.service
mask: true
storage:
files:
- path: /etc/crictl.yaml
mode: 0644
contents:
inline: |
runtime-endpoint: unix:///var/run/crio/crio.sock
timeout: 2
- path: /etc/systemd/zram-generator.conf
mode: 0644
contents:
inline: |
[zram0]
- path: /opt/set_node.sh
overwrite: true
mode: 0644
contents:
inline: |
if [ -f /var/home/core/initialized.txt ]; then
systemctl enable crio.service
systemctl start crio.service
else
sed -i -e "s?pool 2.fedora.pool.ntp.org iburst?server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4?g" /etc/chrony.conf
sed -i -e "s?#SystemMaxUse=?SystemMaxUse=80M?g" /etc/systemd/journald.conf
cd /var/home/core
curl -OL https://github.com/containernetworking/plugins/releases/download/v0.9.1/cni-plugins-linux-amd64-v0.9.1.tgz
mkdir -p /opt/cni/bin
tar zxf cni-plugins-linux-amd64-v0.9.1.tgz -C /opt/cni/bin
curl -OL https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.21.0/crictl-v1.21.0-linux-amd64.tar.gz
tar zxf crictl-v1.21.0-linux-amd64.tar.gz -C /usr/local/bin
mkdir /etc/dnf/modules.d
cat <<EOF > /etc/dnf/modules.d/cri-o.module
[cri-o]
name=cri-o
stream=1.21
profiles=
state=enabled
EOF
sed -i -z s/enabled=0/enabled=1/ /etc/yum.repos.d/fedora-modular.repo
sed -i -z s/enabled=0/enabled=1/ /etc/yum.repos.d/fedora-updates-modular.repo
rpm-ostree install cri-o
echo "dummy" > /var/home/core/initialized.txt
shutdown -r now
fi
- path: /etc/kubernetes/manifests/kube-apiserver.yaml
overwrite: true
mode: 0755
contents:
inline: |
apiVersion: v1
kind: Pod
metadata:
name: kube-apiserver
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-apiserver
image: k8s.gcr.io/kube-apiserver-amd64:v1.21.3
command:
- kube-apiserver
- --bind-address=0.0.0.0
- --etcd-servers=https://10.10.10.11:2379
- --etcd-cafile=/var/lib/fcosaws/pki/ca.crt
- --etcd-certfile=/var/lib/fcosaws/pki/apiserver.crt
- --etcd-keyfile=/var/lib/fcosaws/pki/apiserver.key
- --allow-privileged=true
- --apiserver-count=1
- --endpoint-reconciler-type=lease
- --service-cluster-ip-range=10.3.0.0/24
- --secure-port=6443
- --advertise-address=10.10.10.11
- --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
- --storage-backend=etcd3
- --tls-cert-file=/var/lib/fcosaws/pki/apiserver.crt
- --tls-private-key-file=/var/lib/fcosaws/pki/apiserver.key
- --client-ca-file=/var/lib/fcosaws/pki/ca.crt
- --service-account-key-file=/var/lib/fcosaws/pki/apiserver.key
- --runtime-config=extensions/v1beta1/networkpolicies=true
- --service-node-port-range=25-32767
- --authorization-mode=RBAC
- --cloud-provider=external
- --service-account-issuer=api
- --service-account-signing-key-file=/var/lib/fcosaws/pki/apiserver.key
ports:
- containerPort: 6443
hostPort: 6443
name: https
- containerPort: 8080
hostPort: 8080
name: local
volumeMounts:
- mountPath: /var/lib/fcosaws/pki
name: ssl-certs-kubernetes
readOnly: true
volumes:
- hostPath:
path: /var/lib/fcosaws/pki
name: ssl-certs-kubernetes
- path: /etc/kubernetes/manifests/kube-controller-manager.yaml
overwrite: true
mode: 0755
contents:
inline: |
apiVersion: v1
kind: Pod
metadata:
name: kube-controller-manager
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-controller-manager
image: k8s.gcr.io/kube-controller-manager-amd64:v1.21.3
command:
- kube-controller-manager
- --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml
- --leader-elect=true
- --service-account-private-key-file=/var/lib/fcosaws/pki/apiserver.key
- --root-ca-file=/var/lib/fcosaws/pki/ca.crt
- --cluster-signing-cert-file=/var/lib/fcosaws/pki/ca.crt
- --cluster-signing-key-file=/var/lib/fcosaws/pki/ca.key
- --cluster-cidr=10.244.0.0/16
- --allocate-node-cidrs=true
- --cloud-provider=external
livenessProbe:
httpGet:
host: 127.0.0.1
path: /healthz
port: 10252
initialDelaySeconds: 15
timeoutSeconds: 15
volumeMounts:
- mountPath: /etc/kubernetes/worker-kubeconfig.yaml
name: "kubeconfig"
readOnly: true
- mountPath: /var/lib/fcosaws/pki
name: ssl-certs-kubernetes
readOnly: true
volumes:
- name: "kubeconfig"
hostPath:
path: "/etc/kubernetes/worker-kubeconfig.yaml"
- hostPath:
path: /var/lib/fcosaws/pki
name: ssl-certs-kubernetes
- path: /etc/kubernetes/manifests/kube-scheduler.yaml
overwrite: true
mode: 0755
contents:
inline: |
apiVersion: v1
kind: Pod
metadata:
name: kube-scheduler
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-scheduler
image: k8s.gcr.io/kube-scheduler-amd64:v1.21.3
command:
- kube-scheduler
- --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml
- --leader-elect=true
livenessProbe:
httpGet:
host: 127.0.0.1
path: /healthz
port: 10251
initialDelaySeconds: 15
timeoutSeconds: 15
volumeMounts:
- mountPath: /etc/kubernetes/worker-kubeconfig.yaml
name: "kubeconfig"
readOnly: true
- mountPath: /var/lib/fcosaws/pki
name: "etc-kube-ssl"
readOnly: true
volumes:
- name: "kubeconfig"
hostPath:
path: "/etc/kubernetes/worker-kubeconfig.yaml"
- name: "etc-kube-ssl"
hostPath:
path: "/var/lib/fcosaws/pki"
- path: /etc/kubernetes/manifests/kube-proxy.yaml
overwrite: true
mode: 0755
contents:
inline: |
apiVersion: v1
kind: Pod
metadata:
name: kube-proxy
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-proxy
image: k8s.gcr.io/kube-proxy-amd64:v1.21.3
command:
- kube-proxy
- --hostname-override=ip-10-10-10-11.ap-northeast-1.compute.internal
- --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml
- --proxy-mode=iptables
securityContext:
privileged: true
volumeMounts:
- mountPath: /etc/kubernetes/worker-kubeconfig.yaml
name: "kubeconfig"
readOnly: true
- mountPath: /var/lib/fcosaws/pki
name: "etc-kube-ssl"
readOnly: true
- mountPath: /lib/modules
name: "lib-modules"
readOnly: true
volumes:
- name: "kubeconfig"
hostPath:
path: "/etc/kubernetes/worker-kubeconfig.yaml"
- name: "etc-kube-ssl"
hostPath:
path: "/var/lib/fcosaws/pki"
- name: "lib-modules"
hostPath:
path: "/lib/modules"
- path: /etc/kubernetes/worker-kubeconfig.yaml
overwrite: true
mode: 0755
contents:
inline: |
apiVersion: v1
kind: Config
clusters:
- name: local
cluster:
certificate-authority: /var/lib/fcosaws/pki/ca.crt
server: https://ip-10-10-10-11.ap-northeast-1.compute.internal:6443
users:
- name: kubelet
user:
client-certificate: /var/lib/fcosaws/pki/worker.crt
client-key: /var/lib/fcosaws/pki/worker.key
contexts:
- context:
cluster: local
user: kubelet
name: kubelet-context
current-context: kubelet-context
- path: /etc/kubernetes/kubelet-conf.yaml
overwrite: true
mode: 0755
contents:
inline: |
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: "systemd"
volumePluginDir: "/var/lib/kubelet/volumeplugins"
staticPodPath: "/etc/kubernetes/manifests"
clusterDNS: ["10.3.0.10"]
clusterDomain: "cluster.local"
# Restore default authentication and authorization modes from K8s < 1.9
authentication:
anonymous:
enabled: true # Defaults to false as of 1.10
webhook:
enabled: false # Deafults to true as of 1.10
authorization:
mode: AlwaysAllow # Deafults to webhook as of 1.10
readOnlyPort: 10255 # Used by heapster. Defaults to 0 (disabled) as of 1.10. Needed for metrics.
failSwapOn: false
# kubeletをインストールする
- path: /root/bin/kubelet
mode: 0755
overwrite: true
contents:
source: https://storage.googleapis.com/kubernetes-release/release/v1.21.3/bin/linux/amd64/kubelet
ButaneでYAMLをJSONに変換(次の例はlinuxコマンドでの例)。
$ docker run --interactive --rm -v ${PWD}:/pwd --workdir /pwd quay.io/coreos/butane:release --pretty --strict fcos-aws-1master-crio.yaml > fcos-aws-1master-crio.ign
次の内容でspot-options.jsonを作成しておきます。
{
"MarketType": "spot",
"SpotOptions": {
"MaxPrice": "0.0324",
"SpotInstanceType": "persistent",
"InstanceInterruptionBehavior": "stop"
}
}
次のコマンドを実行。
- subnet-idを置き換える
- key-nameを置き換える
- security-group-idsを置き換える
- iam-instance-profileを置き換える
$ aws ec2 run-instances \
--tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=master},{Key=kubernetes.io/cluster/ucluster,Value=kubernetes.io/cluster/ucluster},{Key=KubernetesCluster,Value=ucluster}]' \
--image-id ami-03894a14941c52b30 \
--count 1 \
--instance-type t3a.small \
--block-device-mappings '[{"DeviceName":"/dev/xvda", "Ebs":{"VolumeSize":13}}]' \
--subnet-id subnet-033e851c267541ce0 \
--key-name keypair2 \
--security-group-ids sg-0e16014e82accb264 \
--instance-market-options file:///path/to/spot-options.json \
--associate-public-ip-address \
--private-ip-address 10.10.10.11 \
--iam-instance-profile Name="demo-profile" \
--user-data file:///path/to/fcos-aws-1master-crio.ign
インスタンスが作成されると、set_node.shに記載のとおり、一度再起動されるので、再起動されるまで待つ。
再起動されたら、
$ aws ec2 modify-instance-attribute --instance-id=$(aws ec2 describe-instances --query 'Reservations[].Instances[].{Stt:State.Name,Name:Tags[?Key==`Name`]|[0].Value,InstId:InstanceId}' | jq '.[] | select(.Stt == "running" and .Name == "master")' | jq '.InstId' | sed s/\"//g) --no-source-dest-check
また、Elastic IP アドレス(${MASTER_IP})をこのインスタンスに関連付けておきます。
KubernetesのためのTLS Assetsを準備
次の手順に従ってRBACに対応するTLS Assetsを生成してください。
(詳しくは次のページを参照。
https://github.com/coreos/coreos-kubernetes/blob/master/Documentation/openssl.md
)
$ openssl genrsa -out ca.key 2048
$ openssl req -x509 -new -nodes -key ca.key -days 10950 -out ca.crt -subj "/CN=kube-ca"
- ${MASTER_IP}をElastic IP アドレスで置き換える
$ echo "[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
DNS.5 = ip-10-10-10-11.ap-northeast-1.compute.internal
IP.1 = 10.3.0.1
IP.2 = 10.10.10.11
IP.3 = ${MASTER_IP}" > openssl.cnf
$ openssl genrsa -out apiserver.key 2048
$ openssl req -new -key apiserver.key -out apiserver.csr -subj "/CN=kube-apiserver" -config openssl.cnf
$ openssl x509 -req -in apiserver.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out apiserver.crt -days 10950 -extensions v3_req -extfile openssl.cnf
$ echo "[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
IP.1 = \$ENV::WORKER_IP" > worker-openssl.cnf
$ openssl genrsa -out kube-worker-1.key 2048
$ WORKER_IP=10.10.10.11 openssl req -new -key kube-worker-1.key -out kube-worker-1.csr -subj "/CN=kube-worker-1/O=system:masters" -config worker-openssl.cnf
$ WORKER_IP=10.10.10.11 openssl x509 -req -in kube-worker-1.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kube-worker-1.crt -days 10950 -extensions v3_req -extfile worker-openssl.cnf
$ openssl genrsa -out admin.key 2048
$ openssl req -new -key admin.key -out admin.csr -subj "/CN=kube-admin/O=system:masters"
$ openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out admin.crt -days 10950
以降の手順では次の8つのキーがtls-assets-fcos-awsディレクトリに生成されていることを仮定します。
- ca.key
- ca.crt
- apiserver.key
- apiserver.crt
- admin.key
- admin.crt
- kube-worker-1.key
- kube-worker-1.crt
生成したキーをインスタンスにコピーします。
$ master=$(aws ec2 describe-instances --query 'Reservations[].Instances[].{Stt:State.Name,Name:Tags[?Key==`Name`]|[0].Value,PublicIp:PublicIpAddress}' | jq '.[] | select(.Stt == "running" and .Name == "master")' | jq '.PublicIp' | sed s/\"//g)
$ scp -r tls-assets-fcos-aws core@$master:/home/core/
$ ssh -l core $master 'sudo rm -rf /var/lib/fcosaws/pki
sudo mkdir -p /var/lib/fcosaws/pki
sudo cp tls-assets-fcos-aws/ca.crt /var/lib/fcosaws/pki/
sudo cp tls-assets-fcos-aws/ca.key /var/lib/fcosaws/pki/
sudo cp tls-assets-fcos-aws/apiserver.crt /var/lib/fcosaws/pki/
sudo cp tls-assets-fcos-aws/apiserver.key /var/lib/fcosaws/pki/
sudo cp tls-assets-fcos-aws/kube-worker-1.crt /var/lib/fcosaws/pki/
sudo cp tls-assets-fcos-aws/kube-worker-1.key /var/lib/fcosaws/pki/
sudo chmod 600 /var/lib/fcosaws/pki/*.key
sudo chown root:root /var/lib/fcosaws/pki/*.key
sudo ln -s /var/lib/fcosaws/pki/kube-worker-1.crt /var/lib/fcosaws/pki/worker.crt
sudo ln -s /var/lib/fcosaws/pki/kube-worker-1.key /var/lib/fcosaws/pki/worker.key
'
$ kubectl label nodes ip-10-10-10-11.ap-northeast-1.compute.internal node-role.kubernetes.io/master=""
etcd3とkubelet2が動作していることを確認
$ ssh -l core ${MASTER_IP}
$ systemctl status etcd3
$ systemctl status kubelet2
$ exit
kubectlをセットアップする
kubectlをダウンロードする。
$ curl -O https://storage.googleapis.com/kubernetes-release/release/v1.21.3/bin/linux/amd64/kubectl
ダウンロードしたら、実行権限を確認し、適切なパスに移動する
$ chmod +x kubectl
$ mv kubectl /usr/local/bin/kubectl
kubectlを設定する。
- ${MASTER_IP}をElastic IP アドレスで置き換える
- Replace ${CA_CERT} with the absolute path to the ca.crt created in previous steps
- Replace ${ADMIN_KEY} with the absolute path to the admin.key created in previous steps
- Replace ${ADMIN_CERT} with the absolute path to the admin.crt created in previous steps
$ kubectl config set-cluster default-cluster --server=https://${MASTER_IP}:6443 --certificate-authority=${CA_CERT}
$ kubectl config set-credentials default-admin --certificate-authority=${CA_CERT} --client-key=${ADMIN_KEY} --client-certificate=${ADMIN_CERT}
$ kubectl config set-context default-system --cluster=default-cluster --user=default-admin
$ kubectl config use-context default-system
kubectlの設定と接続を確認する。
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-10-10-11.ap-northeast-1.compute.internal Ready master 159m v1.21.3
aws-cloud-controller-managerをデプロイする
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-aws/release-1.19/manifests/rbac.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-aws/release-1.19/manifests/aws-cloud-controller-manager-daemonset.yaml
Flannelをデプロイする
$ curl -O https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
$ sed -i -e "s?vxlan?aws-vpc?g" kube-flannel.yml
$ kubectl apply -f kube-flannel.yml
CoreDNSをデプロイする
https://github.com/coredns/deployment/tree/master/kubernetes
に記載の手順でCoreDNSをデプロイする。
$ sudo apt -y install jq
$ curl -O https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed
$ curl -O https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh
$ ./deploy.sh -i 10.3.0.10 | kubectl apply -f -
次のコマンドを実行すると、次の画面のようになるはずです。
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-cloud-controller-manager-x55pm 1/1 Running 0 12m
kube-system coredns-85b78c98f-d6ptr 1/1 Running 0 63m
kube-system kube-apiserver-ip-10-10-10-11.ap-northeast-1.compute.internal 1/1 Running 0 69m
kube-system kube-controller-manager-ip-10-10-10-11.ap-northeast-1.compute.internal 1/1 Running 0 68m
kube-system kube-flannel-ds-zdscm 1/1 Running 0 67m
kube-system kube-proxy-ip-10-10-10-11.ap-northeast-1.compute.internal 1/1 Running 0 69m
kube-system kube-scheduler-ip-10-10-10-11.ap-northeast-1.compute.internal 1/1 Running 0 69m
Podがデプロイできることを確認
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- image: busybox
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
name: busybox
restartPolicy: Always
$ kubectl create -f busybox.yaml
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 3m15s