はじめに

Raspberry Pi 4B で kubernetes クラスタを構築したのですが、ラズパイは構築記事ばかりで運用面などの実例記事は少ないのかなと感じます。アーキテクチャがARMでコンテナイメージが用意されていないものも多いせいでしょうか...
そんな中でのクラスターモニタリングについてのお勉強記事です。

前提環境

H/W
Raspberry Pi 4B
S/W
以下のコマンドでOS(kernel),docker,kubernetesのバージョンが全部わかるみたいですね。

# kubectl get nodes -o wide
NAME    STATUS   ROLES           AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION   CONTAINER-RUNTIME
chino   Ready    control-plane   165d   v1.20.0   10.0.0.1      <none>        Debian GNU/Linux 10 (buster)   5.4.83-v8+       docker://20.10.1
chiya   Ready    worker          165d   v1.20.0   10.0.0.5      <none>        Debian GNU/Linux 10 (buster)   5.4.65-v8+       docker://20.10.1
cocoa   Ready    worker          158d   v1.20.0   10.0.0.2      <none>        Debian GNU/Linux 10 (buster)   5.4.65-v8+       docker://20.10.1
maya    Ready    worker          21d    v1.20.1   10.0.0.7      <none>        Debian GNU/Linux 10 (buster)   5.4.65-v8+       cri-o://1.20.0
megu    Ready    worker          57d    v1.20.1   10.0.0.8      <none>        Debian GNU/Linux 10 (buster)   5.4.83-v8+       cri-o://1.20.0
rize    Ready    worker          158d   v1.20.0   10.0.0.3      <none>        Debian GNU/Linux 10 (buster)   5.4.65-v8+       docker://20.10.1
syaro   Ready    worker          158d   v1.20.0   10.0.0.4      <none>        Debian GNU/Linux 10 (buster)   5.4.65-v8+       docker://20.10.1

構築手順

Goのインストール

ARMアーキテクチャでkubernetesを頑張る場合はGoは必需品になるかと思います。
raspbianにおいても、ちょっと古いですが、aptでインストールできるようです。

# apt install golang
# go version
go version go1.11.6 linux/arm

あと、環境変数GOPATHの設定とPATHへの追加をしておきます。

# cat >> ~/.bashrc << 'EOF'
> export GOPATH=$HOME/go
> export PATH=$PATH:$GOPATH/bin
> EOF
# source ~/.bashrc
# printenv | grep PATH
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/go/bin
GOPATH=/root/go

クラスターモニタリングのデプロイ

最初はkube-state-metricsのARM用のイメージをビルドしてたのですが、調べてる最中にgithubで全部まとめてデプロイできるようにしている方がおりました。神様です。
Cluster Monitoring stack for ARM / X86-64 platforms

# git clone https://github.com/carlosedp/cluster-monitoring
Cloning into 'cluster-monitoring'...
remote: Enumerating objects: 16, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 1398 (delta 2), reused 5 (delta 1), pack-reused 1382
Receiving objects: 100% (1398/1398), 1.04 MiB | 1.16 MiB/s, done.
Resolving deltas: 100% (958/958), done.

# cd cluster-monitoring

# make vendor
rm -rf vendor
/root/go/bin/jb install
GET https://github.com/coreos/kube-prometheus/archive/17989b42aa10b1c6afa07043cb05bcd5ae492284.tar.gz 200
GET https://github.com/coreos/prometheus-operator/archive/e31c69f9b5c6555e0f4a5c1f39d0f03182dd6b41.tar.gz 200
GET https://github.com/ksonnet/ksonnet-lib/archive/0d2f82676817bbf9e4acf6495b2090205f323b9f.tar.gz 200
GET https://github.com/prometheus/node_exporter/archive/08ce3c6dd430deb51798826701a395e460620d60.tar.gz 200
GET https://github.com/prometheus/prometheus/archive/74207c04655e1fd93eea0e9a5d2f31b1cbc4d3d0.tar.gz 200
GET https://github.com/brancz/kubernetes-grafana/archive/57b4365eacda291b82e0d55ba7eec573a8198dda.tar.gz 200
GET https://github.com/coreos/etcd/archive/d8c8f903eee10b8391abaef7758c38b2cd393c55.tar.gz 200
GET https://github.com/kubernetes-monitoring/kubernetes-mixin/archive/b61c5a34051f8f57284a08fe78ad8a45b430252b.tar.gz 200
GET https://github.com/kubernetes/kube-state-metrics/archive/d667979ed55ad1c4db44d331b51d646f5b903aa7.tar.gz 200
GET https://github.com/kubernetes/kube-state-metrics/archive/d667979ed55ad1c4db44d331b51d646f5b903aa7.tar.gz 200
GET https://github.com/grafana/grafonnet-lib/archive/8fb95bd89990e493a8534205ee636bfcb8db67bd.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/881db2241f0c5007c3e831caf34b0c645202b4ab.tar.gz 200

# make deploy
echo "Deploying stack setup manifests..."
Deploying stack setup manifests...
kubectl apply -f ./manifests/setup/
namespace/monitoring created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
service/prometheus-operator created
serviceaccount/prometheus-operator created
echo "Will wait 10 seconds to deploy the additional manifests.."
Will wait 10 seconds to deploy the additional manifests..
sleep 10
kubectl apply -f ./manifests/
alertmanager.monitoring.coreos.com/main created
secret/alertmanager-main created
service/alertmanager-main created
serviceaccount/alertmanager-main created
servicemonitor.monitoring.coreos.com/alertmanager created
secret/grafana-config created
secret/grafana-datasources created
configmap/grafana-dashboard-apiserver created
configmap/grafana-dashboard-cluster-total created
configmap/grafana-dashboard-controller-manager created
configmap/grafana-dashboard-coredns-dashboard created
configmap/grafana-dashboard-k8s-resources-cluster created
configmap/grafana-dashboard-k8s-resources-namespace created
configmap/grafana-dashboard-k8s-resources-node created
configmap/grafana-dashboard-k8s-resources-pod created
configmap/grafana-dashboard-k8s-resources-workload created
configmap/grafana-dashboard-k8s-resources-workloads-namespace created
configmap/grafana-dashboard-kubelet created
configmap/grafana-dashboard-kubernetes-cluster-dashboard created
configmap/grafana-dashboard-namespace-by-pod created
configmap/grafana-dashboard-namespace-by-workload created
configmap/grafana-dashboard-node-cluster-rsrc-use created
configmap/grafana-dashboard-node-rsrc-use created
configmap/grafana-dashboard-nodes created
configmap/grafana-dashboard-persistentvolumesusage created
configmap/grafana-dashboard-pod-total created
configmap/grafana-dashboard-prometheus-dashboard created
configmap/grafana-dashboard-prometheus-remote-write created
configmap/grafana-dashboard-prometheus created
configmap/grafana-dashboard-proxy created
configmap/grafana-dashboard-scheduler created
configmap/grafana-dashboard-statefulset created
configmap/grafana-dashboard-workload-total created
configmap/grafana-dashboards created
deployment.apps/grafana created
service/grafana created
serviceaccount/grafana created
servicemonitor.monitoring.coreos.com/grafana created
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/alertmanager-main created
ingress.extensions/grafana created
ingress.extensions/prometheus-k8s created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
service/kube-state-metrics created
serviceaccount/kube-state-metrics created
servicemonitor.monitoring.coreos.com/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/node-exporter created
clusterrolebinding.rbac.authorization.k8s.io/node-exporter created
daemonset.apps/node-exporter created
service/node-exporter created
serviceaccount/node-exporter created
servicemonitor.monitoring.coreos.com/node-exporter created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created
clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created
clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created
configmap/adapter-config created
deployment.apps/prometheus-adapter created
rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created
service/prometheus-adapter created
serviceaccount/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/prometheus-k8s created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created
service/kube-controller-manager-prometheus-discovery created
service/kube-dns-prometheus-discovery created
service/kube-scheduler-prometheus-discovery created
servicemonitor.monitoring.coreos.com/prometheus-operator created
prometheus.monitoring.coreos.com/k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s-config created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created
service/prometheus-k8s created
serviceaccount/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus created
servicemonitor.monitoring.coreos.com/kube-apiserver created
servicemonitor.monitoring.coreos.com/coredns created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
servicemonitor.monitoring.coreos.com/kube-scheduler created
servicemonitor.monitoring.coreos.com/kubelet created

podはmonitoringというネームスペースにデプロイされます。

# kubectl get pods -n monitoring
NAME                                  READY   STATUS    RESTARTS   AGE
alertmanager-main-0                   2/2     Running   0          18m
grafana-784d46dcb-jjhgj               1/1     Running   0          18m
kube-state-metrics-6cb6df5d4-8w5kl    3/3     Running   0          18m
node-exporter-dksrp                   2/2     Running   0          18m
node-exporter-h5cnc                   2/2     Running   0          18m
node-exporter-jqdpr                   2/2     Running   0          18m
node-exporter-l4vgm                   2/2     Running   0          18m
node-exporter-lmdlz                   2/2     Running   0          18m
node-exporter-pg4ww                   2/2     Running   0          18m
node-exporter-sm4vk                   2/2     Running   0          18m
prometheus-adapter-585b57857b-6tw7k   1/1     Running   0          18m
prometheus-k8s-0                      3/3     Running   1          18m
prometheus-operator-67755f959-sjp7l   2/2     Running   0          19m

サービスは以下のように定義されてました。
私の場合は、prometheus-k8sとgrafanaは利便性を考えてNodePortした方が良いかな。

# kubectl get svc -n monitoring
NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
alertmanager-main       ClusterIP   10.96.231.106   <none>        9093/TCP                     7m5s
alertmanager-operated   ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP   7m5s
grafana                 ClusterIP   10.96.126.125   <none>        3000/TCP                     7m3s
kube-state-metrics      ClusterIP   None            <none>        8443/TCP,9443/TCP            7m3s
node-exporter           ClusterIP   None            <none>        9100/TCP                     7m2s
prometheus-adapter      ClusterIP   10.96.10.168    <none>        443/TCP                      7m2s
prometheus-k8s          ClusterIP   10.96.0.198     <none>        9090/TCP                     6m59s
prometheus-operated     ClusterIP   None            <none>        9090/TCP                     7m
prometheus-operator     ClusterIP   None            <none>        8443/TCP                     40d

manifestsにあるサービスを定義するyamlファイルを更新してapplyしました。
prometheusは30909番、grafanaは30300番にしました。

# cat manifests/prometheus-service.yaml 
apiVersion: v1
kind: Service
metadata:
  labels:
    prometheus: k8s
  name: prometheus-k8s
  namespace: monitoring
spec:
  type: NodePort
  ports:
  - name: web
    port: 9090
    nodePort: 30909
    targetPort: web
  selector:
    app: prometheus
    prometheus: k8s
  sessionAffinity: ClientIP

# kubectl apply -f manifests/prometheus-service.yaml
service/prometheus-k8s configured

# cat manifests/grafana-service.yaml 
apiVersion: v1
kind: Service
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: monitoring
spec:
  type: NodePort
  ports:
  - name: http
    port: 3000
    nodePort: 30300
    targetPort: http
  selector:
    app: grafana

# kubectl apply -f manifests/grafana-service.yaml
service/grafana configured

# kubectl get svc -n monitoring
NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
alertmanager-main       ClusterIP   10.96.231.106   <none>        9093/TCP                     12m
alertmanager-operated   ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP   12m
grafana                 NodePort    10.96.126.125   <none>        3000:30300/TCP               12m
kube-state-metrics      ClusterIP   None            <none>        8443/TCP,9443/TCP            12m
node-exporter           ClusterIP   None            <none>        9100/TCP                     12m
prometheus-adapter      ClusterIP   10.96.10.168    <none>        443/TCP                      12m
prometheus-k8s          NodePort    10.96.0.198     <none>        9090:30909/TCP               12m
prometheus-operated     ClusterIP   None            <none>        9090/TCP                     12m
prometheus-operator     ClusterIP   None            <none>        8443/TCP                     40d

ここまでデプロイできれば、kubectl topでCPUとメモリの使用率が確認できます。

# kubectl top nodes
NAME    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
chino   558m         13%    1456Mi          38%       
chiya   699m         17%    550Mi           15%       
cocoa   282m         7%     2773Mi          73%       
maya    108m         2%     820Mi           21%       
megu    488m         12%    1624Mi          43%       
rize    713m         17%    4648Mi          59%       
syaro   431m         10%    3111Mi          40%

私の環境では、dashboardもデプロイしているので、dashboardにグラフも表示されるようになりました。

Prometheus/Grafana

使い方はkubernetes関連ではいつも参考にさせていただいている高良さんの記事『k8sメトリックスのモニタリングとログ分析について調べたメモ』などを参考に、ただいま勉強中なので詳しく書けませんが、以下のようにラズパイでも使えています。

grafanaの初期ログインユーザは「admin」パスワードは「admin」となっています。新しいパスワードに更新して好きなダッシュボードをインポートしましょう。

トラブルシュート事例?

prometheusのOOM [2020/6/25追記]

数週間稼働させていると、ワーカーの1台のCPU負荷（shellでelasticsearchに送ってkibanaで可視化してました）が定期的に上がっている事に気付きました。
prometheusの画面が開かなくなっていたので、kubernetesのpodを調べたところ、prometheus-k8s-0のprometheusコンテナが「CrashLoopBackOff」になっていました。早速コンテナのlogを確認すると、

# kubectl logs pod/prometheus-k8s-0 -n monitoring -c prometheus
      :
runtime: out of memory: cannot allocate byte block
fatal error: out of memory

どうやらメモリ不足のようでした。調べると結構出てくるのですが、「storage.tsdb.retention.time」を短くするという情報が多かったです。
今回利用しているマニフェストでは、manifestes/prometheus-prometheus.yamlのretentionを短くすれば良いかな？と考えて、以下のように編集したら無事にデプロイできました。

# grep retention manifests/prometheus-prometheus.yaml
  retention: 7d
# kubectl apply -f manifests/prometheus-prometheus.yaml
prometheus.monitoring.coreos.com/k8s configured
# kubectl get pods -n monitoring -o wide -w
    :
  (podの状態遷移が確認できます)
    :

おわりに

ラズパイ(ARM)用のkubernetesクラスターモニタリングとして、Cluster Monitoring stack for ARM / X86-64 platforms をそのまま利用できました。
利用されているコンポーネントもOpenShift4のクラスターモニタリングとほぼ同じ構成なので、OpenShiftの勉強にも使えると考えています。