はじめに
KubernetesクラスタにHarborをプライベートレジストリとしてインストールする際の備忘録。ディプロイ環境はオンプレ想定で構成してみる。
ディプロイのテストでは簡単な自作のAPIサービス(テストアプリ)をPodで動作させてみる。
利用するソフトウェア
- VMware Workstation 17 Pro (Windows 11 / X86_64)
- RHEL 9.5 (VM)
- Dnsmasq (2.79)
- HA-Proxy (1.8.27)
- Kubernetes (v1.32)
- CRI-O (v1.32)
- Calico (v3.29.3)
- MetalLB (v0.14.9)
- Helm (v3.17.3)
- Harbor (2.13.0)
- harbor-helm (1.17.0)
テスト用のアプリケーション作成 (参考)
- go (1.23.6)
- gin-gonic (v1.10.0)
- Podman (5.2.2)
- Visual Studio Code (1.100.0)
- Visual Studio Code拡張: Go (0.46.1)
- Visual Studio Code拡張: Remote Development (0.26.0)
- Visual Studio Code拡張: Remote Explorer (0.5.0)
- Visual Studio Code拡張: Dev Containers (0.413.0)
- Visual Studio Code拡張: YAML (1.18.0)
- コンテナ: ubi9/ubi-minimal (latest)
- コンテナ: ubi9/ubi (latest)
- コンテナ: ubi9/go-toolset (1.23)
※ goのバージョンはコンテナビルド時に利用する「ubi9/go-toolset」イメージでサポートされるgoのバージョンに（マイナーバージョン含めて）合わせたほうがよいかも。
Kubernetesクラスタ構成
以下で構築済みのKubernetesクラスタ、Cephによる永続ボリューム(PV)やPrivateCAを利用する。
Helm values.yamlの設定
[mng ~]$ git clone --branch 1.17.0 https://github.com/goharbor/harbor-helm.git
[mng ~]$ cd harbor-helm
[mng harbor-helm]$ vi values.yaml
[mng harbor-helm]$ git diff values.yaml
git diff values.yaml
diff --git a/values.yaml b/values.yaml
index 232f1bc..b2ff990 100644
--- a/values.yaml
+++ b/values.yaml
@@ -1,21 +1,21 @@
expose:
# Set how to expose the service. Set the type as "ingress", "clusterIP", "nodePort" or "loadBalancer"
# and fill the information in the corresponding section
- type: ingress
+ type: loadBalancer
tls:
# Enable TLS or not.
# Delete the "ssl-redirect" annotations in "expose.ingress.annotations" when TLS is disabled and "expose.type" is "ingress"
# Note: if the "expose.type" is "ingress" and TLS is disabled,
# the port must be included in the command when pulling/pushing images.
# Refer to https://github.com/goharbor/harbor/issues/5291 for details.
- enabled: true
+ enabled: false
# The source of the tls certificate. Set as "auto", "secret"
# or "none" and fill the information in the corresponding section
# 1) auto: generate the tls certificate automatically
# 2) secret: read the tls certificate from the specified secret.
# The tls certificate can be generated manually or by cert manager
# 3) none: configure no tls certificate for the ingress. If the default
# tls certificate is configured in the ingress controller, choose this option
certSource: auto
auto:
# The common name used to generate the certificate, it's necessary
@@ -102,21 +102,21 @@ expose:
#
# Format: protocol://domain[:port]. Usually:
# 1) if "expose.type" is "ingress", the "domain" should be
# the value of "expose.ingress.hosts.core"
# 2) if "expose.type" is "clusterIP", the "domain" should be
# the value of "expose.clusterIP.name"
# 3) if "expose.type" is "nodePort", the "domain" should be
# the IP address of k8s node
#
# If Harbor is deployed behind the proxy, set it as the URL of proxy
-externalURL: https://core.harbor.domain
+externalURL: http://harbor.test.k8s.local
# The persistence is enabled by default and a default StorageClass
# is needed in the k8s cluster to provision volumes dynamically.
# Specify another StorageClass in the "storageClass" or set "existingClaim"
# if you already have existing persistent volumes to use
#
# For storing images and charts, you can also use "azure", "gcs", "s3",
# "swift" or "oss". Set it in the "imageChartStorage" section
persistence:
enabled: true
@@ -126,57 +126,57 @@ persistence:
# and redis components, i.e. they are never deleted automatically)
resourcePolicy: "keep"
persistentVolumeClaim:
registry:
# Use the existing PVC which must be created manually before bound,
# and specify the "subPath" if the PVC is shared with other components
existingClaim: ""
# Specify the "storageClass" used to provision the volume. Or the default
# StorageClass will be used (the default).
# Set it to "-" to disable dynamic provisioning
- storageClass: ""
+ storageClass: "rook-ceph-block"
subPath: ""
accessMode: ReadWriteOnce
- size: 5Gi
+ size: 1Gi
annotations: {}
jobservice:
jobLog:
existingClaim: ""
- storageClass: ""
+ storageClass: "rook-ceph-block"
subPath: ""
accessMode: ReadWriteOnce
- size: 1Gi
+ size: 500Mi
annotations: {}
# If external database is used, the following settings for database will
# be ignored
database:
existingClaim: ""
- storageClass: ""
+ storageClass: "rook-ceph-block"
subPath: ""
accessMode: ReadWriteOnce
- size: 1Gi
+ size: 500Mi
annotations: {}
# If external Redis is used, the following settings for Redis will
# be ignored
redis:
existingClaim: ""
- storageClass: ""
+ storageClass: "rook-ceph-block"
subPath: ""
accessMode: ReadWriteOnce
- size: 1Gi
+ size: 500Mi
annotations: {}
trivy:
existingClaim: ""
- storageClass: ""
+ storageClass: "rook-ceph-block"
subPath: ""
accessMode: ReadWriteOnce
- size: 5Gi
+ size: 1Gi
annotations: {}
# Define which storage backend is used for registry to store
# images and charts. Refer to
# https://github.com/distribution/distribution/blob/release/2.8/docs/configuration.md#storage
# for the detail.
imageChartStorage:
# Specify whether to disable `redirect` for images and chart storage, for
# backends which not supported it (such as using minio for `s3` storage type), please disable
# it. To disable redirects, simply set `disableredirect` to `true` instead.
# Refer to
@@ -325,21 +325,21 @@ internalTLS:
# secret name for trivy's tls certs
secretName: ""
# Content of trivy's TLS key file, only available when `certSource` is "manual"
crt: ""
# Content of trivy's TLS key file, only available when `certSource` is "manual"
key: ""
ipFamily:
# ipv6Enabled set to true if ipv6 is enabled in cluster, currently it affected the nginx related component
ipv6:
- enabled: true
+ enabled: false
# ipv4Enabled set to true if ipv4 is enabled in cluster, currently it affected the nginx related component
ipv4:
enabled: true
imagePullPolicy: IfNotPresent
# Use this set to assign a list of default pullSecrets
imagePullSecrets:
# - name: docker-registry-secret
# - name: internal-registry-secret
インストール
インストール
[mng ~]$ helm repo add harbor https://helm.goharbor.io
[mng ~]$ helm repo update
[mng ~]$ kubectl create namespace harbor
[mng ~]$ helm install harbor ./harbor-helm -n harbor -f harbor-helm/values.yaml
NAME: harbor
LAST DEPLOYED: Mon May 04 01:01:33 2025
NAMESPACE: harbor
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Please wait for several minutes for Harbor deployment to complete.
Then you should be able to visit the Harbor portal at http://harbor.test.k8s.local
For more details, please visit https://github.com/goharbor/harbor
確認
[mng ~]$ kubectl get all -n harbor -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/harbor-core-dc65fd6b9-2qph7 1/1 Running 0 51m 172.20.194.104 k8s-worker1 <none> <none>
pod/harbor-database-0 1/1 Running 0 51m 172.20.194.92 k8s-worker1 <none> <none>
pod/harbor-jobservice-5d89fdb89f-xcnxn 1/1 Running 4 (50m ago) 51m 172.23.229.146 k8s-worker0 <none> <none>
pod/harbor-nginx-d9969d466-pthw9 1/1 Running 0 51m 172.20.194.93 k8s-worker1 <none> <none>
pod/harbor-portal-55798bf78b-94dd9 1/1 Running 0 51m 172.23.229.166 k8s-worker0 <none> <none>
pod/harbor-redis-0 1/1 Running 0 51m 172.20.194.91 k8s-worker1 <none> <none>
pod/harbor-registry-55bcdc8d6-spt55 2/2 Running 0 51m 172.23.229.147 k8s-worker0 <none> <none>
pod/harbor-trivy-0 1/1 Running 0 51m 172.23.229.167 k8s-worker0 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/harbor LoadBalancer 10.110.217.115 10.0.0.63 80:30826/TCP 51m app=harbor,component=nginx,release=harbor
service/harbor-core ClusterIP 10.101.60.60 <none> 80/TCP 51m app=harbor,component=core,release=harbor
service/harbor-database ClusterIP 10.104.178.231 <none> 5432/TCP 51m app=harbor,component=database,release=harbor
service/harbor-jobservice ClusterIP 10.103.187.162 <none> 80/TCP 51m app=harbor,component=jobservice,release=harbor
service/harbor-portal ClusterIP 10.101.129.9 <none> 80/TCP 51m app=harbor,component=portal,release=harbor
service/harbor-redis ClusterIP 10.96.152.7 <none> 6379/TCP 51m app=harbor,component=redis,release=harbor
service/harbor-registry ClusterIP 10.106.174.43 <none> 5000/TCP,8080/TCP 51m app=harbor,component=registry,release=harbor
service/harbor-trivy ClusterIP 10.96.42.46 <none> 8080/TCP 51m app=harbor,component=trivy,release=harbor
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/harbor-core 1/1 1 1 51m core goharbor/harbor-core:v2.13.0 app=harbor,component=core,release=harbor
deployment.apps/harbor-jobservice 1/1 1 1 51m jobservice goharbor/harbor-jobservice:v2.13.0 app=harbor,component=jobservice,release=harbor
deployment.apps/harbor-nginx 1/1 1 1 51m nginx goharbor/nginx-photon:v2.13.0 app=harbor,component=nginx,release=harbor
deployment.apps/harbor-portal 1/1 1 1 51m portal goharbor/harbor-portal:v2.13.0 app=harbor,component=portal,release=harbor
deployment.apps/harbor-registry 1/1 1 1 51m registry,registryctl goharbor/registry-photon:v2.13.0,goharbor/harbor-registryctl:v2.13.0 app=harbor,component=registry,release=harbor
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/harbor-core-dc65fd6b9 1 1 1 51m core goharbor/harbor-core:v2.13.0 app=harbor,component=core,pod-template-hash=dc65fd6b9,release=harbor
replicaset.apps/harbor-jobservice-5d89fdb89f 1 1 1 51m jobservice goharbor/harbor-jobservice:v2.13.0 app=harbor,component=jobservice,pod-template-hash=5d89fdb89f,release=harbor
replicaset.apps/harbor-nginx-d9969d466 1 1 1 51m nginx goharbor/nginx-photon:v2.13.0 app=harbor,component=nginx,pod-template-hash=d9969d466,release=harbor
replicaset.apps/harbor-portal-55798bf78b 1 1 1 51m portal goharbor/harbor-portal:v2.13.0 app=harbor,component=portal,pod-template-hash=55798bf78b,release=harbor
replicaset.apps/harbor-registry-55bcdc8d6 1 1 1 51m registry,registryctl goharbor/registry-photon:v2.13.0,goharbor/harbor-registryctl:v2.13.0 app=harbor,component=registry,pod-template-hash=55bcdc8d6,release=harbor
NAME READY AGE CONTAINERS IMAGES
statefulset.apps/harbor-database 1/1 51m database goharbor/harbor-db:v2.13.0
statefulset.apps/harbor-redis 1/1 51m redis goharbor/redis-photon:v2.13.0
statefulset.apps/harbor-trivy 1/1 51m trivy goharbor/trivy-adapter-photon:v2.13.0
PVCの確認
[mng ~]$ kubectl get pvc -n harbor -o wide
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE VOLUMEMODE
data-harbor-redis-0 Bound pvc-c31276b4-e6ca-4d4b-9dee-3ea072804ff9 500Mi RWO rook-ceph-block <unset> 50m Filesystem
data-harbor-trivy-0 Bound pvc-07d89dc7-6013-4efd-bbe2-01582e3a6d70 1Gi RWO rook-ceph-block <unset> 50m Filesystem
database-data-harbor-database-0 Bound pvc-fdc55a03-25cf-4e09-9540-dd6831b49628 500Mi RWO rook-ceph-block <unset> 50m Filesystem
harbor-jobservice Bound pvc-b92e64c4-30cf-4abf-92e9-50965515966e 500Mi RWO rook-ceph-block <unset> 50m Filesystem
harbor-registry Bound pvc-5738bcb4-7867-4db5-b131-1f0cb2f5153d 1Gi RWO rook-ceph-block <unset> 50m Filesystem
VIPの確認
[mng ~]$ kubectl get svc -n harbor
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ★
harbor LoadBalancer 10.110.217.115 10.0.0.63 80:30826/TCP 12m
harbor-core ClusterIP 10.101.60.60 <none> 80/TCP 12m
harbor-database ClusterIP 10.104.178.231 <none> 5432/TCP 12m
harbor-jobservice ClusterIP 10.103.187.162 <none> 80/TCP 12m
harbor-portal ClusterIP 10.101.129.9 <none> 80/TCP 12m
harbor-redis ClusterIP 10.96.152.7 <none> 6379/TCP 12m
harbor-registry ClusterIP 10.106.174.43 <none> 5000/TCP,8080/TCP 12m
harbor-trivy ClusterIP 10.96.42.46 <none> 8080/TCP 12m
MetalLBにより「10.0.0.63」がVIPとして割り当てられている。
Dnsmasq (DNSサービス)の設定を更新
Web UIのVIPにホスト名を割り当てる。lb.test.k8s.localノードで作業する。
設定ファイルを編集
[lb ~]$ sudo vi /etc/dnsmasq.d/k8s.conf
k8s.conf
# VIP by MetalLB
...
address=/harbor.test.k8s.local/10.0.0.63 # Harbor Web UI
...
Dnsmasqサービスを再起動
[lb ~]$ sudo systemctl restart dnsmasq
Web UIアクセスのTLS設定
Harbor用のサーバ証明書の作成
Web UI用のサーバ証明書を発行する。
Web UI用のサーバ証明書
[mng ~]$ mkdir tls
[mng ~]$ cd tls
# 証明書設定の作成
[mng tls]$ vi openssl.cnf
# 秘密鍵の作成
[mng tls]$ openssl genpkey -algorithm ec -pkeyopt ec_paramgen_curve:P-256 -out harbor.key
＃ CSRの作成
[mng tls]$ openssl req -new -key harbor.key -out harbor.csr -config openssl.cnf
# 証明書作成
[mng tls]$ openssl x509 -req -in harbor.csr -CA private_ca.crt -CAkey private_ca.key -CAcreateserial -out harbor.crt -days 365 -sha256 -extfile openssl.cnf -extensions server-cert
Signature ok
subject=C = JP, O = k8s, OU = test, CN = harbor
Getting CA Private Key
[mng tls]$ openssl x509 -in harbor.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
...
Issuer: CN = private_ca
...
Subject: C = JP, O = k8s, OU = test, CN = harbor
...
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Key Agreement
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:harbor.test.k8s.local, DNS:harbor.harbor.svc.cluster.local, DNS:harbor, IP Address:10.0.0.63, IP Address:10.110.217.115, IP Address:127.0.0.1
...
SAN(alt_name)をService(MetalLB)の情報に合わせて指定しておく。
openssl.cnf
[server-cert]
keyUsage = critical, digitalSignature, keyEncipherment, keyAgreement
extendedKeyUsage = serverAuth
subjectAltName = @alt_name
[req]
distinguished_name = dn
prompt = no
[dn]
C = JP
O = k8s
OU = test
CN = harbor
[alt_name]
DNS.1 = harbor.test.k8s.local # 外部アクセス用のホスト名
DNS.2 = harbor.harbor.svc.cluster.local # クラスタ内のホスト名
DNS.3 = harbor # クラスタ内のホスト名 (同一ネームスペース)
IP.1 = 10.0.0.63 # 外部アクセス用のIP
IP.2 = 10.110.217.115 # ClusterIP
IP.3 = 127.0.0.1 # ローカルホストからのアクセス用
Web UIサーバ証明書のSecretを作成
作成
[mng tls]$ kubectl create secret tls harbor-tls-cert --cert=harbor.crt --key=harbor.key -n harbor
secret/harbor-tls-cert created
確認
[mng tls]$ kubectl get secret -n harbor harbor-tls-cert -o yaml
apiVersion: v1
data:
tls.crt: xxxx...
...
tls.key: yyyy...
...
kind: Secret
metadata:
creationTimestamp: "2025-05-11T03:09:45Z"
name: harbor-tls-cert
namespace: harbor
resourceVersion: "472992"
uid: 21afac23-3a45-4fa2-9e45-95ab86736891
type: kubernetes.io/tls
Helm values.yamlの設定
[mng ~]$ cd harbor-helm
[mng harbor-helm]$ vi values.yaml
[mng harbor-helm]$ git diff values.yaml
git diff values.yaml
diff --git a/values.yaml b/values.yaml
index 232f1bc..f71979e 100644
--- a/values.yaml
+++ b/values.yaml
@@ -1,7 +1,7 @@
expose:
# Set how to expose the service. Set the type as "ingress", "clusterIP", "nodePort" or "loadBalancer"
# and fill the information in the corresponding section
- type: ingress
+ type: loadBalancer
tls:
# Enable TLS or not.
# Delete the "ssl-redirect" annotations in "expose.ingress.annotations" when TLS is disabled and "expose.type" is "ingress"
@@ -16,7 +16,7 @@ expose:
# The tls certificate can be generated manually or by cert manager
# 3) none: configure no tls certificate for the ingress. If the default
# tls certificate is configured in the ingress controller, choose this option
- certSource: auto
+ certSource: secret
auto:
# The common name used to generate the certificate, it's necessary
# when the type isn't "ingress"
@@ -25,7 +25,7 @@ expose:
# The name of secret which contains keys named:
# "tls.crt" - the certificate
# "tls.key" - the private key
- secretName: ""
+ secretName: "harbor-tls-cert"
...
設定を更新
[mng ~]$ helm upgrade harbor ./harbor-helm -n harbor -f harbor-helm/values.yaml
Release "harbor" has been upgraded. Happy Helming!
NAME: harbor
LAST DEPLOYED: Mon May 11 03:23:34 2025
NAMESPACE: harbor
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
Please wait for several minutes for Harbor deployment to complete.
Then you should be able to visit the Harbor portal at http://harbor.test.k8s.local
For more details, please visit https://github.com/goharbor/harbor
Service設定を確認
mng ~]$ kubectl get svc -n harbor
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
harbor LoadBalancer 10.110.217.115 10.0.0.63 80:30826/TCP,443:30520/TCP 11h ★
harbor-core ClusterIP 10.101.60.60 <none> 80/TCP 11h
harbor-database ClusterIP 10.104.178.231 <none> 5432/TCP 11h
harbor-jobservice ClusterIP 10.103.187.162 <none> 80/TCP 11h
harbor-portal ClusterIP 10.101.129.9 <none> 80/TCP 11h
harbor-redis ClusterIP 10.96.152.7 <none> 6379/TCP 11h
harbor-registry ClusterIP 10.106.174.43 <none> 5000/TCP,8080/TCP 11h
harbor-trivy ClusterIP 10.96.42.46 <none> 8080/TCP 11h
※MetalLBにより「10.0.0.63:443」が追加されている。
HTTPSアクセス確認
[mng ~]$ curl -v https://harbor.test.k8s.local/ --cacert tls/private_ca.crt
...
* Connected to harbor.test.k8s.local (10.0.0.63) port 443 (#0)
...
* successfully set certificate verify locations:
...
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=JP; O=k8s; OU=test; CN=harbor
...
* subjectAltName: host "harbor.test.k8s.local" matched cert's "harbor.test.k8s.local"
* issuer: CN=private_ca
* SSL certificate verify ok.
...
> GET / HTTP/1.1
> Host: harbor.test.k8s.local
> User-Agent: curl/7.61.1
> Accept: */*
>
...
< HTTP/1.1 200 OK
< Server: nginx
...
<
<!DOCTYPE html>
<html>
...
</html>
ちなみにHTTPアクセスに対してのリダイレクト設定も入っている模様。
[mng ~]$ curl -v http://harbor.test.k8s.local/
...
* Connected to harbor.test.k8s.local (10.0.0.63) port 80 (#0)
...
> GET / HTTP/1.1
> Host: harbor.test.k8s.local
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.26.2
...
< Location: https://harbor.test.k8s.local/
<
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.26.2</center>
</body>
</html>
Web UIアクセス確認
User: admin
Password: values.yamlのharborAdminPasswordフィールド値
アプリケーションのディプロイテスト
簡単なAPIサービスをPodで動作させて実施してみる。
事前準備
PrivateCA証明書をノードシステムに登録
今回構築したプライベートレジストリ(Harbor)とのTLS通信はContainer runtime (CRI-O)により実行されるため、PrivateCA証明書をワーカーノードのシステム(kubernetesクラスタの外側)へ登録しておく(またコントロールプレーンノードのシステムにも一応登録しておく)。
PrivateCA証明書の登録 (k8s-worker0の例)
[mng ~]$ scp tls/private_ca.crt k8s-worker0.test.k8s.local:/tmp/
[mng ~]$ ssh k8s-worker0.test.k8s.local "sudo -S cp /tmp/private_ca.crt /etc/pki/ca-trust/source/anchors/test_k8s_local_ca.crt && sudo -S update-ca-trust extract && curl -v https://harbor.test.k8s.local"
...
* Connected to harbor.test.k8s.local (10.0.0.63) port 443 ...
...
* SSL connection using TLSv1.3 ...
...
HTTP/1.1 200 OK
...
※ 登録したPrivatreCA証明書が各ノードの「/etc/pki/tls/certs/ca-bundle.crt」へマージされる。
上記を全ノードに対して実施する。
- k8s-worker0.test.k8s.local
- k8s-worker1.test.k8s.local
- k8s-worker2.test.k8s.local
- k8s-master0.test.k8s.local
- k8s-master1.test.k8s.local
- k8s-master2.test.k8s.local
テストアプリケーションの作成
[mng ~]$ wget https://go.dev/dl/go1.23.6.linux-amd64.tar.gz
[mng ~]$ sudo rm -rf /usr/local/go && tar -C /usr/local -xzf go1.23.6.linux-amd64.tar.gz
[mng ~]$ export PATH=$PATH:/usr/local/go/bin
[mng ~]$ go version
go version go1.23.6 linux/amd64
[mng ~]$ mkdir k8s-test-app
[mng ~]$ cd k8s-test-app
[mng k8s-test-app]$ go mod init test.k8s.local/test-app
[mng k8s-test-app]$ vi main.go
[mng k8s-test-app]$ go mod tidy
[mng k8s-test-app]$ go build
[mng k8s-test-app]$ ls
go.mod go.sum main.go test-app
main.go
package main
import (
"github.com/gin-gonic/gin"
"net/http"
)
func main() {
r := gin.Default()
r.GET("/ping", func(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{"message": "pong"})
})
r.Run(":8080")
}
テストアプリケーションのコンテナイメージを作成
[mng ~]$ sudo dnf install -y podman
[mng ~]$ podman --version
podman version 5.2.2
[mng ~]$ cd k8s-test-app
[mng k8s-test-app]$ podman build -t k8s-test-app:latest -f Containerfile .
[mng k8s-test-app]$ podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/k8s-test-app latest 1aba92ed3d43 7 seconds ago 229 MB
<none> <none> ff57c1ac1fe0 25 minutes ago 1.53 GB
<none> <none> 3932eb3d7723 29 minutes ago 1.38 GB
<none> <none> 384b325ddbb4 34 minutes ago 1.13 GB
registry.access.redhat.com/ubi9/go-toolset 1.23 d8663eae6e1a 5 days ago 1.13 GB
registry.access.redhat.com/ubi9 latest 18ac20acd5ec 13 days ago 217 MB
Containerfile
# Stage 1: Build
FROM registry.access.redhat.com/ubi9/go-toolset:1.23 AS builder
USER root
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o app
# Stage 2: Runtime
FROM registry.access.redhat.com/ubi9
WORKDIR /app
COPY --from=builder /app/app .
EXPOSE 8080
CMD ["./app"]
[mng k8s-test-app]$ podman run -d --name my-k8s-test-app -p 8080:8080 k8s-test-app:latest
[mng k8s-test-app]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dd9355810bc5 localhost/k8s-test-app:latest ./app 16 seconds ago Up 16 seconds 0.0.0.0:8080->8080/tcp, 8080/tcp my-k8s-test-app
[mng k8s-test-app]$ curl http://localhost:8080/ping |jq .
{
"message": "pong"
}
[mng k8s-test-app]$ podman stop my-k8s-test-app
[mng k8s-test-app]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
コンテナイメージをレジストリ(Harbor)へPush
プロジェクトを新規作成
※Web UIから作成してもOK。
新規作成
[mng k8s-test-app]$ curl -u admin:<password> -X POST --cacert ../tls/private_ca.crt "https://harbor.test.k8s.local/api/v2.0/projects" \
-H "Content-Type: application/json" \
-d '{
"project_name": "k8s-test-app",
"public": false
}'
確認
[mng k8s-test-app]$ curl -u admin:Harbor12345 -X GET --cacert ../ceph-test/tls/private_ca.crt "https://harbor.test.k8s.local/api/v2.0/projects/k8s-test-app" | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 389 100 389 0 0 14961 0 --:--:-- --:--:-- --:--:-- 15560
{
"creation_time": "2025-05-04T13:30:08.963Z",
"current_user_role_id": 1,
"current_user_role_ids": [
1
],
"cve_allowlist": {
"creation_time": "0001-01-01T00:00:00.000Z",
"id": 2,
"items": [],
"project_id": 2,
"update_time": "0001-01-01T00:00:00.000Z"
},
"metadata": {
"public": "false"
},
"name": "k8s-test-app",
"owner_id": 1,
"owner_name": "admin",
"project_id": 2,
"repo_count": 0,
"update_time": "2025-05-04T13:30:08.963Z"
}
Podman用にPrivateCA証明書を登録
[mng k8s-test-app]$ sudo mkdir -p /etc/containers/certs.d/harbor.test.k8s.local
[mng k8s-test-app]$ sudo cp ../tls/private_ca.crt /etc/containers/certs.d/harbor.test.k8s.local
[mng k8s-test-app]$ ls /etc/containers/certs.d/harbor.test.k8s.local
private_ca.crt
コンテナイメージをPush
タグ付け
[mng k8s-test-app]$ podman tag k8s-test-app:latest harbor.test.k8s.local/k8s-test-app/k8s-test-app:latest
ログイン
[mng k8s-test-app]$ podman login harbor.test.k8s.local
Username: admin
Password:
Login Succeeded!
Push
[mng k8s-test-app]$ podman push k8s-test-app harbor.test.k8s.local/k8s-test-app/k8s-test-app:latest
Getting image source signatures
Copying blob d13b1160182f done |
Copying blob 58ff4fd5ef9c done |
Copying config 1aba92ed3d done |
Writing manifest to image destination
レジストリアクセス用の認証情報でSecretを作成
作成
[mng k8s-test-app]$ kubectl create secret docker-registry harbor-creds --docker-server=harbor.test.k8s.local --docker-username=admin --docker-password='password' --docker-email=admin@test.k8s.local
確認
[mng k8s-test-app]$ kubectl get secret
NAME TYPE DATA AGE
ceph-delete-bucket Opaque 2 4d19h
harbor-creds kubernetes.io/dockerconfigjson 1 11s
[mng k8s-test-app]$ kubectl get secret harbor-creds -o yaml
apiVersion: v1
data:
.dockerconfigjson: eyJhdXRo...
kind: Secret
metadata:
creationTimestamp: "2025-05-13T02:44:01Z"
name: harbor-creds
namespace: default
resourceVersion: "589568"
uid: 764b6c0d-c39a-4a8a-ade0-4e68d934595e
type: kubernetes.io/dockerconfigjson
[mng k8s-test-app]$ echo eyJhdXRo... | base64 -d | jq .
{
"auths": {
"harbor.test.k8s.local": {
"username": "admin",
"password": "password",
"email": "admin@test.k8s.local",
"auth": "xxxxxx="
}
}
}
テストアプリケーションのPodをクラスタへディプロイ
ディプロイ実施
[mng k8s-test-app]$ kubectl apply -f k8s-test-app-deploy.yaml
k8s-test-app-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: k8s-test-app
spec:
replicas: 3
selector:
matchLabels:
app: k8s-test-app
template:
metadata:
labels:
app: k8s-test-app
spec:
imagePullSecrets:
- name: harbor-creds
containers:
- name: k8s-test-app
image: harbor.test.k8s.local/k8s-test-app/k8s-test-app:latest
imagePullPolicy: Always
ports:
- containerPort: 8080
ディプロイ確認
[k8s-test-app]$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
k8s-test-app-f6689b7d6-lcxvn 1/1 Running 0 11m 172.20.194.119 k8s-worker1 <none> <none>
k8s-test-app-f6689b7d6-vk42g 1/1 Running 0 10m 172.30.126.48 k8s-worker2 <none> <none>
k8s-test-app-f6689b7d6-vk5fv 1/1 Running 0 11m 172.23.229.161 k8s-worker0 <none> <none>
shellへ接続確認
[mng k8s-test-app]$ kubectl exec -it k8s-test-app-f6689b7d6-lcxvn -- bash
[root@k8s-test-app-f6689b7d6-lcxvn etc]# cat /etc/redhat-release
Red Hat Enterprise Linux release 9.5 (Plow)
[root@k8s-test-app-f6689b7d6-lcxvn app]# pwd
/app
[root@k8s-test-app-f6689b7d6-lcxvn app]# ls -la
total 11364
drwxr-xr-x. 2 root root 17 May 04 13:14 .
dr-xr-xr-x. 1 root root 40 May 04 02:53 ..
-rwxr-xr-x. 1 root root 11632648 May 04 12:49 app
[root@k8s-test-app-f6689b7d6-lcxvn etc]# exit
テストアプリケーション用のService(MetalLB)を作成
作成
[mng k8s-test-app]$ kubectl apply -f k8s-test-app-svc.yaml
k8s-test-app-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: k8s-test-app
spec:
type: LoadBalancer
selector:
app: k8s-test-app
ports:
- protocol: TCP
port: 80
targetPort: 8080
確認
[mng k8s-test-app]$ kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
k8s-test-app LoadBalancer 10.100.133.21 10.0.0.64 80:32278/TCP 14s app=k8s-test-app
...
※ MetalLBにより「10.0.0.64」がVIPとして割り当てられている。
テスト実施
外部クライアントからアクセス疎通確認 (管理端末)
[mng k8s-test-app]$ curl http://10.0.0.64/ping | jq .
{
"message": "pong"
}
他のPodからアクセス疎通確認
テスト用Podの実行
[mng k8s-test-app]$ kubectl run testpod --image=registry.access.redhat.com/ubi9/ubi-minimal --restart=Never -it -- sh
# アクセス疎通確認
sh-5.1# curl http://k8s-test-app/ping
{"message":"pong"}
sh-5.1# exit
テスト用Podを削除
[mng k8s-test-app]$ k8s-test-app]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
testpod 0/1 Completed 0 4m37s
...
[mng k8s-test-app]$ kubectl delete pod testpod
pod "testpod" deleted