LoginSignup
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

@dai624fuji(hiroshi fujioka)

OpenshiftにNGINX Ingress Controllerを追加してみる

Posted at

目的

Openshiftでデフォルトで提供されるRouterの他に、NGINX Ingress Controllerも使いたい。

環境

Openshift 4.6

  • Openshift自体はInternet 接続なし
  • bastion nodeのみInternet接続可能
  • IBM CloudのVirtual Private Cloudを利用
  • コンテナImageを管理するプライベートレジストリにIBM Cloud Container Registryを利用

参考情報

NGINX Ingress Controller を Openshiftにデプロイ

helm v3でNGINX Ingress ControllerをOpenshiftにデプロイする

1. bastion nodeにhelmコマンドを用意

[user01@bastion-node ~]$ curl -o helm.tar.gz -k https://get.helm.sh/helm-v3.4.2-linux-amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12.7M  100 12.7M    0     0  7776k      0  0:00:01  0:00:01 --:--:-- 7778k
[user01@bastion-node ~]$ tar xf helm.tar.gz
[user01@bastion-node ~]$ mv linux-amd64/helm ~/.local/bin/.
[user01@bastion-node ~]$ PATH=$PATH:~/.local/bin
[user01@bastion-node ~]$ helm version
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}
[user01@bastion-node ~]$

2. ingress-nginx chartをダウンロード

[user01@bastion-node ~]$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
"ingress-nginx" has been added to your repositories
[user01@bastion-node ~]$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ingress-nginx" chart repository
...Successfully got an update from the "ibm-helm" chart repository
Update Complete. ⎈Happy Helming!⎈
[user01@bastion-node ~]$
[user01@bastion-node ~]$ helm repo list
NAME            URL
ibm-helm        https://raw.githubusercontent.com/IBM/charts/master/repo/ibm-helm
ingress-nginx   https://kubernetes.github.io/ingress-nginx
[user01@bastion-node ~]$
[user01@bastion-node ~]$ helm search repo ingress-nginx/ingress-nginx
WARNING: Repo "ibm-helm" is corrupt or missing. Try 'helm repo update'.
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
ingress-nginx/ingress-nginx     3.15.2          0.41.2          Ingress controller for Kubernetes using NGINX a...
[user01@bastion-node ~]$
[user01@bastion-node ~]$ mkdir ~/work/ingress-nginx
[user01@bastion-node ~]$ cd ~/work/ingress-nginx
[user01@bastion-node ingress-nginx]$ helm pull ingress-nginx/ingress-nginx
[user01@bastion-node ingress-nginx]$ tar xf ingress-nginx-3.15.2.tgz
[user01@bastion-node ingress-nginx]$ ls
ingress-nginx  ingress-nginx-3.15.2.tgz
[user01@bastion-node ingress-nginx]$

3. 必要なイメージをプライベートレジストリに用意

  • helmチャート内のファイルからデプロイに使うimageを確認。
[user01@bastion-node ingress-nginx]$ grep image: ingress-nginx/values.yaml -A2
  image:
    repository: k8s.gcr.io/ingress-nginx/controller
    tag: "v0.41.2"
--
  #    image: nginx:latest
  #  - name: lemonldap-ng-controller
  #    image: lemonldapng/lemonldap-ng-controller:0.2.0
  #    args:
  #      - /lemonldap-ng-controller
--
  #   image: busybox
  #   command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice; sleep 2; done;']

--
      image:
        repository: docker.io/jettech/kube-webhook-certgen
        tag: v1.5.0
--
  image:
    repository: k8s.gcr.io/defaultbackend-amd64
    tag: "1.5"
[user01@bastion-node ingress-nginx]$

対象imageは以下

  • k8s.gcr.io/ingress-nginx/controller:v0.41.2

  • docker.io/jettech/kube-webhook-certgen:v1.5.0

  • k8s.gcr.io/defaultbackend-amd64:1.5

  • bastion nodeでimageをpullして、tagを付与し、プライベートレジストリにpush

docker pull k8s.gcr.io/defaultbackend-amd64:1.5
docker tag k8s.gcr.io/defaultbackend-amd64:1.5 jp.icr.io/test/defaultbackend-amd64:1.5
docker push jp.icr.io/test/defaultbackend-amd64:1.5

上記の手順と同様に他のimageもプライベートレジストリにpushする

  • クラスターがプライベートレジストリにアクセスできるようにPullシークレットをOpenshiftに追加
    ※pull secretの中身（user id, password）を用意する手順は省略
[user01@bastion-node ingress-nginx]$ oc -n user01-test create secret docker-registry containerregistry-test --docker-server=jp.icr.io --docker-username=iamapikey --docker-password=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX --docker-email=e34559@jp.ibm.com
secret/containerregistry-test created
[user01@bastion-node ingress-nginx]$

4. helmチャートのファイルを更新

internet接続のない環境なのでimageなど一部を変更

ingress-nginx/values.yaml

annotationsの部分の記載は、NGINX Ingress Controllerの前段のロードバランサーをIBM Cloudで作るためのもの。

[user01@bastion-node ingress-nginx]$ cp ingress-nginx/values.yaml ingress-nginx/values.yaml.org
[user01@bastion-node ingress-nginx]$ vi ingress-nginx/values.yaml
[user01@bastion-node ingress-nginx]$ diff ingress-nginx/values.yaml ingress-nginx/values.yaml.org
13c13
<     repository: jp.icr.io/test/ingress-nginx/controller
---
>     repository: k8s.gcr.io/ingress-nginx/controller
15c15
<     #digest: sha256:1f4f402b9c14f3ae92b11ada1dfe9893a88f0faeb0b2f4b903e2c67a0c3bf0de
---
>     digest: sha256:1f4f402b9c14f3ae92b11ada1dfe9893a88f0faeb0b2f4b903e2c67a0c3bf0de
269a270
>
352,353c353
<     annotations:
<       service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: private
---
>     annotations: {}
481c481
<         repository: jp.icr.io/test/jettech/kube-webhook-certgen
---
>         repository: docker.io/jettech/kube-webhook-certgen
490,491c490
<       #runAsUser: 2000
<       runAsUser: 1000622000
---
>       runAsUser: 2000
597c596
<     repository: jp.icr.io/test/defaultbackend-amd64
---
>     repository: k8s.gcr.io/defaultbackend-amd64
714,715c713
< imagePullSecrets:
< - name: containerregistry-test
---
> imagePullSecrets: []
[user01@bastion-node ingress-nginx]$

ingress-nginx/templates/controller-role.yaml

以下追記。

[user01@bastion-node ingress-nginx]$ vi ingress-nginx/templates/controller-role.yaml
[user01@bastion-node ingress-nginx]$ tail -n 10 ingress-nginx/templates/controller-role.yaml
{{- end }}
  - apiGroups:
    - security.openshift.io
    resourceNames:
    - privileged
    resources:
    - securitycontextconstraints
    verbs:
    - use

[user01@bastion-node ingress-nginx]$

※ templates以下のファイルは全て読み込まれてしまうので、元ファイルを別名でコピーして同じディレクトリに置いたりしないこと。

5. NGINX Ingress Controllerのデプロイ実行

  • ocコマンドでopenshiftにログインし、NGINX Ingress Controllerを導入するproject(namespace)に切り替える
oc login -u user01
oc project user01-test
  • デプロイ実行
[user01@bastion-node ingress-nginx]$ helm install ingress-nginx-test ./ingress-nginx
NAME: ingress-nginx-test
LAST DEPLOYED: Mon Dec 21 16:53:25 2020
NAMESPACE: user01-test
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace user01-test get services -o wide -w ingress-nginx-test-controller'

An example Ingress that makes use of the controller:

  apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls
[user01@bastion-node ingress-nginx]$
[user01@bastion-node ingress-nginx]$ oc get all
NAME                                                READY   STATUS    RESTARTS   AGE
pod/ingress-nginx-test-controller-cfc699764-lpm5t   1/1     Running   0          19m

NAME                                              TYPE           CLUSTER-IP      EXTERNAL-IP                          PORT(S)                      AGE
service/ingress-nginx-test-controller             LoadBalancer   172.21.22.87    d1b930bf-jp-tok.lb.appdomain.cloud   80:31140/TCP,443:32343/TCP   19m
service/ingress-nginx-test-controller-admission   ClusterIP      172.21.199.70   <none>                               443/TCP                      19m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-test-controller   1/1     1            1           19m

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-test-controller-cfc699764   1         1         1       19m
[user01@bastion-node ingress-nginx]$

DNS サブドメインへの VPC ロード・バランサーのホスト名の登録

[user01@bastion-node ~]$ ibmcloud ks nlb-dns create vpc-gen2 --cluster tmpcluster --lb-host d1b930bf-jp-tok.lb.appdomain.cloud --type private
Creating NLB DNS...
OK
NLB hostname was created as test-cluster-xx1.jp-tok.containers.appdomain.cloud
[user01@bastion-node ~]$
[user01@bastion-node ~]$ ibmcloud ks nlb-dns ls --cluster tmpcluster
OK
Subdomain                                                                            Load Balancer Hostname               SSL Cert Status   SSL Cert Secret Name                               Secret Namespace
test-cluster-xx0.jp-tok.containers.appdomain.cloud   818aee49-jp-tok.lb.appdomain.cloud   created           test-cluster-xx0   openshift-ingress
test-cluster-xx1.jp-tok.containers.appdomain.cloud   d1b930bf-jp-tok.lb.appdomain.cloud   created           test-cluster-xx1   default
[user01@bastion-node ~]$

shardingを設定

このままだと、NGINX Ingress Controllerでサービスを公開すると、デフォルトのRouterにも変更が加わる。（そんなに問題はないのですが。）
デフォルトのRouterには変更を行わないようにするために追加設定を行う。

  • ingress controllerに設定を追加
oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"routeSelector":{"matchExpressions":[{"key":"nginxingress", "operator":"DoesNotExist"}]}}}' --type=merge

NGINX Ingress Controllerのテスト

  • テスト用イメージ用意
docker pull openshift/hello-openshift:latest
docker tag openshift/hello-openshift:latest jp.icr.io/test/test/openshift/hello-openshift:latest
docker push jp.icr.io/test/test/openshift/hello-openshift:latest
  • テスト用podをデプロイ
oc new-app --docker-image=jp.icr.io/test/test/openshift/hello-openshift:latest --name user01-test -o yaml > test_deploy.yaml
vi test_deploy.yaml
# 以下の様にpull secretや envを追記
---
      spec:
        imagePullSecrets:
        - name: containerregistry-test
        containers:
        - image: ' '
          name: user01-test
          ports:
          - containerPort: 8080
            protocol: TCP
          - containerPort: 8888
            protocol: TCP
          resources: {}
          env:
          - name: RESPONSE
            value: Hello World!
~~
---

oc create -f test_deploy.yaml
  • サービスをingressで公開
[user01@bastion-node ingress-nginx]$ cat test_ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-myservicea
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: user01-test.test-cluster-xx1.jp-tok.containers.appdomain.cloud
    http:
      paths:
      - path: /
        backend:
          serviceName: user01-test
          servicePort: 8080
[user01@bastion-node ingress-nginx]$
[user01@bastion-node ingress-nginx]$ oc create -f test_ingress.yaml
ingress.networking.k8s.io/ingress-user01-test created
[user01@bastion-node ingress-nginx]$
[user01@bastion-node ingress-nginx]$ oc get ingress
NAME                  CLASS    HOSTS                                                                                                        ADDRESS                              PORTS   AGE
ingress-user01-test   <none>   user01-test.test-cluster-xx1.jp-tok.containers.appdomain.cloud   d1b930bf-jp-tok.lb.appdomain.cloud   80      46s
[user01@bastion-node ingress-nginx]$
  • 接続確認
[user01@bastion-node ingress-nginx]$ curl http://user01-test.test-cluster-xx1.jp-tok.containers.appdomain.cloud
Hello World!
[user01@bastion-node ingress-nginx]$

以上

1
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?