Help us understand the problem. What is going on with this article?

GKEでもNGINXのIngressは使える

GKEでもNGINXのIngressは使える

はじめに

GKEでIngressといったらGoogle Cloud Load Blancingと思っていたが、Nginxも使えたみたいなので、ちょっと触ってみる。

NginxのIngressについて

↓に記載されている
https://cloud.google.com/community/tutorials/nginx-ingress-gke

NginxのIngressはKubernetesコミュニティが管理しているkubernetes/ingress-nginxとNginx社管理のnginxinc/kubernetes-ingressは別物である。

主な違いは以下を参照

https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/nginx-ingress-controllers.md

Aspect or Feature kubernetes/ingress-nginx nginxinc/kubernetes-ingress with NGINX nginxinc/kubernetes-ingress with NGINX Plus
Fundamental
Authors Kubernetes community NGINX Inc and community NGINX Inc and community
NGINX version Custom NGINX build that includes several third-party modules NGINX official mainline build NGINX Plus
Commercial support N/A N/A Included
Load balancing configuration via the Ingress resource
Merging Ingress rules with the same host Supported Supported via Mergeable Ingresses Supported via Mergeable Ingresses
HTTP load balancing extensions - Annotations See the supported annotations See the supported annotations See the supported annotations
HTTP load balancing extensions -- ConfigMap See the supported ConfigMap keys See the supported ConfigMap keys See the supported ConfigMap keys
TCP/UDP Supported via a ConfigMap Supported via a ConfigMap with native NGINX configuration Supported via a ConfigMap with native NGINX configuration
Websocket Supported Supported via an annotation Supported via an annotation
TCP SSL Passthrough Supported via a ConfigMap Not supported Not supported
JWT validation Not supported Not supported Supported
Session persistence Supported via a third-party module Not supported Supported
Canary testing (by header, cookie, weight) Supported via annotations Supported via custom resources Supported via custom resources
Configuration templates *1 See the template See the templates See the templates
Load balancing configuration via Custom Resources
HTTP load balancing Not supported See VirtualServer and VirtualServerRoute resources. See VirtualServer and VirtualServerRoute resources.
Deployment
Command-line arguments *2 See the arguments See the arguments See the arguments
TLS certificate and key for the default server Required as a command-line argument/ auto-generated Required as a command-line argument Required as a command-line argument
Helm chart Supported Supported Supported
Operational
Reporting the IP address(es) of the Ingress controller into Ingress resources Supported Supported Supported
Extended Status Supported via a third-party module Not supported Supported
Prometheus Integration Supported Supported Supported
Dynamic reconfiguration of endpoints (no configuration reloading) Supported with a third-party Lua module Not supported Supported

環境構築

前提条件

  • GKEクラスタに使用するSubnetは作成済みとする

GKEクラスタ作成

gcloud beta container --project "PROJECT-NAME" clusters create "nginx-ingress" \
--zone "asia-northeast1-a" --node-locations "asia-northeast1-a","asia-northeast1-b" \
--no-enable-basic-auth --release-channel "regular" --machine-type "n1-standard-1" \
--image-type "COS" --disk-type "pd-standard" --disk-size "100" --metadata disable-legacy-endpoints=true \
--scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
--num-nodes "1" --enable-stackdriver-kubernetes --enable-ip-alias \
--network "projects/PROJECT-NAME/global/networks/sakon-network" --subnetwork "projects/PROJECT-NAME/regions/asia-northeast1/subnetworks/sakon-gke-subnet-1" \
--cluster-secondary-range-name "sakon-gkesub1-2nd-range-2" --services-secondary-range-name "sakon-gkesub1-2nd-range-1" \
--enable-intra-node-visibility --default-max-pods-per-node "110" --addons HorizontalPodAutoscaling,HttpLoadBalancing \
--enable-autoupgrade --enable-autorepair
$ kubectl version --short
Client Version: v1.14.7
Server Version: v1.14.3-gke.11

Tillerのインストール

$ kubectl create serviceaccount --namespace kube-system tiller
serviceaccount/tiller created
$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io/tiller-cluster-rule created
$ helm init --service-account tiller
$HELM_HOME has been configured at /Users/sakonju/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation

tillerがインストールされてるか確認

$ kubectl get deploy -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
event-exporter-v0.2.5 1/1 1 1 34h
fluentd-gcp-scaler 1/1 1 1 34h
heapster-v1.6.1 1/1 1 1 34h
kube-dns 2/2 2 2 34h
kube-dns-autoscaler 1/1 1 1 34h
l7-default-backend 1/1 1 1 34h
metrics-server-v0.3.1 1/1 1 1 34h
stackdriver-metadata-agent-cluster-level 1/1 1 1 34h
tiller-deploy 1/1 1 1 34h

GKEにアプリケーションをデプロイ

$ kubectl run --restart=Always hello-app --image=gcr.io/google-samples/hello-app:1.0 --port=8080
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/hello-app created

serviceを作って外部公開する
```
$ kubectl expose deployment hello-app
service/hello-app exposed

$ kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
hello-app ClusterIP 10.121.227.201 8080/TCP 32s run=hello-app
kubernetes ClusterIP 10.120.0.1 443/TCP 34h
```

Helmを使用してNGINX Ingress Controllerをdeploy

以下の様な構成となる

image.png

nginx-ingressHelmでdeployする

$ helm install --name nginx-ingress stable/nginx-ingress --set rbac.create=true --set controller.publishService.enabled=true
NAME:   nginx-ingress
LAST DEPLOYED: Sun Oct  6 14:23:12 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                                            READY  STATUS             RESTARTS  AGE
nginx-ingress-controller-6cb795cdc5-r9j5c       0/1    ContainerCreating  0         1s
nginx-ingress-default-backend-6b8dc9d88f-hfv54  0/1    ContainerCreating  0         1s

==> v1/Service
NAME                           TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)                     AGE
nginx-ingress-controller       LoadBalancer  10.122.205.4  <pending>    80:31003/TCP,443:32313/TCP  1s
nginx-ingress-default-backend  ClusterIP     10.121.57.91  <none>       80/TCP                      1s

==> v1/ServiceAccount
NAME                   SECRETS  AGE
nginx-ingress          1        1s
nginx-ingress-backend  1        1s

==> v1beta1/ClusterRole
NAME           AGE
nginx-ingress  1s

==> v1beta1/ClusterRoleBinding
NAME           AGE
nginx-ingress  1s

==> v1beta1/Deployment
NAME                           READY  UP-TO-DATE  AVAILABLE  AGE
nginx-ingress-controller       0/1    1           0          1s
nginx-ingress-default-backend  0/1    1           0          1s

==> v1beta1/Role
NAME           AGE
nginx-ingress  1s

==> v1beta1/RoleBinding
NAME           AGE
nginx-ingress  1s


NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w nginx-ingress-controller'

An example Ingress that makes use of the controller:

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

↓L4ロードバランサが作成される

==> v1/Service
NAME                           TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)                     AGE
nginx-ingress-controller       LoadBalancer  10.122.205.4  <pending>    80:31003/TCP,443:32313/TCP  1s
nginx-ingress-default-backend  ClusterIP     10.121.57.91  <none>       80/TCP                      1s

しばらくするとEXTERNAL-IPも取得されている。

$ kubectl get service
NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
hello-app                       ClusterIP      10.121.227.201   <none>         8080/TCP                     19h
kubernetes                      ClusterIP      10.120.0.1       <none>         443/TCP                      2d5h
nginx-ingress-controller        LoadBalancer   10.122.205.4     34.84.181.58   80:31003/TCP,443:32313/TCP   2m44s
nginx-ingress-default-backend   ClusterIP      10.121.57.91     <none>         80/TCP                       2m44s

スクリーンショット 2019-10-06 14.28.48.png

2番目のサービスである、 nginx-ingress-default-backendに注目。default-backendは全てのURLパスを処理し、NGINXコントローラが理解できないホスト(つまり、Ingressリソースにマッピングされていない全てのリクエスト)をホストするサービス。デフォルトのバックエンドは2つのURLを公開する。

  • /healthz 200を返す
  • / 404を返す
$ curl -v http://34.84.181.58/healthz
*   Trying 34.84.181.58...
* TCP_NODELAY set
* Connected to 34.84.181.58 (34.84.181.58) port 80 (#0)
> GET /healthz HTTP/1.1
> Host: 34.84.181.58
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.1
< Date: Sun, 06 Oct 2019 06:32:28 GMT
< Content-Type: text/html
< Content-Length: 0
< Connection: keep-alive
<
$ curl -v http://34.84.181.58/
*   Trying 34.84.181.58...
* TCP_NODELAY set
* Connected to 34.84.181.58 (34.84.181.58) port 80 (#0)
> GET / HTTP/1.1
> Host: 34.84.181.58
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: openresty/1.15.8.1
< Date: Sun, 06 Oct 2019 06:41:45 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 21
< Connection: keep-alive
<
* Connection #0 to host 34.84.181.58 left intact

NGINX Ingress Controllerを使用するためのIngressリソースの構成

GCPの場合、annotationで定義が無い場合はGCLBが作成される。

NGINXの場合
annotations: kubernetes.io/ingress.class: nginx
GCLBの場合
annotations: kubernetes.io/ingress.class: gce

ingress-resource.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - path: /hello
        backend:
          serviceName: hello-app
          servicePort: 8080

applyする

$ kubectl apply -f ingress-resource.yaml
ingress.networking.k8s.io/ingress-resource created

作成されたリソースを確認

$ kubectl describe ing
Name:             ingress-resource
Namespace:        default
Address:          34.84.181.58
Default backend:  default-http-backend:80 (10.124.0.9:8080)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *
        /hello   hello-app:8080 (10.124.1.11:8080)
Annotations:
  kubernetes.io/ingress.class:                       nginx
  nginx.ingress.kubernetes.io/ssl-redirect:          false
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/ssl-redirect":"false"},"name":"ingress-resource","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"hello-app","servicePort":8080},"path":"/hello"}]}}]}}

Events:  <none>

アクセスしてみる

/helloではバックエンドのwebアプリケーションにアクセス出来るようになっている!

$ curl http://34.84.181.58/hello
Hello, world!
Version: 1.0.0
Hostname: hello-app-84d9648f89-vzh5v

/hello以外だと、404を返すので、デフォルトバックエンドサービスが適切に機能している。

$ curl http://34.84.181.58/test
default backend - 404
HirokiSakonju
インフラ系のエンジニアやっとります。 特にGCP/AWSを日々イジイジしとります。
Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
Comments
No comments
Sign up for free and join this conversation.
If you already have a Qiita account
Why do not you register as a user and use Qiita more conveniently?
You need to log in to use this function. Qiita can be used more conveniently after logging in.
You seem to be reading articles frequently this month. Qiita can be used more conveniently after logging in.
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
ユーザーは見つかりませんでした