More than 5 years have passed since last update.

GKEでもNGINXのIngressは使える

Posted at 2019-10-16

GKEでもNGINXのIngressは使える

はじめに

GKEでIngressといったらGoogle Cloud Load Blancingと思っていたが、Nginxも使えたみたいなので、ちょっと触ってみる。

NginxのIngressについて

↓に記載されている
https://cloud.google.com/community/tutorials/nginx-ingress-gke

NginxのIngressはKubernetesコミュニティが管理しているkubernetes/ingress-nginxとNginx社管理のnginxinc/kubernetes-ingressは別物である。

主な違いは以下を参照

Aspect or Feature	kubernetes/ingress-nginx	nginxinc/kubernetes-ingress with NGINX	nginxinc/kubernetes-ingress with NGINX Plus
Fundamental
Authors	Kubernetes community	NGINX Inc and community	NGINX Inc and community
NGINX version	Custom NGINX build that includes several third-party modules	NGINX official mainline build	NGINX Plus
Commercial support	N/A	N/A	Included
Load balancing configuration via the Ingress resource
Merging Ingress rules with the same host	Supported	Supported via Mergeable Ingresses	Supported via Mergeable Ingresses
HTTP load balancing extensions - Annotations	See the supported annotations	See the supported annotations	See the supported annotations
HTTP load balancing extensions -- ConfigMap	See the supported ConfigMap keys	See the supported ConfigMap keys	See the supported ConfigMap keys
TCP/UDP	Supported via a ConfigMap	Supported via a ConfigMap with native NGINX configuration	Supported via a ConfigMap with native NGINX configuration
Websocket	Supported	Supported via an annotation	Supported via an annotation
TCP SSL Passthrough	Supported via a ConfigMap	Not supported	Not supported
JWT validation	Not supported	Not supported	Supported
Session persistence	Supported via a third-party module	Not supported	Supported
Canary testing (by header, cookie, weight)	Supported via annotations	Supported via custom resources	Supported via custom resources
Configuration templates *1	See the template	See the templates	See the templates
Load balancing configuration via Custom Resources
HTTP load balancing	Not supported	See VirtualServer and VirtualServerRoute resources.	See VirtualServer and VirtualServerRoute resources.
Deployment
Command-line arguments *2	See the arguments	See the arguments	See the arguments
TLS certificate and key for the default server	Required as a command-line argument/ auto-generated	Required as a command-line argument	Required as a command-line argument
Helm chart	Supported	Supported	Supported
Operational
Reporting the IP address(es) of the Ingress controller into Ingress resources	Supported	Supported	Supported
Extended Status	Supported via a third-party module	Not supported	Supported
Prometheus Integration	Supported	Supported	Supported
Dynamic reconfiguration of endpoints (no configuration reloading)	Supported with a third-party Lua module	Not supported	Supported

環境構築

前提条件

GKEクラスタに使用するSubnetは作成済みとする

GKEクラスタ作成

gcloud beta container --project "PROJECT-NAME" clusters create "nginx-ingress" \
--zone "asia-northeast1-a" --node-locations "asia-northeast1-a","asia-northeast1-b" \
--no-enable-basic-auth --release-channel "regular" --machine-type "n1-standard-1" \
--image-type "COS" --disk-type "pd-standard" --disk-size "100" --metadata disable-legacy-endpoints=true \
--scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
--num-nodes "1" --enable-stackdriver-kubernetes --enable-ip-alias \
--network "projects/PROJECT-NAME/global/networks/sakon-network" --subnetwork "projects/PROJECT-NAME/regions/asia-northeast1/subnetworks/sakon-gke-subnet-1" \
--cluster-secondary-range-name "sakon-gkesub1-2nd-range-2" --services-secondary-range-name "sakon-gkesub1-2nd-range-1" \
--enable-intra-node-visibility --default-max-pods-per-node "110" --addons HorizontalPodAutoscaling,HttpLoadBalancing \
--enable-autoupgrade --enable-autorepair

$ kubectl version --short
Client Version: v1.14.7
Server Version: v1.14.3-gke.11

Tillerのインストール

$ kubectl create serviceaccount --namespace kube-system tiller
serviceaccount/tiller created

$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io/tiller-cluster-rule created

$ helm init --service-account tiller
$HELM_HOME has been configured at /Users/sakonju/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation

tillerがインストールされてるか確認

$ kubectl get deploy -n kube-system
NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
event-exporter-v0.2.5                      1/1     1            1           34h
fluentd-gcp-scaler                         1/1     1            1           34h
heapster-v1.6.1                            1/1     1            1           34h
kube-dns                                   2/2     2            2           34h
kube-dns-autoscaler                        1/1     1            1           34h
l7-default-backend                         1/1     1            1           34h
metrics-server-v0.3.1                      1/1     1            1           34h
stackdriver-metadata-agent-cluster-level   1/1     1            1           34h
tiller-deploy                              1/1     1            1           34h

GKEにアプリケーションをデプロイ

$ kubectl run --restart=Always hello-app --image=gcr.io/google-samples/hello-app:1.0 --port=8080
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/hello-app created

serviceを作って外部公開する

$ kubectl expose deployment hello-app
service/hello-app exposed

$ kubectl get svc -o wide
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE   SELECTOR
hello-app    ClusterIP   10.121.227.201   <none>        8080/TCP   32s   run=hello-app
kubernetes   ClusterIP   10.120.0.1       <none>        443/TCP    34h   <none>

Helmを使用してNGINX Ingress Controllerをdeploy

以下の様な構成となる

nginx-ingressをHelmでdeployする

$ helm install --name nginx-ingress stable/nginx-ingress --set rbac.create=true --set controller.publishService.enabled=true
NAME:   nginx-ingress
LAST DEPLOYED: Sun Oct  6 14:23:12 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                                            READY  STATUS             RESTARTS  AGE
nginx-ingress-controller-6cb795cdc5-r9j5c       0/1    ContainerCreating  0         1s
nginx-ingress-default-backend-6b8dc9d88f-hfv54  0/1    ContainerCreating  0         1s

==> v1/Service
NAME                           TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)                     AGE
nginx-ingress-controller       LoadBalancer  10.122.205.4  <pending>    80:31003/TCP,443:32313/TCP  1s
nginx-ingress-default-backend  ClusterIP     10.121.57.91  <none>       80/TCP                      1s

==> v1/ServiceAccount
NAME                   SECRETS  AGE
nginx-ingress          1        1s
nginx-ingress-backend  1        1s

==> v1beta1/ClusterRole
NAME           AGE
nginx-ingress  1s

==> v1beta1/ClusterRoleBinding
NAME           AGE
nginx-ingress  1s

==> v1beta1/Deployment
NAME                           READY  UP-TO-DATE  AVAILABLE  AGE
nginx-ingress-controller       0/1    1           0          1s
nginx-ingress-default-backend  0/1    1           0          1s

==> v1beta1/Role
NAME           AGE
nginx-ingress  1s

==> v1beta1/RoleBinding
NAME           AGE
nginx-ingress  1s


NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w nginx-ingress-controller'

An example Ingress that makes use of the controller:

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

↓L4ロードバランサが作成される

==> v1/Service
NAME                           TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)                     AGE
nginx-ingress-controller       LoadBalancer  10.122.205.4  <pending>    80:31003/TCP,443:32313/TCP  1s
nginx-ingress-default-backend  ClusterIP     10.121.57.91  <none>       80/TCP                      1s

しばらくするとEXTERNAL-IPも取得されている。

$ kubectl get service
NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
hello-app                       ClusterIP      10.121.227.201   <none>         8080/TCP                     19h
kubernetes                      ClusterIP      10.120.0.1       <none>         443/TCP                      2d5h
nginx-ingress-controller        LoadBalancer   10.122.205.4     34.84.181.58   80:31003/TCP,443:32313/TCP   2m44s
nginx-ingress-default-backend   ClusterIP      10.121.57.91     <none>         80/TCP                       2m44s

2番目のサービスである、 nginx-ingress-default-backendに注目。default-backendは全てのURLパスを処理し、NGINXコントローラが理解できないホスト(つまり、Ingressリソースにマッピングされていない全てのリクエスト)をホストするサービス。デフォルトのバックエンドは2つのURLを公開する。

/healthz 200を返す
/ 404を返す

$ curl -v http://34.84.181.58/healthz
*   Trying 34.84.181.58...
* TCP_NODELAY set
* Connected to 34.84.181.58 (34.84.181.58) port 80 (#0)
> GET /healthz HTTP/1.1
> Host: 34.84.181.58
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.1
< Date: Sun, 06 Oct 2019 06:32:28 GMT
< Content-Type: text/html
< Content-Length: 0
< Connection: keep-alive
<

$ curl -v http://34.84.181.58/
*   Trying 34.84.181.58...
* TCP_NODELAY set
* Connected to 34.84.181.58 (34.84.181.58) port 80 (#0)
> GET / HTTP/1.1
> Host: 34.84.181.58
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: openresty/1.15.8.1
< Date: Sun, 06 Oct 2019 06:41:45 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 21
< Connection: keep-alive
<
* Connection #0 to host 34.84.181.58 left intact

NGINX Ingress Controllerを使用するためのIngressリソースの構成

GCPの場合、annotationで定義が無い場合はGCLBが作成される。

NGINXの場合
annotations: kubernetes.io/ingress.class: nginx
GCLBの場合
annotations: kubernetes.io/ingress.class: gce

ingress-resource.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - path: /hello
        backend:
          serviceName: hello-app
          servicePort: 8080

applyする

$ kubectl apply -f ingress-resource.yaml
ingress.networking.k8s.io/ingress-resource created

作成されたリソースを確認

$ kubectl describe ing
Name:             ingress-resource
Namespace:        default
Address:          34.84.181.58
Default backend:  default-http-backend:80 (10.124.0.9:8080)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *
        /hello   hello-app:8080 (10.124.1.11:8080)
Annotations:
  kubernetes.io/ingress.class:                       nginx
  nginx.ingress.kubernetes.io/ssl-redirect:          false
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/ssl-redirect":"false"},"name":"ingress-resource","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"hello-app","servicePort":8080},"path":"/hello"}]}}]}}

Events:  <none>

アクセスしてみる

/helloではバックエンドのwebアプリケーションにアクセス出来るようになっている！

$ curl http://34.84.181.58/hello
Hello, world!
Version: 1.0.0
Hostname: hello-app-84d9648f89-vzh5v

/hello以外だと、404を返すので、デフォルトバックエンドサービスが適切に機能している。

$ curl http://34.84.181.58/test
default backend - 404

You get articles that match your needs
You can efficiently read back useful information
You can use dark theme

What you can do with signing up