はじめに

MirantisKubernetes を構築にあたりクラスタ構成ファイルを作成します。
前２回では予めクラスタ構成ファイルを作成しましたが、今回は設定項目について確認したいと思います。

インストール時の設定項目

launchpad.yaml の設定項目は以下から確認できます。

Mirantis Launchpad Configuration File
https://github.com/Mirantis/launchpad/blob/master/docs/configuration-file.md

よく使いそうなパラメータをピックアップして説明します。
※今回は launchpad 1.1.1 での設定項目となります

共通：共通の設定

kind：MKE（k8s）のみであれば mke、MSR（Registory）も利用するのであれば mke+msr で設定します
metadata
- name：クラスタの名前です。今の所、ログやクライアント設定などの内部的に利用されます

ホスト（hosts）：各ホストの設定

address：各のノードに接続する際のIPアドレスです
privateInterface：ネットワークを構成する際のNICが指定できます（初期値：eth0）
role：クラスタ内の役割を指定できます（manager、worker、msr）
engineConfig：下階層で Docker Engine の設定ファイルとしてマッピングされます(daemon.json に変換)
上記の URL の例ではデバッグモードの指定とログオプション（ファイルサイズとローテーション）を指定しています
ssh：接続の方式が SSH だった場合の設定です
- user：SSH 接続する際のユーザ名です（初期値：root）
- port：SSH 接続する際のポート番号です（初期値：22）
- keypath：SSH 接続する際の秘密鍵のパスです（初期値：~/.ssh/id_rsa）

MKE（mke）：MKE の設定

version：インストールする MKE のバージョン（初期値：3.3.3）
adminUsername：管理者アカウントの名前（初期値：admin）
adminPassword：管理者アカウントのパスワード（初期値：指定なしでランダムな文字列を生成）
installFlags：InstallFlags で説明
configFile：config.toml で説明
configData：config.toml で説明

DockerEngine（engine）：DockerEngine の設定

version：DockerEngineのバージョン

InstallFlags

installFlags の項目を設定することで、詳細な設定をすることが出来ます。
項目は docker がインストールされている VM で以下を実行して確認できます。

$ docker run -t -i --rm docker/ucp:3.3.0 install --help
Docker Universal Control Plane Tool

   install - Install UCP on this node

USAGE:
   docker run --rm -it \
        -v /var/run/docker.sock:/var/run/docker.sock \
        docker/ucp \
        install [command options]

DESCRIPTION:

This command initializes a new swarm, turns this node into a manager, and installs
Docker Universal Control Plane (UCP).

When installing UCP you can customize:

  * The certificates used by the UCP web server. Create a volume
    named 'ucp-controller-server-certs' and copy the ca.pem, cert.pem, and key.pem
    files to the root directory. Then run the install command with the
    '--external-server-cert' flag.

  * The license used by UCP, by bind-mounting the file at
    '/config/docker_subscription.lic' in the tool.  E.g. -v /path/to/my/config/docker_subscription.lic:/config/docker_subscription.lic
    or by specifying with '--license "$(cat license.lic)"

If you're joining more nodes to this swarm, open the following ports in your
firewall:

  * 443 or the '--controller-port'
  * 2376 or the '--swarm-port'
  * 2377 or the swarm-mode listen port
  * 179, 6443, 6444, 10250, 12376, 12378 - 12386
  * 4789(udp) and 7946(tcp/udp) for overlay networking

If you have SELinux policies enabled for your Docker install, you will need to
use 'docker run --rm -it --security-opt label=disable ...' when running this
command.


OPTIONS:
   --debug, -D                        Enable debug mode
   --jsonlog                          Produce json formatted output for easier parsing
   --interactive, -i                  Run in interactive mode and prompt for configuration values
   --admin-username value             The UCP administrator username [$UCP_ADMIN_USER]
   --admin-password value             The UCP administrator password [$UCP_ADMIN_PASSWORD]
   --swarm-grpc-port value            Port for communication between nodes (default: 2377)
   --san value                        Add subject alternative names to certificates (e.g. --san www1.acme.com --san www2.acme.com) [$UCP_HOSTNAMES]
   --host-address value               The network address to advertise to other nodes. Format: IP address or network interface name [$UCP_HOST_ADDRESS]
   --data-path-addr value             Address or interface to use for data path traffic. Format: IP address or network interface name [$UCP_DATA_PATH_ADDR]
   --jwk-url value                    JWK url of external OpenID provider [$JWK_URL]
   --force-minimums                   Force the install/upgrade even if the system does not meet the minimum requirements
   --controller-port value            Port for the web UI and API (default: 443)
   --kube-apiserver-port value        Port for the Kubernetes API server (default: 6443)
   --swarm-port value                 Port for the Docker Swarm manager. Used for backwards compatibility (default: 2376)
   --cni-installer-url value          A URL pointing to a kubernetes YAML file to be used as an installer for the CNI plugin of the cluster. If specified, the default CNI plugin will not be installed. If the URL is using the HTTPS scheme, no certificate verification will be performed
   --unmanaged-cni                    Flag to indicate if cni provider is calico and managed by UCP (calico is the default CNI provider)
   --secure-overlay                   Flag to enable IPSec network encryption using SecureOverlay in Kubernetes
   --calico-vxlan                     Flag to indicate if calico CNI should choose VXLAN as default dataplane. Default is vxlan
   --istio                            Flag to indicate if Istio Ingress should be enabled upon startup
   --vxlan-vni value                  Flag allows to choose vxlan-vni ID (10000-20000) if VXLAN dataplane mode is selected (Default: 10000) (default: 10000)
   --pod-cidr value                   Kubernetes cluster IP pool for the pods to allocated IP from (Default: 192.168.0.0/16) (default: "192.168.0.0/16")
   --service-cluster-ip-range value   Kubernetes Cluster IP Range for Services (Default: 10.96.0.0/16) (default: "10.96.0.0/16")
   --nodeport-range value             Allowed port range for Kubernetes services of type NodePort (Default: 32768-35535) (default: "32768-35535")
   --cloud-provider value             The cloud provider for the cluster
   --default-node-orchestrator value  The default orchestrator for the nodes in cluster. Can be either swarm or kubernetes. By default, UCP nodes use swarm as the orchestrator.
   --azure-ip-count value             Configure the Number of IP Address to be provisioned for each Azure Virtual Machine (Default: 64) (default: "64")
   --storage-iscsi                    Enable ISCSI based Persistent Volumes in Kubernetes
   --iscsiadm-path value              Absolute path to host iscsiadm binary. Option applicable only when --storage-iscsi is specified. Symlinks not allowed
   --iscsidb-path value               Absolute path to host iscsi DB. Option applicable only when --storage-iscsi is specified. Symlinks not allowed
   --storage-expt-enabled             Flag to enable experimental features in Kubernetes storage
   --dns value                        Set custom DNS servers for the UCP containers [$DNS]
   --dns-opt value                    Set DNS options for the UCP containers [$DNS_OPT]
   --dns-search value                 Set custom DNS search domains for the UCP containers [$DNS_SEARCH]
   --unlock-key value                 The unlock key for this swarm-mode cluster, if one exists. [$UNLOCK_KEY]
   --existing-config                  Use the latest existing UCP config during this installation. The install will fail if a config is not found
   --skip-cloud-provider-check        Disables checks which rely on detecting which (if any) cloud provider the cluster is currently running on
   --pull value                       Pull UCP images: 'always', when 'missing', or 'never' (default: "missing")
   --registry-username value          Username to use when pulling images [$REGISTRY_USERNAME]
   --registry-password value          Password to use when pulling images [$REGISTRY_PASSWORD]
   --kv-timeout value                 Timeout in milliseconds for the key-value store (default: 5000) [$KV_TIMEOUT]
   --kv-snapshot-count value          Number of changes between key-value store snapshots (default: 20000) [$KV_SNAPSHOT_COUNT]
   --disable-tracking                 Disable anonymous tracking and analytics
   --disable-usage                    Disable anonymous usage reporting
   --external-server-cert             Customize the certificates used by the UCP web server
   --preserve-certs                   Don't generate certificates if they already exist
   --binpack                          Set the Docker Swarm scheduler to binpack mode. Used for backwards compatibility
   --random                           Set the Docker Swarm scheduler to random mode. Used for backwards compatibility
   --external-service-lb value        Set the IP address of the load balancer that published services are expected to be reachable on
   --enable-profiling                 Enable performance profiling
   --license value                    Add a license: e.g. --license "$(cat license.lic)" [$UCP_LICENSE]
   --force-insecure-tcp               Force install to continue even with unauthenticated Docker Engine ports.

config.toml

MKE のインストール設定は他にも設定ファイルとして渡すことも出来ます。
以下コマンドでサンプルファイルを作成し、launchpad.yaml でパスを指定します。
※設定ファイル内の値を「configData」に直接渡すことも出来ます
※項目の説明は以下のページでも説明されています
https://docs.mirantis.com/docker-enterprise/v3.1/dockeree-products/mke/mke-configure/mke-configuration-file.html

shell-sessionconfig.toml作成

$ docker container run --rm mirantis/ucp:3.3.2 example-config

config.toml

# Example UCP configuration file
[auth]

  # The name of the auth backend to use, either "managed" or "ldap".
  backend = "managed"

  # NOTE: LDAP options may only be configured via the UCP UI or API.

  # Indicates if SAML is enabled on this UCP.
  samlEnabled = false

  # The customized SAML login button text
  samlLoginText = ""

  # Indicates if SCIM is enabled on this UCP.
  scimEnabled = false

  # Indicates if managed password is disabled.
  managedPasswordDisabled = false

  # The fallback user when the managed password authentication is disabled.
  managedPasswordFallbackUser = ""

  # The role that new users will get for their private collections. Can be
  # "admin", "viewonly", "scheduler", "restrictedcontrol", or "fullcontrol".
  default_new_user_role="restrictedcontrol"

  # JSON Web Keys url of external OpenID provider.
  jwkUrl="http:///www.example.service/openid-connect/certs"

  [auth.sessions]
    # Specifies the initial lifetime (in minutes) of a session from the moment
    # it is generated.
    lifetime_minutes = 60

    # Indicates a period of time (in minutes) before the expiration of a session
    # where, if used, a session will be extended by the current configured lifetime
    # from then, a zero value disables session extension.
    renewal_threshold_minutes = 20

    # Indicates the maximum number of sessions that any user can have active at any
    # given time. If creating a new session would put a user over this limit then
    # the least recently used session will be deleted. A value of zero disables
    # limiting the number of sessions that users may have.
    per_user_limit = 10

    # Set token to be stored in sessionStorage instead of localStorage
    store_token_per_session = false

  # SAML related settings
  [auth.saml]

    # Service Provider Host
    spHost = "52.38.7.31"

    # Identity Provider URL
    idpMetadataURL = "https://www.testshib.org/metadata/testshib-providers.xml"

    # Root Certs for IdP metadata
    rootCerts = ""

    # TLS Skip verify for IdP Metadata
    tlsSkipVerify = false

  # SCIM related settings
  [auth.scim]

    # SCIM API token
    token = "4e102f46-6258-4501-9a54-4e5993fabcf5"

    # Organization name
    orgName = "docker-datacenter"

[[registries]]
  # The address used to connect to the DTR instance tied to this UCP cluster
  host_address = ""

  # The DTR instance's OpenID Connect Client ID, as registered with our auth provider.
  service_id = ""

  # The root CA bundle for the DTR instance (if using a custom CA).
  ca_bundle = ""

  # Whether or not this DTR can look up image scanning data in batches
  batch_scanning_data_enabled = true

[scheduling_configuration]

  # Allow admins to schedule containers on managers
  # Set to true to allow admins to schedule on manager
  enable_admin_ucp_scheduling = true

  # The type of orchestrator to use for new nodes that are joined to the
  # cluster. Can be either "swarm" or "kubernetes".
  default_node_orchestrator = "swarm"


[tracking_configuration]

  # Disable analytics of usage information
  # Set to true to disable analytics
  disable_usageinfo = false

  # Disable analytics of API call information
  # Set to true to disable analytics
  disable_tracking = false

  # Anonymize analytic data
  # Set to true to hide your license ID
  anonymize_tracking = false

  # Set an optional label to be included with analytics that are sent to
  # Docker indicating whether this UCP cluster is a production cluster or a
  # development cluster. This allows Docker to analyze the data in UCP
  # analytics more accurately.
  cluster_label = "prod"


[trust_configuration]

  # Require images be signed by content trust
  require_content_trust = false

  # Specify users or teams which must sign images
  require_signature_from = ["team1", "team2"]


[log_configuration]

  # Set the logging level for UCP components - uses syslog levels
  level = "DEBUG"

[audit_log_configuration]

  # Specify the audit logging level. Leave empty for disabling audit logs. Other
  # legal values are "metadata" and "request". TODO: add link to level descriptions
  level = ""

  # When set to true, support dumps will include audit logs in the logs
  # of the ucp-controller container of each manager node.
  support_dump_include_audit_logs = false

[license_configuration]

  # Enable attempted automatic license renewal when the license nears expiration
  # If disabled, you must manually upload renewed licesnse after expiration.
  auto_refresh = true

[[custom_api_server_headers]]
  # The name of the HTTP header.
  name = "X-Some-Custom-Name"
  # The value of the HTTP header.
  value = "Some Custom Value"

[pki]

  # A custom Root CA bundle to use for client TLS certificate authentication in
  # addition to UCP's internal cluster CAs.
  client_ca_bundle = ""

[user_workload_defaults]
  # A map describing default values to set on Swarm services at creation time
  # if those fields are not explicitly set in the service spec.
  [user_workload_defaults.swarm_defaults]
    [user_workload_defaults.swarm_defaults."tasktemplate.restartpolicy.delay"]
      value = "5m"
    [user_workload_defaults.swarm_defaults."tasktemplate.restartpolicy.maxattempts"]
      value = "3"

[cluster_config]

  # Configures the port the ucp-controller listens to
  controller_port = 443

  # Configures the port the Kubernetes API server listens to
  kube_apiserver_port = 6443

  # Configures the port the ucp-swarm-manager listens to
  swarm_port = 2376

  # Configures Swarm scheduler strategy for container scheduling
  # This does not affect swarm-mode services
  swarm_strategy = "spread"

  # Configures DNS settings for the UCP components
  dns = []
  dns_opt = []
  dns_search = []

  # Turn on specialized debugging endpoints for profiling UCP performance
  profiling_enabled = false

  # Turn off Swarm API polling to increase performance of UCP
  swarm_polling_disabled = false

  # Enable aggregation of DTR image vulnerability scans
  image_scan_aggregation_enabled = false

  # Tune the KV store timeout and snapshot settings
  kv_timeout = 5000
  kv_snapshot_count = 20000

  # Specify an optional external LB for default links to services with expose ports in the UI
  external_service_lb = ""

  # Specify the URL of a kubernetes YAML file to be used for installing a CNI
  # plugin. Only applies during initial installation. If empty, the default CNI
  # plugin will be used
  cni_installer_url = ""

  # Set to true if ucp should not manage the cni provider lifecycle.
  # UCP only manages Calico plugin if the unmanaged-flag is set to false.
  unmanaged_cni = false

  # Set to true if ucp should deploy secure-overlay which enables IPsec
  # Network Encryption in Kubernetes
  secure_overlay = false

  # Set to true if cluster should use vxlan as dataplane.
  calico_vxlan = false

  # Set VXLAN VNI id for vxlan data plane.
  vxlan_vni = 10000

  # Adjust the metrics retention time
  metrics_retention_time = ""

  # Set the interval for how frequently managers gather metrics from nodes in the cluster
  metrics_scrape_interval = "1m"

  # Set the size of the cache used by UCP's RethinkDB servers. The installed
  # default is 1GB, but leaving this field empty or specifying the special
  # value "auto" will instruct RethinkDB to determine a cache size
  # automatically (about half of currently available memory) which ensures a
  # reliable operation in most scenarios, but the cache size can be tuned
  # manually to accommodate increased memory usage by other processes or to
  # maximize query performance. A larger cache improves the database’s
  # performance, but you must consider other sources of memory consumption to
  # avoid out of memory conditions.
  # https://www.rethinkdb.com/docs/memory-usage/#cache-size
  rethinkdb_cache_size = "16GB"

  # If true, disable headers in UCP API responses that indicate which
  # UCP controller is returning the response
  exclude_server_identity_headers = true

  # Set the cloud provider for the kubernetes cluster. Depending on the cloud provider
  # different configurations to kubernetes plugin might be need.For eg This could be used
  # for cases where we need specific CNI plugin depending on the cloud provider to support
  # pod networking
  cloud_provider = ""

  # Enable the cluster's ISCSI storage backend. Enabling this prepares the
  # Kubernetes worker nodes for creation and management of ISCSI based
  # persistent volumes.
  iscsi_enabled = false

  # iscsiadm is the absolute path to the host's iscsiadm binary. This is used when
  # iscsi_enabled is set to true. This is necessary for the containerized
  # kubelet to act as a iscsi initiator. Defaults to "/usr/sbin/iscsiadm"
  # Symlinks not allowed.
  iscsiadm_path = ""

  # iscsidb is the absolute path to the host's iscsi database. This is used when
  # iscsi_enabled is set to true. This is necessary for the containerized
  # kubelet to act as a iscsi initiator. Defaults to "/etc/iscsi". Symlinks not allowed.
  iscsidb_path = ""

  # Set cluster Pod subnet pool. This will be the subnet pool from which the IP for the pod
  # would be allocated by the CNI ipam plugin
  pod_cidr = "192.168.0.0/16"

  # storage_expt_enabled flag is used to enable experimental storage features in the cluster.
  storage_expt_enabled = false

  # Set the MTU (maximum transmission unit) size for the Calico plugin.
  calico_mtu = "1480"

  # Set the VXLAN MTU (maximum transmission unit) size for the Calico plugin.
  calico_vxlan_mtu = "1450"

  # Set the VXLAN Port for the Calico plugin.
  calico_vxlan_port = "4789"

  # Set the IPIP MTU size for the calico IPIP tunnel interface
  ipip_mtu = "1480"

  # Set the IP count for azure allocator to allocate IPs per azure VM
  azure_ip_count = "64"

  # Set the ServiceClusterIPRange used by Kubernates for Services
  service_cluster_ip_range = "10.96.0.0/16"

  # Set the port range that for Kubernetes services of type NodePort can be
  # exposed in.
  nodeport_range = "32768-35535"

  # Enable the use of a KMS (Key Management Service) plugin. The KMS plugin
  # must be deployed on every manager node. UCP will configure the API server
  # with the specified endpoint socket.
  kms_enabled = false

  # Name of the KMS resource
  kms_name = ""

  # Host path of the KMS endpoint socket
  kms_endpoint = ""

  # Number of data encryption keys to be cached in the clear by the KMS.
  kms_cachesize = 0

  # ADVANCED: Set any additional flags on the Kubernetes system components that
  # you may require. You must ensure that any custom flags do not conflict with
  # configuration options already set by UCP.
  custom_kube_api_server_flags = ["--event-ttl=1h0m0s", "--service-node-port-range=30000-32767"]
  custom_kube_controller_manager_flags = ["--service-sync-period=5m0s"]
  custom_kubelet_flags = ["--http-check-frequency=20s", "--kube-api-qps=200", "--kube-api-burst=200"]
  custom_kube_scheduler_flags = ["--algorithm-provider=DefaultProvider", "--kube-api-qps=200", "--kube-api-burst=200"]
  custom_kube_proxy_flags = ["--conntrack-min=300000", "--conntrack-max-per-core=200000"]

  # Store data about collections for volumes in UCP's local KV store instead of
  # on the volume labels. This is used for enforcing access control on volumes.
  # Most users will not need to turn this on; this is only necessary if you
  # use third-party volume drivers.
  local_volume_collection_mapping = false

  # Reserve resources for Docker UCP and Kubernetes components which are
  # running on manager nodes.
  # For more details, see:
  #   https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#kube-reserved
  manager_kube_reserved_resources = "cpu=1,memory=2Gi,ephemeral-storage=4Gi"

  # Reserve resources for Docker UCP and Kubernetes components which are
  # running on worker nodes.
  # For more details, see:
  #   https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#kube-reserved
  worker_kube_reserved_resources = "cpu=100m,memory=300Mi,ephemeral-storage=500Mi"

  # Set Number of Pods that can run on this Kubelet.
  kubelet_max_pods = 110

  # Set pre logon message to alert users before they proceed with login.
  pre_logon_message = ""

  # Enable Windows GMSA support in Kubernetes. Enabling this deploys the webhook
  # admission controller, and creates the Kubernetes resources, required for GMSA
  # workloads in Kubernetes.
  windows_gmsa = false

  [cluster_config.service_mesh]
    # Disable Istio Ingress and Service Mesh
    enabled = false

    # Set the Number of Ingress Replicas
    ingress_num_replicas = 2

    # Set the List of External IPs for Ingress Gateway Service
    ingress_external_ips = []

    # Enable external load balancer
    ingress_enable_lb = false

    # Enable preserving inbound traffic source IP
    ingress_preserve_client_ip = false

    # Set exposed ports
    [[cluster_config.service_mesh.ingress_exposed_ports]]
      name = "name"
      port = 1
      target_port = 2
      node_port = 3

    # Set node affinity
    [cluster_config.service_mesh.ingress_node_affinity]
       nodetype = "ingress"

    # Set node toleration
    [[cluster_config.service_mesh.ingress_node_toleration]]
      key = "com.docker.ucp.manager"
      value = "dummy"
      operator = "Exists"
      effect = "NoSchedule"

まとめ

今回はインストール時の設定方法、設定値を確認してみました。
必須項目からコアな設定もあるため、必要なところから設定をしていき適宜設定を適用する形がよいのではと思います。