はじめに
MirantisKubernetes を構築にあたりクラスタ構成ファイルを作成します。
前２回では予めクラスタ構成ファイルを作成しましたが、今回は設定項目について確認したいと思います。
インストール時の設定項目
launchpad.yaml の設定項目は以下から確認できます。
Mirantis Launchpad Configuration File
https://github.com/Mirantis/launchpad/blob/master/docs/configuration-file.md
よく使いそうなパラメータをピックアップして説明します。
※今回は launchpad 1.1.1 での設定項目となります
共通：共通の設定
- kind：MKE（k8s）のみであれば
mke、MSR（Registory）も利用するのであればmke+msrで設定します - metadata
- name：クラスタの名前です。今の所、ログやクライアント設定などの内部的に利用されます
ホスト（hosts）：各ホストの設定
- address：各のノードに接続する際のIPアドレスです
- privateInterface：ネットワークを構成する際のNICが指定できます（初期値：
eth0） - role：クラスタ内の役割を指定できます（
manager、worker、msr） - engineConfig：下階層で Docker Engine の 設定ファイルとしてマッピングされます(daemon.json に変換)
上記の URL の例ではデバッグモードの指定とログオプション（ファイルサイズとローテーション）を指定しています - ssh：接続の方式が SSH だった場合の設定です
- user：SSH 接続する際のユーザ名です（初期値：
root） - port：SSH 接続する際のポート番号です（初期値：22）
- keypath：SSH 接続する際の秘密鍵のパスです（初期値：
~/.ssh/id_rsa）
- user：SSH 接続する際のユーザ名です（初期値：
MKE（mke）：MKE の設定
- version：インストールする MKE のバージョン（初期値：3.3.3）
- adminUsername：管理者アカウントの名前（初期値：admin）
- adminPassword：管理者アカウントのパスワード（初期値：指定なしでランダムな文字列を生成）
- installFlags：InstallFlags で説明
- configFile：config.toml で説明
- configData：config.toml で説明
DockerEngine（engine）：DockerEngine の設定
- version：DockerEngineのバージョン
InstallFlags
installFlags の項目を設定することで、詳細な設定をすることが出来ます。
項目は docker がインストールされている VM で以下を実行して確認できます。
$ docker run -t -i --rm docker/ucp:3.3.0 install --help
Docker Universal Control Plane Tool
install - Install UCP on this node
USAGE:
docker run --rm -it \
-v /var/run/docker.sock:/var/run/docker.sock \
docker/ucp \
install [command options]
DESCRIPTION:
This command initializes a new swarm, turns this node into a manager, and installs
Docker Universal Control Plane (UCP).
When installing UCP you can customize:
* The certificates used by the UCP web server. Create a volume
named 'ucp-controller-server-certs' and copy the ca.pem, cert.pem, and key.pem
files to the root directory. Then run the install command with the
'--external-server-cert' flag.
* The license used by UCP, by bind-mounting the file at
'/config/docker_subscription.lic' in the tool. E.g. -v /path/to/my/config/docker_subscription.lic:/config/docker_subscription.lic
or by specifying with '--license "$(cat license.lic)"
If you're joining more nodes to this swarm, open the following ports in your
firewall:
* 443 or the '--controller-port'
* 2376 or the '--swarm-port'
* 2377 or the swarm-mode listen port
* 179, 6443, 6444, 10250, 12376, 12378 - 12386
* 4789(udp) and 7946(tcp/udp) for overlay networking
If you have SELinux policies enabled for your Docker install, you will need to
use 'docker run --rm -it --security-opt label=disable ...' when running this
command.
OPTIONS:
--debug, -D Enable debug mode
--jsonlog Produce json formatted output for easier parsing
--interactive, -i Run in interactive mode and prompt for configuration values
--admin-username value The UCP administrator username [$UCP_ADMIN_USER]
--admin-password value The UCP administrator password [$UCP_ADMIN_PASSWORD]
--swarm-grpc-port value Port for communication between nodes (default: 2377)
--san value Add subject alternative names to certificates (e.g. --san www1.acme.com --san www2.acme.com) [$UCP_HOSTNAMES]
--host-address value The network address to advertise to other nodes. Format: IP address or network interface name [$UCP_HOST_ADDRESS]
--data-path-addr value Address or interface to use for data path traffic. Format: IP address or network interface name [$UCP_DATA_PATH_ADDR]
--jwk-url value JWK url of external OpenID provider [$JWK_URL]
--force-minimums Force the install/upgrade even if the system does not meet the minimum requirements
--controller-port value Port for the web UI and API (default: 443)
--kube-apiserver-port value Port for the Kubernetes API server (default: 6443)
--swarm-port value Port for the Docker Swarm manager. Used for backwards compatibility (default: 2376)
--cni-installer-url value A URL pointing to a kubernetes YAML file to be used as an installer for the CNI plugin of the cluster. If specified, the default CNI plugin will not be installed. If the URL is using the HTTPS scheme, no certificate verification will be performed
--unmanaged-cni Flag to indicate if cni provider is calico and managed by UCP (calico is the default CNI provider)
--secure-overlay Flag to enable IPSec network encryption using SecureOverlay in Kubernetes
--calico-vxlan Flag to indicate if calico CNI should choose VXLAN as default dataplane. Default is vxlan
--istio Flag to indicate if Istio Ingress should be enabled upon startup
--vxlan-vni value Flag allows to choose vxlan-vni ID (10000-20000) if VXLAN dataplane mode is selected (Default: 10000) (default: 10000)
--pod-cidr value Kubernetes cluster IP pool for the pods to allocated IP from (Default: 192.168.0.0/16) (default: "192.168.0.0/16")
--service-cluster-ip-range value Kubernetes Cluster IP Range for Services (Default: 10.96.0.0/16) (default: "10.96.0.0/16")
--nodeport-range value Allowed port range for Kubernetes services of type NodePort (Default: 32768-35535) (default: "32768-35535")
--cloud-provider value The cloud provider for the cluster
--default-node-orchestrator value The default orchestrator for the nodes in cluster. Can be either swarm or kubernetes. By default, UCP nodes use swarm as the orchestrator.
--azure-ip-count value Configure the Number of IP Address to be provisioned for each Azure Virtual Machine (Default: 64) (default: "64")
--storage-iscsi Enable ISCSI based Persistent Volumes in Kubernetes
--iscsiadm-path value Absolute path to host iscsiadm binary. Option applicable only when --storage-iscsi is specified. Symlinks not allowed
--iscsidb-path value Absolute path to host iscsi DB. Option applicable only when --storage-iscsi is specified. Symlinks not allowed
--storage-expt-enabled Flag to enable experimental features in Kubernetes storage
--dns value Set custom DNS servers for the UCP containers [$DNS]
--dns-opt value Set DNS options for the UCP containers [$DNS_OPT]
--dns-search value Set custom DNS search domains for the UCP containers [$DNS_SEARCH]
--unlock-key value The unlock key for this swarm-mode cluster, if one exists. [$UNLOCK_KEY]
--existing-config Use the latest existing UCP config during this installation. The install will fail if a config is not found
--skip-cloud-provider-check Disables checks which rely on detecting which (if any) cloud provider the cluster is currently running on
--pull value Pull UCP images: 'always', when 'missing', or 'never' (default: "missing")
--registry-username value Username to use when pulling images [$REGISTRY_USERNAME]
--registry-password value Password to use when pulling images [$REGISTRY_PASSWORD]
--kv-timeout value Timeout in milliseconds for the key-value store (default: 5000) [$KV_TIMEOUT]
--kv-snapshot-count value Number of changes between key-value store snapshots (default: 20000) [$KV_SNAPSHOT_COUNT]
--disable-tracking Disable anonymous tracking and analytics
--disable-usage Disable anonymous usage reporting
--external-server-cert Customize the certificates used by the UCP web server
--preserve-certs Don't generate certificates if they already exist
--binpack Set the Docker Swarm scheduler to binpack mode. Used for backwards compatibility
--random Set the Docker Swarm scheduler to random mode. Used for backwards compatibility
--external-service-lb value Set the IP address of the load balancer that published services are expected to be reachable on
--enable-profiling Enable performance profiling
--license value Add a license: e.g. --license "$(cat license.lic)" [$UCP_LICENSE]
--force-insecure-tcp Force install to continue even with unauthenticated Docker Engine ports.
config.toml
MKE のインストール設定は他にも設定ファイルとして渡すことも出来ます。
以下コマンドでサンプルファイルを作成し、launchpad.yaml でパスを指定します。
※設定ファイル内の値を「configData」に直接渡すことも出来ます
※項目の説明は以下のページでも説明されています
https://docs.mirantis.com/docker-enterprise/v3.1/dockeree-products/mke/mke-configure/mke-configuration-file.html
shell-sessionconfig.toml作成
$ docker container run --rm mirantis/ucp:3.3.2 example-config
config.toml
# Example UCP configuration file
[auth]
# The name of the auth backend to use, either "managed" or "ldap".
backend = "managed"
# NOTE: LDAP options may only be configured via the UCP UI or API.
# Indicates if SAML is enabled on this UCP.
samlEnabled = false
# The customized SAML login button text
samlLoginText = ""
# Indicates if SCIM is enabled on this UCP.
scimEnabled = false
# Indicates if managed password is disabled.
managedPasswordDisabled = false
# The fallback user when the managed password authentication is disabled.
managedPasswordFallbackUser = ""
# The role that new users will get for their private collections. Can be
# "admin", "viewonly", "scheduler", "restrictedcontrol", or "fullcontrol".
default_new_user_role="restrictedcontrol"
# JSON Web Keys url of external OpenID provider.
jwkUrl="http:///www.example.service/openid-connect/certs"
[auth.sessions]
# Specifies the initial lifetime (in minutes) of a session from the moment
# it is generated.
lifetime_minutes = 60
# Indicates a period of time (in minutes) before the expiration of a session
# where, if used, a session will be extended by the current configured lifetime
# from then, a zero value disables session extension.
renewal_threshold_minutes = 20
# Indicates the maximum number of sessions that any user can have active at any
# given time. If creating a new session would put a user over this limit then
# the least recently used session will be deleted. A value of zero disables
# limiting the number of sessions that users may have.
per_user_limit = 10
# Set token to be stored in sessionStorage instead of localStorage
store_token_per_session = false
# SAML related settings
[auth.saml]
# Service Provider Host
spHost = "52.38.7.31"
# Identity Provider URL
idpMetadataURL = "https://www.testshib.org/metadata/testshib-providers.xml"
# Root Certs for IdP metadata
rootCerts = ""
# TLS Skip verify for IdP Metadata
tlsSkipVerify = false
# SCIM related settings
[auth.scim]
# SCIM API token
token = "4e102f46-6258-4501-9a54-4e5993fabcf5"
# Organization name
orgName = "docker-datacenter"
[[registries]]
# The address used to connect to the DTR instance tied to this UCP cluster
host_address = ""
# The DTR instance's OpenID Connect Client ID, as registered with our auth provider.
service_id = ""
# The root CA bundle for the DTR instance (if using a custom CA).
ca_bundle = ""
# Whether or not this DTR can look up image scanning data in batches
batch_scanning_data_enabled = true
[scheduling_configuration]
# Allow admins to schedule containers on managers
# Set to true to allow admins to schedule on manager
enable_admin_ucp_scheduling = true
# The type of orchestrator to use for new nodes that are joined to the
# cluster. Can be either "swarm" or "kubernetes".
default_node_orchestrator = "swarm"
[tracking_configuration]
# Disable analytics of usage information
# Set to true to disable analytics
disable_usageinfo = false
# Disable analytics of API call information
# Set to true to disable analytics
disable_tracking = false
# Anonymize analytic data
# Set to true to hide your license ID
anonymize_tracking = false
# Set an optional label to be included with analytics that are sent to
# Docker indicating whether this UCP cluster is a production cluster or a
# development cluster. This allows Docker to analyze the data in UCP
# analytics more accurately.
cluster_label = "prod"
[trust_configuration]
# Require images be signed by content trust
require_content_trust = false
# Specify users or teams which must sign images
require_signature_from = ["team1", "team2"]
[log_configuration]
# Set the logging level for UCP components - uses syslog levels
level = "DEBUG"
[audit_log_configuration]
# Specify the audit logging level. Leave empty for disabling audit logs. Other
# legal values are "metadata" and "request". TODO: add link to level descriptions
level = ""
# When set to true, support dumps will include audit logs in the logs
# of the ucp-controller container of each manager node.
support_dump_include_audit_logs = false
[license_configuration]
# Enable attempted automatic license renewal when the license nears expiration
# If disabled, you must manually upload renewed licesnse after expiration.
auto_refresh = true
[[custom_api_server_headers]]
# The name of the HTTP header.
name = "X-Some-Custom-Name"
# The value of the HTTP header.
value = "Some Custom Value"
[pki]
# A custom Root CA bundle to use for client TLS certificate authentication in
# addition to UCP's internal cluster CAs.
client_ca_bundle = ""
[user_workload_defaults]
# A map describing default values to set on Swarm services at creation time
# if those fields are not explicitly set in the service spec.
[user_workload_defaults.swarm_defaults]
[user_workload_defaults.swarm_defaults."tasktemplate.restartpolicy.delay"]
value = "5m"
[user_workload_defaults.swarm_defaults."tasktemplate.restartpolicy.maxattempts"]
value = "3"
[cluster_config]
# Configures the port the ucp-controller listens to
controller_port = 443
# Configures the port the Kubernetes API server listens to
kube_apiserver_port = 6443
# Configures the port the ucp-swarm-manager listens to
swarm_port = 2376
# Configures Swarm scheduler strategy for container scheduling
# This does not affect swarm-mode services
swarm_strategy = "spread"
# Configures DNS settings for the UCP components
dns = []
dns_opt = []
dns_search = []
# Turn on specialized debugging endpoints for profiling UCP performance
profiling_enabled = false
# Turn off Swarm API polling to increase performance of UCP
swarm_polling_disabled = false
# Enable aggregation of DTR image vulnerability scans
image_scan_aggregation_enabled = false
# Tune the KV store timeout and snapshot settings
kv_timeout = 5000
kv_snapshot_count = 20000
# Specify an optional external LB for default links to services with expose ports in the UI
external_service_lb = ""
# Specify the URL of a kubernetes YAML file to be used for installing a CNI
# plugin. Only applies during initial installation. If empty, the default CNI
# plugin will be used
cni_installer_url = ""
# Set to true if ucp should not manage the cni provider lifecycle.
# UCP only manages Calico plugin if the unmanaged-flag is set to false.
unmanaged_cni = false
# Set to true if ucp should deploy secure-overlay which enables IPsec
# Network Encryption in Kubernetes
secure_overlay = false
# Set to true if cluster should use vxlan as dataplane.
calico_vxlan = false
# Set VXLAN VNI id for vxlan data plane.
vxlan_vni = 10000
# Adjust the metrics retention time
metrics_retention_time = ""
# Set the interval for how frequently managers gather metrics from nodes in the cluster
metrics_scrape_interval = "1m"
# Set the size of the cache used by UCP's RethinkDB servers. The installed
# default is 1GB, but leaving this field empty or specifying the special
# value "auto" will instruct RethinkDB to determine a cache size
# automatically (about half of currently available memory) which ensures a
# reliable operation in most scenarios, but the cache size can be tuned
# manually to accommodate increased memory usage by other processes or to
# maximize query performance. A larger cache improves the database’s
# performance, but you must consider other sources of memory consumption to
# avoid out of memory conditions.
# https://www.rethinkdb.com/docs/memory-usage/#cache-size
rethinkdb_cache_size = "16GB"
# If true, disable headers in UCP API responses that indicate which
# UCP controller is returning the response
exclude_server_identity_headers = true
# Set the cloud provider for the kubernetes cluster. Depending on the cloud provider
# different configurations to kubernetes plugin might be need.For eg This could be used
# for cases where we need specific CNI plugin depending on the cloud provider to support
# pod networking
cloud_provider = ""
# Enable the cluster's ISCSI storage backend. Enabling this prepares the
# Kubernetes worker nodes for creation and management of ISCSI based
# persistent volumes.
iscsi_enabled = false
# iscsiadm is the absolute path to the host's iscsiadm binary. This is used when
# iscsi_enabled is set to true. This is necessary for the containerized
# kubelet to act as a iscsi initiator. Defaults to "/usr/sbin/iscsiadm"
# Symlinks not allowed.
iscsiadm_path = ""
# iscsidb is the absolute path to the host's iscsi database. This is used when
# iscsi_enabled is set to true. This is necessary for the containerized
# kubelet to act as a iscsi initiator. Defaults to "/etc/iscsi". Symlinks not allowed.
iscsidb_path = ""
# Set cluster Pod subnet pool. This will be the subnet pool from which the IP for the pod
# would be allocated by the CNI ipam plugin
pod_cidr = "192.168.0.0/16"
# storage_expt_enabled flag is used to enable experimental storage features in the cluster.
storage_expt_enabled = false
# Set the MTU (maximum transmission unit) size for the Calico plugin.
calico_mtu = "1480"
# Set the VXLAN MTU (maximum transmission unit) size for the Calico plugin.
calico_vxlan_mtu = "1450"
# Set the VXLAN Port for the Calico plugin.
calico_vxlan_port = "4789"
# Set the IPIP MTU size for the calico IPIP tunnel interface
ipip_mtu = "1480"
# Set the IP count for azure allocator to allocate IPs per azure VM
azure_ip_count = "64"
# Set the ServiceClusterIPRange used by Kubernates for Services
service_cluster_ip_range = "10.96.0.0/16"
# Set the port range that for Kubernetes services of type NodePort can be
# exposed in.
nodeport_range = "32768-35535"
# Enable the use of a KMS (Key Management Service) plugin. The KMS plugin
# must be deployed on every manager node. UCP will configure the API server
# with the specified endpoint socket.
kms_enabled = false
# Name of the KMS resource
kms_name = ""
# Host path of the KMS endpoint socket
kms_endpoint = ""
# Number of data encryption keys to be cached in the clear by the KMS.
kms_cachesize = 0
# ADVANCED: Set any additional flags on the Kubernetes system components that
# you may require. You must ensure that any custom flags do not conflict with
# configuration options already set by UCP.
custom_kube_api_server_flags = ["--event-ttl=1h0m0s", "--service-node-port-range=30000-32767"]
custom_kube_controller_manager_flags = ["--service-sync-period=5m0s"]
custom_kubelet_flags = ["--http-check-frequency=20s", "--kube-api-qps=200", "--kube-api-burst=200"]
custom_kube_scheduler_flags = ["--algorithm-provider=DefaultProvider", "--kube-api-qps=200", "--kube-api-burst=200"]
custom_kube_proxy_flags = ["--conntrack-min=300000", "--conntrack-max-per-core=200000"]
# Store data about collections for volumes in UCP's local KV store instead of
# on the volume labels. This is used for enforcing access control on volumes.
# Most users will not need to turn this on; this is only necessary if you
# use third-party volume drivers.
local_volume_collection_mapping = false
# Reserve resources for Docker UCP and Kubernetes components which are
# running on manager nodes.
# For more details, see:
# https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#kube-reserved
manager_kube_reserved_resources = "cpu=1,memory=2Gi,ephemeral-storage=4Gi"
# Reserve resources for Docker UCP and Kubernetes components which are
# running on worker nodes.
# For more details, see:
# https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#kube-reserved
worker_kube_reserved_resources = "cpu=100m,memory=300Mi,ephemeral-storage=500Mi"
# Set Number of Pods that can run on this Kubelet.
kubelet_max_pods = 110
# Set pre logon message to alert users before they proceed with login.
pre_logon_message = ""
# Enable Windows GMSA support in Kubernetes. Enabling this deploys the webhook
# admission controller, and creates the Kubernetes resources, required for GMSA
# workloads in Kubernetes.
windows_gmsa = false
[cluster_config.service_mesh]
# Disable Istio Ingress and Service Mesh
enabled = false
# Set the Number of Ingress Replicas
ingress_num_replicas = 2
# Set the List of External IPs for Ingress Gateway Service
ingress_external_ips = []
# Enable external load balancer
ingress_enable_lb = false
# Enable preserving inbound traffic source IP
ingress_preserve_client_ip = false
# Set exposed ports
[[cluster_config.service_mesh.ingress_exposed_ports]]
name = "name"
port = 1
target_port = 2
node_port = 3
# Set node affinity
[cluster_config.service_mesh.ingress_node_affinity]
nodetype = "ingress"
# Set node toleration
[[cluster_config.service_mesh.ingress_node_toleration]]
key = "com.docker.ucp.manager"
value = "dummy"
operator = "Exists"
effect = "NoSchedule"
まとめ
今回はインストール時の設定方法、設定値を確認してみました。
必須項目からコアな設定もあるため、必要なところから設定をしていき適宜設定を適用する形がよいのではと思います。