docker
kubernetes

Docker Enterprise Edition 確認

Docker Enterprise Edition セットアップ

https://docs.docker.com/install/linux/docker-ee/centos/

準備

DockerEE Install

# export DOCKERURL="https://storebits.docker.com/ee/centos/sub-XXXX"
# echo $DOCKERURL/centos > /etc/yum/vars/dockerurl
# yum install -y yum-utils device-mapper-persistent-data lvm2
# yum-config-manager --add-repo "$DOCKERURL/centos/docker-ee.repo"
# yum install docker-ee
# yum list docker-ee --showduplicates | sort -r
# systemctl start docker
# docker run hello-world
# yum install bash-completion

# groupadd docker
# useradd -m hoge
# usermod -aG docker hoge
# su - hoge
$

UCP(Universal Control Plane)Install

$ docker image pull docker/ucp:2.2.10

$ docker container run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:2.2.10 install --host-address AAA.AAA.AAA.AAA --interactive

DTR(Docker Trusted Registry) Install

  • DTR は UCPにより管理されたSwarm上で動くコンテナApp
  • なのでSwarmが必要・・・!?
  • Workerノード上であること
  • 固定のホスト名をもつ
  • 80/443 used

https://docs.docker.com/datacenter/dtr/2.4/guides/architecture/#under-the-hood
https://docs.docker.com/datacenter/dtr/2.4/guides/admin/install/system-requirements/
https://docs.docker.com/datacenter/dtr/2.4/guides/admin/install/

  • Admin Settings -> Docker Trusted Registry
$ docker pull docker/dtr:2.4.4
2.4.4: Pulling from docker/dtr
605ce1bd3f31: Already exists
d91aa78802c7: Pull complete
2309fc511a17: Pull complete
d25a2b267e92: Pull complete
Digest: sha256:4c6a428021bd5a1f69969502c5579a517b9639b97cca2c806178b17ad4389394
Status: Downloaded newer image for docker/dtr:2.4.4

$ docker run -it --rm docker/dtr install  --ucp-node worker02  --ucp-username admin  --ucp-url https://XXX.XXX.XXX.XXX  --ucp-ca "-----BEGIN CERTIFICATE-----
> MIIBgTCCASegAwIBAgIUTGt5qZPo9lxIQHULKQ0l9KAQbCEwCgYIKoZIzj0EAwIw
> HTEbMBkGA1UEAxMSVUNQIENsaWVudCBSb290IENBMB4XDTE4MDYxMjIzNTMwMFoX
> DTIzMDYxMTIzNTMwMFowHTEbMBkGA1UEAxMSVUNQIENsaWVudCBSb290IENBMFkw
> EwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEb5iHVmmZ8h4xmYBPp1KmjkRiZXFIFJK4
> Q8Los7BjhyEpSN3zm+RrvRf/fcaLrOTrY96/RrnMbO2Uuiy9fgElz6NFMEMwDgYD
> VR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQIwHQYDVR0OBBYEFJRoHop/
> C+thDl9igNC1Xu/AZHXaMAoGCCqGSM49BAMCA0gAMEUCIGUuXC0hM9iFHob4dULC
> N5HYClbFNag4ZyKk74Zsy1H0AiEA4KfQseEHlzryUW6zOteWjbk5BWtKPZscw5gU
> fVo6/kE=
> -----END CERTIFICATE-----
> "
INFO[0000] Beginning Docker Trusted Registry installation
ucp-password:
INFO[0002] Validating UCP cert
INFO[0002] Connecting to UCP
INFO[0002] health checking ucp
INFO[0002] The UCP cluster contains the following nodes without port conflicts: worker01, worker02
INFO[0002] Searching containers in UCP for DTR replicas
INFO[0002] Searching containers in UCP for DTR replicas
INFO[0003] verifying [80 443] ports on worker02
INFO[0019] Waiting for running dtr-phase2 container to finish
INFO[0019] starting phase 2
INFO[0000] Validating UCP cert
INFO[0000] Connecting to UCP
INFO[0000] health checking ucp
INFO[0000] Verifying your system is compatible with DTR
INFO[0000] Checking if the node is okay to install on
INFO[0000] Creating network: dtr-ol
INFO[0000] Connecting to network: dtr-ol
INFO[0000] Waiting for phase2 container to be known to the Docker daemon
INFO[0001] Setting up replica volumes...
INFO[0002] Creating initial CA certificates
INFO[0002] Bootstrapping rethink...
INFO[0002] Creating dtr-rethinkdb-0893da3497a7...
INFO[0019] Establishing connection with Rethinkdb
INFO[0020] Waiting for database dtr2 to exist
INFO[0021] Establishing connection with Rethinkdb
INFO[0021] Generated TLS certificate.                    dnsNames=[*.com *.*.com example.com *.dtr *.*.dtr] domains=[*.com *.*.com 172.17.0.1 example.com *.dtr *.*.dtr] ipAddresses=[172.17.0.1]
INFO[0021] License config copied from UCP.
INFO[0021] Migrating db...
INFO[0000] Establishing connection with Rethinkdb
INFO[0000] Migrating database schema                     fromVersion=0 toVersion=9
INFO[0007] Waiting for database notaryserver to exist
INFO[0008] Waiting for database notarysigner to exist
INFO[0008] Waiting for database jobrunner to exist
INFO[0010] Migrated database from version 0 to 9
INFO[0032] Starting all containers...
INFO[0032] Getting container configuration and starting containers...
INFO[0033] Recreating dtr-rethinkdb-0893da3497a7...
INFO[0038] Creating dtr-registry-0893da3497a7...
INFO[0047] Creating dtr-garant-0893da3497a7...
INFO[0056] Creating dtr-api-0893da3497a7...
INFO[0090] Creating dtr-notary-server-0893da3497a7...
INFO[0100] Recreating dtr-nginx-0893da3497a7...
INFO[0109] Creating dtr-jobrunner-0893da3497a7...
INFO[0149] Creating dtr-notary-signer-0893da3497a7...
INFO[0158] Creating dtr-scanningstore-0893da3497a7...
INFO[0170] Trying to get the kv store connection back after reconfigure
INFO[0170] Establishing connection with Rethinkdb
INFO[0171] Verifying auth settings...
INFO[0175] Successfully registered dtr with UCP
INFO[0175] Establishing connection with Rethinkdb
INFO[0176] Background tag migration started
INFO[0176] Installation is complete
INFO[0176] Replica ID is set to: 0893da3497a7

INFO[0176] You can use flag '--existing-replica-id 0893da3497a7' when joining other replicas to your Docker Trusted Registry Cluster
$
  • フラグを付けて、ほかのDTRクラスタにレプリカをジョインできるらしい
  • UCP -> Shared Resources -> Stacks でBasic Containersとしてリストされているのを確認
  • DTR Web UI -> https://<host addr where DTR is working>/

  • ログイン後、設定(System -> )

    • CA for TLS Coommunication(GENERAL -> Domain & proxies -> LOAD BALANCER / PUBLIC ADDRESSにDTRが動いているホストのアドレスを入れてSave、これをしないと login できなかった)
    • Backend Storage
    • Security
    • Garbage collection(Remove Untagged Images , upgradeは試験的)
  • Users ->

    • ユーザ(hoge)を作ってみる
  • System -> Repositories -> "CREATE REPOSITORY ON PUSH" をオンにしないと「denied: requested access to the resource is denied」でおこられる

DTRにPushしてみる

  • Docker Hubから落としてタグ付け(ex: ZZZ.ZZZ.ZZZ.ZZZ/hoge/irohaboard
  • docker login ZZZ.ZZZ.ZZZ.ZZZ(作ったユーザにて)
  • docker push ZZZ.ZZZ.ZZZ.ZZZ/hoge/irohaboard
$ docker login ZZZ.ZZZ.ZZZ.ZZZ
$ docker push ZZZ.ZZZ.ZZZ.ZZZ/hoge/hello-world
The push refers to a repository [ZZZ.ZZZ.ZZZ.ZZZ/hoge/hello-world]
2b8cbd0846c5: Pushed
latest: digest: sha256:d5c74e6f8efc7bdf42a5e22bd764400692cf82360d86b8c587a7584b03f51520 size: 524

DTR Web UIからも確認

DTRからPullしてみる

$ docker login ZZZ.ZZZ.ZZZ.ZZZ
Username: hoge
Password:
Login Succeeded

$ docker pull ZZZ.ZZZ.ZZZ.ZZZ/hoge/irohaboard
Using default tag: latest
latest: Pulling from hoge/irohaboard
d9aaf4d82f24: Pull complete
4ae61ec9aa1f: Pull complete
Digest: sha256:d206780920d470eafa0d192a04588a52f0a48429b00b4f269bfea126efa558dc
Status: Downloaded newer image for ZZZ.ZZZ.ZZZ.ZZZ/hoge/irohaboard:latest
$
  • 比較的直観的にリポジトリを管理できそう

デフォルトオーケストレーションタイプの変更

  • デフォルト mixed ?
  • Settings -> Scheduler page にて変更可能

Workerノード追加(Swarm)

  • Worker対象ノードにもDockerEEインストール、デーモン起動
  • Shared Resources -> Nodes -> Add Node でコマンドコピペ
  • 対象ノード上で実行
# docker swarm join --token SWMTKN-1-270gpjyei5f77czsszmzsgaf07lm6h0jea6tqa7uj5wwkh9794-f5itn0ncolqfjehv0p4xoiupx XXX.XXX.XXX.XXX:2377
This node joined a swarm as a worker.

Workerノード追加(k8s)

kubernetes CLI インストール

https://docs.docker.com/ee/ucp/user-access/kubectl/

# curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
# chmod + ./kubectl
# mv ./kubectl /usr/local/bin/kubectl

UCP Client Bundleを入れないとエラーになる

https://docs.docker.com/ee/ucp/user-access/cli/#use-client-certificates

  • My Profile -> Client Bundles -> New Client Bundle -> Generate Client Bundle
    • Certificate Bundleのダウンロードが始まる
  • unzip して、以下実行

ダウンロードしたファイルは、マネージャが名前解決したものになって、このはIPアドレスによりクラスタができてるので、IPアドレスに関係するファイルを直した。

$ cd client-bundle/
$ eval "$(<env.sh)"
Cluster "XXX.XXX.XXX.XXX:6443_admin" set.
User "ucp_XXX.XXX.XXX.XXX:6443_admin" set.
Context "ucp_XXX.XXX.XXX.XXX:6443_admin" created.

$ kubectl config current-context
ucp_sun-t-XXX.XXX.XXX.XXX_admin

$ kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   1h

$ kubectl get ns
NAME          STATUS    AGE
default       Active    1h
kube-public   Active    1h
kube-system   Active    1h

試し、コマンドからNamespace作って、UIから確認

$ cat namespace.yml
apiVersion: v1
kind: Namespace
metadata:
  name: devel

$ kubectl apply -f namespace.yml
namespace "devel" created

$ kubectl get ns devel
NAME      STATUS    AGE
devel     Active    1m

UCPでも確認

UCP Uninstall

$ docker container run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:2.2.10 uninstall-ucp --interactive
INFO[0000] Your engine version 17.06.2-ee-13, build ac44d73 (3.10.0-862.2.3.el7.x86_64) is compatible
INFO[0000] We're about to uninstall from this swarm cluster. UCP ID: x6zb1r849byqgcme942trrnqw
Do you want to proceed with the uninstall? (y/n): y
INFO[0000] Uninstalling UCP on each node...
INFO[0010] UCP has been removed from this cluster successfully.
INFO[0012] Removing UCP Services

マネージャノード、IPアドレス変えたら(テスト用)

  • Uninstall UCP(docker container run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:2.2.10 uninstall-ucp)
  • docker swarm leave --force
  • docker swarm init --advertise-addr XXX.XXX.XXX.XXX
  • Reinstall UCP