Edited at

【Docker】etcd+docker-machineでdocker-swarmを使う

More than 1 year has passed since last update.

etcdを管理情報を扱うBackendにして、docker-machineを使ってSwarm Clusterを構築する。


 環境


  • docker(version 1.12.3)

  • docker-machine(version 0.7.0)

  • etcd(version 2.3.7)


Vagrantfile

# -*- mode: ruby -*-

# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# If true, then any SSH connections made will enable agent forwarding.
# Default value: false
config.ssh.forward_agent = true

config.vm.define "manager" do |d|
d.vm.box = "ubuntu/trusty64"
d.vm.network :private_network, ip: "192.168.33.10", virtualbox__intnet: "intnet"
end

config.vm.define "node1" do |d|
d.vm.box = "ubuntu/trusty64"
d.vm.network :private_network, ip: "192.168.33.20", virtualbox__intnet: "intnet"
end

config.vm.define "node2" do |d|
d.vm.box = "ubuntu/trusty64"
d.vm.network :private_network, ip: "192.168.33.30", virtualbox__intnet: "intnet"
end

config.vm.define "kvstore" do |d|
d.vm.box = "ubuntu/trusty64"
d.vm.network :private_network, ip: "192.168.33.40", virtualbox__intnet: "intnet"
end

config.vm.define "host" do |d|
d.vm.box = "ubuntu/trusty64"
d.vm.network :private_network, ip: "192.168.33.50", virtualbox__intnet: "intnet"
end
end

IP
用途

manager
192.168.33.10
Swarmクラスターのmaster

node1
192.168.33.20
Swarmクラスターのnode1

node2
192.168.33.30
Swarmクラスターのmaster

kvstore
192.168.33.40
SwarmクラスターBackendのetcd用

host
192.168.33.50
作業場所

hostからmanager, node1, node2, kvstoreにSSHで入れるようにしておく。

最初にetcd作ってからSwarmを構築していくような形。

作業は全てhostVM上でやる。


etcd

kvstoreにetcdコンテナを立てる。

$ docker-machine create \

--driver generic --generic-ip-address=192.168.33.40 \
--generic-ssh-user vagrant --generic-ssh-key ~/.ssh/id_rsa \
kvstore
$ eval $(docker-machine env kvstore)
$ docker pull quay.io/coreos/etcd:v2.3.7
$ docker run -d --name etcd \
-p 2379:2379 -p 4001:4001 \
quay.io/coreos/etcd:v2.3.7 \
--data-dir=/tmp/default.etcd \
--advertise-client-urls 'http://192.168.33.40:2379,http://192.168.33.40:4001' \
--listen-client-urls 'http://0.0.0.0:2379,http://0.0.0.0:4001'
$ ETCDCTL_ENDPOINT="http://192.168.33.40:2379" ./etcdctl --no-sync get /


はまったポイント

--advertise-client-urls'http://0.0.0.0:2379,http://0.0.0.0:4001'ではなくhttp://192.168.33.40:2379,http://192.168.33.40:4001にする必要がある。


Swarm Clusterの構築


swarm-masterを作る

manager

$ docker-machine create \

--driver generic --generic-ip-address=192.168.33.10 \
--generic-ssh-user vagrant --generic-ssh-key ~/.ssh/id_rsa \
--swarm --swarm-master \
--swarm-discovery="etcd://$(docker-machine ip kvstore):2379/swarm" \
--engine-opt="cluster-store=etcd://$(docker-machine ip kvstore):2379" \
--engine-opt="cluster-advertise=eth1:2376" \
manager


swarm-nodeを作る

node1

$ docker-machine create \

--driver generic --generic-ip-address=192.168.33.20 \
--generic-ssh-user vagrant --generic-ssh-key ~/.ssh/id_rsa \
--swarm \
--swarm-discovery="etcd://$(docker-machine ip kvstore):2379/swarm" \
node1

node2

$ docker-machine create \

--driver generic --generic-ip-address=192.168.33.30 \
--generic-ssh-user vagrant --generic-ssh-key ~/.ssh/id_rsa \
--swarm \
--swarm-discovery="etcd://$(docker-machine ip kvstore):2379/swarm" \
node2


動作確認

etcd

etcdのAPIで状態を確認する。

manager, node1, node2が登録されている。

{

"node": {
"nodes": [
{
"createdIndex": 4,
"modifiedIndex": 4,
"nodes": [
{
"createdIndex": 4,
"modifiedIndex": 4,
"nodes": [
{
"createdIndex": 4,
"modifiedIndex": 4,
"nodes": [
{
"createdIndex": 4,
"modifiedIndex": 4,
"nodes": [
{
"createdIndex": 100,
"modifiedIndex": 100,
"ttl": 134,
"expiration": "2016-11-23T11:45:51.259602538Z",
"value": "192.168.33.10:2376",
"key": "/swarm/docker/swarm/nodes/192.168.33.10:2376"
},
{
"createdIndex": 102,
"modifiedIndex": 102,
"ttl": 168,
"expiration": "2016-11-23T11:46:25.014658247Z",
"value": "192.168.33.20:2376",
"key": "/swarm/docker/swarm/nodes/192.168.33.20:2376"
},
{
"createdIndex": 101,
"modifiedIndex": 101,
"ttl": 137,
"expiration": "2016-11-23T11:45:53.997167245Z",
"value": "192.168.33.30:2376",
"key": "/swarm/docker/swarm/nodes/192.168.33.30:2376"
}
],
"dir": true,
"key": "/swarm/docker/swarm/nodes"
}
],
"dir": true,
"key": "/swarm/docker/swarm"
}
],
"dir": true,
"key": "/swarm/docker"
}
],
"dir": true,
"key": "/swarm"
}
],
"dir": true
},
"action": "get"
}

cluster

docker-machineで状態確認。

192.168.33.10をmasterにしてRunningになっている。

$ docker-machine ls

NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
kvstore - generic Running tcp://192.168.33.40:2376 v1.12.3
manager * (swarm) generic Running tcp://192.168.33.10:2376 manager (master) v1.12.3
node1 - generic Running tcp://192.168.33.20:2376 manager v1.12.3
node2 - generic Running tcp://192.168.33.30:2376 manager v1.12.3

swarm-managerをdockerホストにしてdockerコマンドを起動。

クラスタの情報が確認できる。

$ eval $(docker-machine env --swarm manager)

$ docker info
...
Nodes: 3
manager: 192.168.33.10:2376
└ ID: 7MOY:Q7KV:KRXU:ORDJ:7LNB:LGSM:GT5H:N4J3:BMC7:WC7C:RMEA:UPA6
└ Status: Healthy
└ Containers: 2 (2 Running, 0 Paused, 0 Stopped)
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 514.5 MiB
└ Labels: kernelversion=3.13.0-95-generic, operatingsystem=Ubuntu 14.04.5 LTS, provider=generic, storagedriver=aufs
└ UpdatedAt: 2016-11-23T11:49:04Z
└ ServerVersion: 1.12.3
node1: 192.168.33.20:2376
└ ID: C6GI:OKAT:H2LH:VCBU:OF65:YWST:QVQ7:X6RP:6JY5:2CQK:4PVM:LVCZ
└ Status: Healthy
└ Containers: 1 (1 Running, 0 Paused, 0 Stopped)
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 514.5 MiB
└ Labels: kernelversion=3.13.0-95-generic, operatingsystem=Ubuntu 14.04.5 LTS, provider=generic, storagedriver=aufs
└ UpdatedAt: 2016-11-23T11:49:16Z
└ ServerVersion: 1.12.3
node2: 192.168.33.30:2376
└ ID: IJAI:XSRL:3T5S:5XEF:AUVZ:TLYM:RVAH:MYAW:Q43J:CVCF:J5JH:BGZA
└ Status: Healthy
└ Containers: 1 (1 Running, 0 Paused, 0 Stopped)
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 514.5 MiB
└ Labels: kernelversion=3.13.0-95-generic, operatingsystem=Ubuntu 14.04.5 LTS, provider=generic, storagedriver=aufs
└ UpdatedAt: 2016-11-23T11:49:16Z
└ ServerVersion: 1.12.3
...


適当なコンテナを立ててみる

ubuntuをpullして10個ぐらい立てる。

$ docker pull ubuntu

$ docker run -d ubuntu tail -f /dev/nul
.....
$ docker run -d ubuntu tail -f /dev/nul
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
adb3b7d5e716 ubuntu "tail -f /dev/null" 4 seconds ago Up 3 seconds node1/cocky_stallman
dcd7f4f99864 ubuntu "tail -f /dev/null" 22 seconds ago Up 21 seconds node2/furious_payne
734878f86e59 ubuntu "tail -f /dev/null" 23 seconds ago Up 22 seconds node1/small_ardinghelli
0d6099e7cc05 ubuntu "tail -f /dev/null" 25 seconds ago Up 24 seconds node2/nauseous_hoover
f06a02774421 ubuntu "tail -f /dev/null" 26 seconds ago Up 25 seconds node1/big_kare
b24d42d433f4 ubuntu "tail -f /dev/null" 29 seconds ago Up 28 seconds node2/hungry_engelbart
7daa0be7d630 ubuntu "tail -f /dev/null" 54 seconds ago Up 52 seconds node1/stupefied_joliot
9ddaf2e88b93 ubuntu "tail -f /dev/null" 2 seconds ago Up 1 seconds manager/elated_northcutt
66b4da48e09d ubuntu "tail -f /dev/null" 24 seconds ago Up 23 seconds manager/elated_stonebraker
bb97b9a39644 ubuntu "tail -f /dev/null" 27 seconds ago Up 26 seconds manager/determined_lichterman

デフォルトのstrategyがspreadなのでmanager, node1, node2にそれぞれ均等にコンテナが立っている。


参考

https://www.linux.com/learn/how-use-docker-machine-create-swarm-cluster

https://docs.docker.com/swarm/provision-with-machine/

http://qiita.com/ohhara_shiojiri/items/47e4111a81431c7cb3a7

https://docs.docker.com/swarm/install-w-machine/

http://blog.scottlowe.org/2015/04/19/running-etcd-backed-docker-swarm-cluster/

https://docs.docker.com/machine/reference/create/#/specifying-docker-swarm-options-for-the-created-machine

http://trainingdevops.com/insights-and-tutorials/building-cloud-with-etcd-docker-swarm-and-overlay-network

https://docs.docker.com/engine/userguide/networking/get-started-overlay/#/create-a-swarm-cluster

http://stackoverflow.com/questions/34892377/the-cluster-store-and-cluster-advertise-dont-work

https://github.com/docker/swarm/issues/1710

https://github.com/kubernetes/kubernetes/issues/19235

https://docs.docker.com/swarm/discovery/