docker swarm で compose を使いつつ、api の blue-green deploy を行う

Docker 1.11 の頃の Tips です

環境

IP	OS	ROLE
172.27.100.141	CentOS 7.2	manager + node + consul + registrator
172.27.100.142	CentOS 7.2	node + registrator
172.27.100.143	CentOS 7.2	node + registrator
172.27.100.144	CentOS 7.2	node + registrator

172.27.100.* docker を入れる

yum update
curl -fsSL https://get.docker.com/ | sh

172.27.100.* docker.conf の設定をする

mkdir -p /etc/systemd/system/docker.service.d
cat << EOT > /etc/systemd/system/docker.service.d/docker.conf
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon --insecure-registry=172.27.100.141:5000 --cluster-store=consul://172.27.100.141:8500 --cluster-advertise=${HOST_IP}:2375 -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock
# 172.27.100.141 のみ label を貼る(globalを貼ってるhostが141のため
ExecStart=/usr/bin/docker daemon --insecure-registry=172.27.100.141:5000 --label global=enable --cluster-store=consul://172.27.100.141:8500 --cluster-advertise=${HOST_IP}:2375 -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock
EOT
systemctl daemon-reload
systemctl restart docker
docker run hello-world
ps aux | grep docker | grep -v grep

172.27.100.141 manager の設定をする

docker run -d -p 8500:8500 --name=consul progrium/consul -server -bootstrap
docker run -d -p 4000:4000 swarm manage -H :4000 --replication --advertise 172.27.100.141:4000  consul://172.27.100.141:8500

172.27.100.* node の設定をする

docker run -d swarm join --advertise=${HOST_IP}:2375 consul://172.27.100.141:8500

172.27.100.141 docker info で swarm できているか確認する

export DOCKER_HOST="tcp://localhost:4000"
docker info

こんな出力になったら成功

Containers: 51
 Running: 19
 Paused: 0
 Stopped: 32
Images: 80
Server Version: swarm/1.1.3
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 4
 vm-st-050: 172.27.100.141:2375
  └ Status: Healthy
  └ Containers: 19
  └ Reserved CPUs: 0 / 8
  └ Reserved Memory: 0 B / 14.22 GiB
  └ Labels: executiondriver=native-0.2, global=enable, kernelversion=3.10.0-327.10.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
  └ Error: (none)
  └ UpdatedAt: 2016-04-19T02:37:20Z
 vm-st-051: 172.27.100.142:2375
  └ Status: Healthy
  └ Containers: 11
  └ Reserved CPUs: 0 / 8
  └ Reserved Memory: 0 B / 16.29 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.13.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
  └ Error: (none)
  └ UpdatedAt: 2016-04-19T02:37:11Z
 vm-st-052: 172.27.100.143:2375
  └ Status: Healthy
  └ Containers: 11
  └ Reserved CPUs: 0 / 8
  └ Reserved Memory: 0 B / 16.29 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.13.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
  └ Error: (none)
  └ UpdatedAt: 2016-04-19T02:37:40Z
 vm-st-053: 172.27.100.144:2375
  └ Status: Healthy
  └ Containers: 10
  └ Reserved CPUs: 0 / 8
  └ Reserved Memory: 0 B / 16.29 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.13.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
  └ Error: (none)
  └ UpdatedAt: 2016-04-19T02:37:47Z
Plugins: 
 Volume: 
 Network: 
Kernel Version: 3.10.0-327.10.1.el7.x86_64
Operating System: linux
Architecture: amd64
CPUs: 32
Total Memory: 63.09 GiB
Name: f52c9ae71a74

registory を立てておく

cd ${HOME}
docker run -d -p 5000:5000 --name registry -e constraint:global==enable \
      -v `pwd`/docker-registry:/var/lib/registry \
      registry:2

172.27.100.* registrator を立てる

docker run -d \
  --name=registrator \
  -h ${HOST_IP} \
  --volume=/var/run/docker.sock:/tmp/docker.sock \
  gliderlabs/registrator:latest \
  consul://172.27.100.141:8500

image を push するところは割愛

TODO consul を冗長化する

TODO manager を冗長化する

docker-compose する

docker-compose.yml

version: '2'
services:
  bg:
    image: hanzel/blue-green
    container_name: bg
    ports:
      - "80:80"
      - "8080:8080"
    environment:
      - CONSUL_URL=172.27.100.141:8500
      - BLUE_NAME=blue
      - GREEN_NAME=green
      - LIVE=blue
      - "constraint:global==enable"
    depends_on:
      - green
      - blue
    networks:
      - management

  blue:
    image: 172.27.100.141:5000/api-develop:latest
    ports:
      - '9000'
    environment:
      - SERVICE_9000_NAME=blue
    networks:
      - management

  green:
    image: 172.27.100.141:5000/api-develop:latest
    ports:
      - '9000'
    depends_on:
      - postgres
    environment:
    - SERVICE_9000_NAME=green
    networks:
      - management

  postgres:
    container_name: 'postgres'
    environment:
      - POSTGRES_PASSWORD=mysecretpassword
      - POSTGRES_DB=default
    image: postgres
    ports:
      - 5432:5432
    networks:
      - management

networks:
  management:
    driver: overlay

docker-compose up -d

nginx の proxy pass 設定を確認する

docker -H 172.27.100.141:2375 exec bg cat /etc/nginx/nginx.conf

nginx.conf

---- 8< ----
http {
        upstream blue {
            least_conn;

            server 172.27.100.144:32812 max_fails=3 fail_timeout=60 weight=1;
        }

        upstream green {
            least_conn;

            server 172.27.100.142:32778 max_fails=3 fail_timeout=60 weight=1;
        }
---- 8< ----

scale して nginx.conf を確認する

docker-compose scale green=3 blue=3
docker -H 172.27.100.141:2375 exec bg cat /etc/nginx/nginx.conf

nginx.conf

---- 8< ----
http {
        upstream blue {
            least_conn;

            server 172.27.100.142:32779 max_fails=3 fail_timeout=60 weight=1;
            server 172.27.100.143:32773 max_fails=3 fail_timeout=60 weight=1;
            server 172.27.100.144:32812 max_fails=3 fail_timeout=60 weight=1;
        }

        upstream green {
            least_conn;

            server 172.27.100.142:32778 max_fails=3 fail_timeout=60 weight=1;
            server 172.27.100.143:32772 max_fails=3 fail_timeout=60 weight=1;
            server 172.27.100.144:32813 max_fails=3 fail_timeout=60 weight=1;
        }
---- 8< ----

blue で動いていることを確認する

docker -H 172.27.100.141:2375 exec bg cat /var/live
> blue

green に切り替える

docker -H 172.27.100.141:2375 exec bg switch green
docker -H 172.27.100.141:2375 exec bg cat /var/live
> green