LoginSignup
7
8

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

@quickguard

kubernetes と Vitess を使って MySQL クラスタを構築

Last updated at Posted at 2015-12-25

概要

Vitess を使って MySQL クラスタを構築する

vitess
https://github.com/youtube/vitess/

事前準備

  • プロジェクトの作成
  • サービスアカウントキーの取得（JSON形式）
  • gcloud 認証済み

コンテナクラスタ作成

以前作成した terraform のコードをベースに構築

■tf ファイル

provider.tf
provider "google" {
  credentials = "${file("account.json")}"
  project     = "${var.project}"
  region      = "${var.region}"
}
container_cluster.tf
resource "google_container_cluster" "default" {
  name = "${var.cluster_name}"
  zone = "${var.zone}"
  network = "${var.network}"
  initial_node_count = "${var.initial_node_count}"

  node_config {
    machine_type = "${var.machine_type}"
    disk_size_gb = "${var.disk_size}"
    oauth_scopes = [
      "https://www.googleapis.com/auth/devstorage.read_write"
    ]
  }

  master_auth {
    username = "${var.master_auth_username}"
    password = "${var.master_auth_password}"
  }
}
storage_bucket.tf
resource "google_storage_bucket" "image-store" {
  name = "${var.project}-backup-bucket"
  location = "${var.location}"
}
variables.tf
variable "project" {}
variable "location" {
  default = "ASIA"
}
variable "region" {
  default = "asia-east1"
}
variable "cluster_name" {}
variable "zone" {}
variable "network" {}
variable "initial_node_count" {}
variable "machine_type" {}
variable "disk_size" {}
variable "master_auth_username" {}
variable "master_auth_password" {}

■terraform.tfvars

terraform.tfvars
## project
project = "**********"

## cluster config
cluster_name = "vitess-cluster"
zone = "asia-east1-b"
network = "default"
initial_node_count = "4"

## node config
machine_type = "n1-standard-2"
disk_size = "10"

## master auth
master_auth_username = "**********"
master_auth_password = "**********"
  • 構築内容
    • cluster名 ： vitess-cluster (asia-east1-b)
    • node数 ： 4台 (n1-standard-2) ※これ以下のスペックだと無理
    • node に対して、ストレージへの read/write 権限を付与
    • Bucket名 ： [プロジェクト名]-backup-bucket で作成(ASIA)

クラスタ作成

$ terraform apply

その他初期設定

$ gcloud config set project [プロジェクト名]
$ gcloud config set compute/zone asia-east1-b
$ gcloud config set container/cluster vitess-cluster
$ gcloud container clusters get-credentials vitess-cluster

Kubernetes 上で Vitess 環境を構築

vtctlclient のインストール

$ export GOPATH=`pwd`
$ go get github.com/youtube/vitess/go/cmd/vtctlclient
$ cp $GOPATH/bin/vtctlclient ~/bin/

Vitess 初期設定

$ cd $GOPATH/src/github.com/youtube/vitess/examples/kubernetes
$ ./condigure.sh
Backup Storage (file, gcs) [gcs]: 
Google Developers Console Project [プロジェクト名]:
Google Cloud Storage bucket for Vitess backups: [プロジェクト名]-backup-bucket
Saving config.sh...

etcd クラスタ構築

etcd 起動前に外部からアクセスできるようにサービス用テンプレートを編集
(これしないと外部から WEB UI(vtctld) にアクセスした際に Topology Browser が参照できない )

$ vi ./etcd-service-template.yaml
 spec:
+  type: LoadBalancer
   ports:
     - port: 4001

etcd 起動

$ ./etcd-up.sh
Creating etcd service for global cell...
service "etcd-global" created
service "etcd-global-srv" created
Creating etcd replicationcontroller for global cell...
replicationcontroller "etcd-global" created
Creating etcd service for test cell...
service "etcd-test" created
service "etcd-test-srv" created
Creating etcd replicationcontroller for test cell...
replicationcontroller "etcd-test" created

サービスの状態確認

$ kubectl get svc
NAME              CLUSTER_IP      EXTERNAL_IP       PORT(S)               SELECTOR                                AGE
etcd-global       10.31.247.111   ***.***.***.***   4001/TCP              app=vitess,cell=global,component=etcd   2m
etcd-global-srv   None            <none>            7001/TCP              app=vitess,cell=global,component=etcd   2m
etcd-test         10.31.244.184   ***.***.***.***   4001/TCP              app=vitess,cell=test,component=etcd     2m
etcd-test-srv     None            <none>            7001/TCP              app=vitess,cell=test,component=etcd     2m

vtctld の起動
外部からアクセスできるようにサービス用テンプレートを編集

$ vi ./vtctld-service.yaml
 spec:
+  type: LoadBalancer
   ports:
     - name: web

vtctld 起動

$ ./vtctld-up.sh
Creating vtctld service...
service "vtctld" created
Creating vtctld replicationcontroller...
replicationcontroller "vtctld" created

To access vtctld web UI, start kubectl proxy in another terminal:
  kubectl proxy --port=8001
Then visit http://localhost:8001/api/v1/proxy/namespaces/default/services/vtctld:web/

サービスの状態確認

$ kubectl get svc
NAME              CLUSTER_IP      EXTERNAL_IP      PORT(S)               SELECTOR                                AGE
vtctld            10.55.250.136   ***.***.***.***  15000/TCP,15999/TCP   app=vitess,component=vtctld             1m

以下のアドレスにて WEB UI にアクセス可能
http://[ EXTERNAL_IP ]:15000/api/v1/proxy/namespaces/default/services/vtctld:web/

01.png

vttablet, MySQL Pod の作成

デフォルトのパラメータではリソース不足でエラーになるため修正

$ ./vttablet-pod-template.yaml
@@ -30,8 +30,8 @@
       resources:
         limits:
-          memory: "1Gi"
-          cpu: "500m"
+          memory: "512Mi"
+          cpu: "100m"

@@ -88,8 +88,8 @@
       resources:
         limits:
-          memory: "1Gi"
-          cpu: "500m"
+          memory: "512Mi"
+          cpu: "100m"

pod, keyspace 作成

$ ./vttablet-up.sh
Creating test_keyspace.shard-0 pods in cell ...
Creating pod for tablet test-0000000100...
pod "vttablet-100" created
Creating pod for tablet test-0000000101...
pod "vttablet-101" created
Creating pod for tablet test-0000000102...
pod "vttablet-102" created
Creating pod for tablet test-0000000103...
pod "vttablet-103" created
Creating pod for tablet test-0000000104...
pod "vttablet-104" created

$ kubectl get pod
NAME                READY     STATUS    RESTARTS   AGE
vttablet-100        2/2       Running   1          1m
vttablet-101        2/2       Running   1          1m
vttablet-102        2/2       Running   0          1m
vttablet-103        2/2       Running   0          1m
vttablet-104        2/2       Running   0          1m

MySQL 初期化

$ ./kvtctl.sh RebuildKeyspaceGraph test_keyspace
Starting port forwarding to vtctld...

$ ./kvtctl.sh InitShardMaster -force test_keyspace/0 test-0000000100
Starting port forwarding to vtctld...
W1224 04:59:53.996276   20753 main.go:43] W1224 04:59:53.974546 logger.go:256] master-elect tablet est-0000000100 is not the shard master, proceeding anyway as -force was used
W1224 04:59:53.997008   20753 main.go:43] W1224 04:59:53.975036 logger.go:256] master-elect tablet est-0000000100 is not a master in the shard, proceeding anyway as -force was used

$ ./kvtctl.sh ListAllTablets test
Starting port forwarding to vtctld...
test-0000000100 test_keyspace 0 master 10.28.2.10:15002 10.28.2.10:3306 []
test-0000000101 test_keyspace 0 replica 10.28.1.9:15002 10.28.1.9:3306 []
test-0000000102 test_keyspace 0 replica 10.28.0.8:15002 10.28.0.8:3306 []
test-0000000103 test_keyspace 0 rdonly 10.28.3.6:15002 10.28.3.6:3306 []
test-0000000104 test_keyspace 0 rdonly 10.28.3.7:15002 10.28.3.7:3306 []

WEB UI から Shard Status 確認
02.png

テーブル作成

./kvtctl.sh ApplySchema -sql "$(cat create_test_table.sql)" test_keyspace

作成したスキーマの確認

$ ./kvtctl.sh GetSchema test-0000000100
Starting port forwarding to vtctld...
{
  "database_schema": "CREATE DATABASE /*!32312 IF NOT EXISTS*/ `{{.DatabaseName}}` /*!40100 DEFAULT CHARACTER SET utf8 */",
  "table_definitions": [
    {
      "name": "messages",
      "schema": "CREATE TABLE `messages` (\n  `page` bigint(20) unsigned NOT NULL DEFAULT '0',\n  `time_created_ns` bigint(20) unsigned NOT NULL DEFAULT '0',\n  `keyspace_id` bigint(20) unsigned DEFAULT NULL,\n  `message` varchar(10000) DEFAULT NULL,\n  PRIMARY KEY (`page`,`time_created_ns`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8",
      "columns": [
        "page",
        "time_created_ns",
        "keyspace_id",
        "message"
      ],
      "primary_key_columns": [
        "page",
        "time_created_ns"
      ],
      "type": "BASE TABLE",
      "data_length": 16384,
      "row_count": 3
    }
  ],
  "version": "6428d6f8bc7250889e9b57c878333211"
}

vtctl で取得する場合は以下

$ vtctlclient -server [vtctld の EXTERNAL_IP]:15999 GetSchema test-0000000100

vtgate 起動

./vtgate-up.sh
Creating vtgate service...
service "vtgate" created
Creating vtgate replicationcontroller...
replicationcontroller "vtgate" created

$ kubectl get svc
NAME              CLUSTER_IP      EXTERNAL_IP       PORT(S)               SELECTOR                                AGE
vtgate            10.31.252.42    ***.***.***.***   15001/TCP             app=vitess,component=vtgate             2m

サンプルアプリ実行
guestbook 起動

$ ./guestbook-up.sh
Creating guestbook service...
service "guestbook" created
Creating guestbook replicationcontroller...
replicationcontroller "guestbook" created

$ kubectl get svc
NAME              CLUSTER_IP      EXTERNAL_IP       PORT(S)               SELECTOR                                AGE
guestbook         10.31.250.204   ***.***.***.***   80/TCP                app=guestbook,component=frontend        2m

http://[ EXTERNAL_IP ]/ にアクセス
top.png

random page. のリンクより適当に入力
03.png

vtctl より確認

$ vtctlclient -server [vtctld EXTERNAL_IP]:15999 VtGateExecuteShards -server [vtgate EXTERNAL_IP]:15001 -keyspace test_keyspace -shards 0 -tablet_type rdonly "show tables"
{
  "Fields": [
    {
      "name": "Tables_in_vt_test_keyspace",
      "type": 6165
    }
  ],
  "RowsAffected": 1,
  "InsertID": 0,
  "Rows": [
    [
      "messages"
    ]
  ]
}


$ vtctlclient -server [vtctld EXTERNAL_IP]:15999 VtGateExecuteShards -server [vtgate EXTERNAL_IP]:15001 -keyspace test_keyspace -shards 0 -tablet_type rdonly "SELECT message FROM messages WHERE page=10 ORDER BY time_created_ns"
{
  "Fields": [
    {
      "name": "message",
      "type": 6165
    }
  ],
  "RowsAffected": 5,
  "InsertID": 0,
  "Rows": [
    [
      "1111111111"
    ],
    [
      "2222222222"
    ],
    [
      "3333333333"
    ],
    [
      "4444444444"
    ],
    [
      "5555555555"
    ]
  ]
}

master 切り替え
切り替え前

$ ./kvtctl.sh ListAllTablets test
test-0000000100 test_keyspace 0 master 10.28.3.6:15002 10.28.3.6:3306 []
test-0000000101 test_keyspace 0 spare 10.28.1.6:15002 10.28.1.6:3306 []
test-0000000102 test_keyspace 0 replica 10.28.2.5:15002 10.28.2.5:3306 []
test-0000000103 test_keyspace 0 rdonly 10.28.0.6:15002 10.28.0.6:3306 []
test-0000000104 test_keyspace 0 rdonly 10.28.3.7:15002 10.28.3.7:3306 []

切り替え

$ vtctlclient -server 130.211.252.96:15999 TabletExternallyReparented test-101

test-101 が master になる

$ ./kvtctl.sh ListAllTablets test
test-0000000100 test_keyspace 0 replica 10.28.3.6:15002 10.28.3.6:3306 []
test-0000000101 test_keyspace 0 master 10.28.1.6:15002 10.28.1.6:3306 []
test-0000000102 test_keyspace 0 replica 10.28.2.5:15002 10.28.2.5:3306 []
test-0000000103 test_keyspace 0 rdonly 10.28.0.6:15002 10.28.0.6:3306 []
test-0000000104 test_keyspace 0 rdonly 10.28.3.7:15002 10.28.3.7:3306 []

バックアップ
rdonly Pod に対して以下コマンド実行

$ ./kvtctl.sh Backup test-0000000104

terraform で作成した Bucket にアップロードされる

04png.png

コマンドで確認

$ ./kvtctl.sh ListBackups test_keyspace/0
2015-12-24.051117.test-0000000104

スケールアウト
試せていないので別途・・・

7
8
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
7
8

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?