MariaDB の列指向データベース、MariaDB ColumnStore は 1.1 から GlusterFS を用いた冗長構成をサポートしているようです。

公式ページから 1.1.3 RPM tarballをダウンロード

RPMがアーカイブされたtar ballをダウンロード、展開します。まだColumnStore用のyumレポジトリはないので、yum install でのインストール，yum updateで更新はできないようです。

wget https://downloads.mariadb.com/ColumnStore/1.1.3/centos/x86_64/7/mariadb-columnstore-1.1.3-1-centos7.x86_64.rpm.tar.gz
tar xvzf mariadb-columnstore-1.1.3-1-centos7.x86_64.rpm.tar.gz

MariaDB, 旧バージョンのColumnStoreをアンインストール

MariaDB Server、古いバージョンのColumnStoreがインストールされている場合はアンインストールします。

yum -y remove "mariadb-*" "MariaDB-*"

yum -y remove "mariadb-columnstore-*"

MariaDB ColumnStore インストール

PM1(Performance Module 1)にのみ MariaDB ColumnStore 1.1.3 を手動インストールします。boostなどの依存RPMもインストールされます。

yum -y localinstall mariadb-columnstore-1.1.*.rpm

GlusterFS のインストール

https://mariadb.com/kb/en/library/preparing-for-columnstore-installation-11x/
を参照、GlusterFSをインストールします。

yum -y install centos-release-gluster
yum -y install glusterfs glusterfs-fuse glusterfs-server
systemctl enable glusterd.service
systemctl start glusterd.service

postConfigure

インストール完了後 PM1 上で /usr/local/mariadb/columnstore/bin/postConfigure 設定ウィザードを実行します。
GlusterFSのvolume作成等も自動的に行われます。
postConfigureのソースによると、gluster volume create によるボリューム作成の際、forceオプションがついているようなので、/パーティション上にdbrootが作成されます。
なお、PM1から各PMにroot SSHログインできるよう、SSH鍵などを設定する必要があります。

# /usr/local/mariadb/columnstore/bin/postConfigure

This is the MariaDB ColumnStore System Configuration and Installation tool.
It will Configure the MariaDB ColumnStore System and will perform a Package
Installation of all of the Servers within the System that is being configured.

IMPORTANT: This tool should only be run on the Parent OAM Module
           which is a Performance Module, preferred Module #1

Prompting instructions:

        Press 'enter' to accept a value in (), if available or
        Enter one of the options within [], if available, or
        Enter a new value


===== Setup System Server Type Configuration =====

There are 2 options when configuring the System Server Type: single and multi

  'single'  - Single-Server install is used when there will only be 1 server configured
              on the system. It can also be used for production systems, if the plan is
              to stay single-server.

  'multi'   - Multi-Server install is used when you want to configure multiple servers now or
              in the future. With Multi-Server install, you can still configure just 1 server
              now and add on addition servers/modules in the future.

Select the type of System Server install [1=single, 2=multi] (2) >

type of System Server install は通常2(multi)を選択します。1 は主にデモ/テスト/トレーニング向けで運用開始後PM(Performance Module)を増やせません。

===== Setup System Module Type Configuration =====

There are 2 options when configuring the System Module Type: separate and combined

  'separate' - User and Performance functionality on separate servers.

  'combined' - User and Performance functionality on the same server

Select the type of System Module Install [1=separate, 2=combined] (2) >

ここではcombindを選択しました。UM と PM が同じノードにインストールされる形になります。

Combined Server Installation will be performed.
The Server will be configured as a Performance Module.
All MariaDB ColumnStore Processes will run on the Performance Modules.

NOTE: The MariaDB ColumnStore Schema Sync feature will replicate all of the
      schemas and InnoDB tables across the User Module nodes. This feature can be enabled
      or disabled, for example, if you wish to configure your own replication post installation.

MariaDB ColumnStore Schema Sync feature is Enabled, do you want to leave enabled? [y,n] (y) >

Schema Sync は通常有効にします。

NOTE: MariaDB ColumnStore Replication Feature is enabled

Enter System Name (columnstore-1) > 


===== Setup Storage Configuration =====


----- Setup Performance Module DBRoot Data Storage Mount Configuration -----

There are 3 options when configuring the storage: internal, external, or DataRedundancy

  'internal' -    This is specified when a local disk is used for the DBRoot storage.
                  High Availability Server Failover is not Supported in this mode

  'external' -    This is specified when the DBRoot directories are mounted.
                  High Availability Server Failover is Supported in this mode.

  'DataRedundancy' - This is specified when gluster is installed and you want
                  the DBRoot directories to be controlled by ColumnStore Data Redundancy.
                  High Availability Server Failover is Supported in this mode.
                  NOTE: glusterd service must be running and enabled on all PMs.

Select the type of Data Storage [1=internal, 2=external, 3=DataRedundancy] (3) >

type of Data Storage の選択で 3=DataRedundancy を選択すると、GlusterFS を利用した冗長構成が組めます。

/usr/local/mariadb/columnstore/bin/columnstoreAlias にあるaliase設定を行うと、ColumnStore用クライアントmcsmysqlと管理コマンドmcsadminがaliasとして設定されます。

===== Setup the Module Configuration =====


----- Performance Module Configuration -----

Enter number of Performance Modules [1,1024] (3) > 3

*** Parent OAM Module Performance Module #1 Configuration ***

Enter Nic Interface #1 Host Name (cs11-pm1) > 
Enter Nic Interface #1 IP Address of cs11-pm1 (10.128.0.2) > 
Enter Nic Interface #2 Host Name (unassigned) > 
Enter the list (Nx,Ny,Nz) or range (Nx-Nz) of DBRoot IDs assigned to module 'pm1' (1) > 

*** Performance Module #2 Configuration ***

Enter Nic Interface #1 Host Name (pm2) > 
Enter Nic Interface #1 IP Address of pm2 (10.128.0.3) > 
Enter Nic Interface #2 Host Name (unassigned) > 
Enter the list (Nx,Ny,Nz) or range (Nx-Nz) of DBRoot IDs assigned to module 'pm2' (2) > 

*** Performance Module #3 Configuration ***

Enter Nic Interface #1 Host Name (pm3) > 
Enter Nic Interface #1 IP Address of pm3 (10.128.0.4) > 
Enter Nic Interface #2 Host Name (unassigned) > 
Enter the list (Nx,Ny,Nz) or range (Nx-Nz) of DBRoot IDs assigned to module 'pm3' (3) >

今回冗長構成にしますので、3PM構成にします。

===== System Installation =====

System Configuration is complete, System Installation is the next step.
Would you like to continue with the System Installation? [y,n] (y) >

System Installation で y(es)を選択すると、PM1から他のPMにColumnStoreの各RPMがリモートインストールされます。

Performing a MariaDB ColumnStore System install using RPM packages
located in the /root directory.


Next step is to enter the password to access the other Servers.
This is either your password or you can default to using a ssh key
If using a password, the password needs to be the same on all Servers.

Enter password, hit 'enter' to default to using a ssh key, or 'exit' > 


===== Running the MariaDB ColumnStore MariaDB ColumnStore setup scripts =====

post-mysqld-install Successfully Completed
post-mysql-install Successfully Completed

----- Performing Install on 'pm2 / pm2' -----

Install log file is located here: /tmp/pm2_rpm_install.log


----- Performing Install on 'pm3 / pm3' -----

Install log file is located here: /tmp/pm3_rpm_install.log


MariaDB ColumnStore Package being installed, please wait ...  DONE

正常にPM2/PM3にインストールが行われた場合は上記のようなメッセージが表示されます。

===== Configuring MariaDB ColumnStore Data Redundancy Functionality =====

----- Setup Data Redundancy Copy Count Configuration -----

Setup the Number of Copies: This is the total number of copies of the data
in the system. At least 2, but not more than the number of PMs(3), are required.

Enter Number of Copies [2-3] (2) >

Data Redundancy Copy Count をここでは 2 を選択します。

----- Setup Data Redundancy Network Configuration -----

  'existing'  -   This is specified when using previously configured network devices. (NIC Interface #1)
                  No additional network configuration is required with this option.

  'dedicated' -   This is specified when it is desired for Data Redundancy traffic to use
                  a separate network than one previously configured for ColumnStore.
                  You will be prompted to provide Hostname and IP information for each PM.


Select the data redundancy network [1=existing, 2=dedicated] (1) >

今回 NIC が一つのサーバでテストしていますので、1 を選択します。

----- Performing Data Redundancy Configuration -----

gluster peer probe 10.128.0.2
gluster peer probe 10.128.0.3
gluster peer probe 10.128.0.4
Gluster create and start volume dbroot1...DONE
Gluster create and start volume dbroot2...DONE
Gluster create and start volume dbroot3...DONE

----- Data Redundancy Configuration Complete -----

上記のように、GlusterFS の peer/volume 等の設定も自動で行われます。

GlusterFS の設定を確認するには以下のコマンドを実行します。

gluster peer status
gluster volume info

===== Checking MariaDB ColumnStore System Logging Functionality =====

The MariaDB ColumnStore system logging is setup and working on local server

===== MariaDB ColumnStore System Startup =====

System Installation is complete. If any part of the install failed,
the problem should be investigated and resolved before continuing.

package installed and the associated service started.

Would you like to startup the MariaDB ColumnStore System? [y,n] (y) >

ここで、y を選択すると、ColumnStoreを起動します。

mcsadmin

以下のコマンドでColumnStore管理ツールmcsadminのヘルプが確認できます。mcsはMariadb ColumnStore の略です。

mcsadmin help

システムステータス取得

mcsadmin> getSystemStatus
getsystemstatus   Sat Sep 23 18:02:17 2017

System columnstore-1

System and Module statuses

Component     Status                       Last Status Change
------------  --------------------------   ------------------------
System        ACTIVE                       Sat Sep 23 18:02:01 2017

Module pm1    ACTIVE                       Sat Sep 23 18:01:29 2017
Module pm2    ACTIVE                       Sat Sep 23 18:01:24 2017
Module pm3    ACTIVE                       Sat Sep 23 18:01:24 2017

Active Parent OAM Performance Module is 'pm1'
Primary Front-End MariaDB ColumnStore Module is 'pm1'
MariaDB ColumnStore Replication Feature is enabled

プロセスステータス取得

mcsadmin> getProcessStatus
getprocessstatus   Sat Sep 23 18:02:59 2017

MariaDB ColumnStore Process statuses

Process             Module    Status            Last Status Change        Process ID
------------------  ------    ---------------   ------------------------  ----------
ProcessMonitor      pm1       ACTIVE            Sat Sep 23 17:59:57 2017       18033
ProcessManager      pm1       ACTIVE            Sat Sep 23 18:00:03 2017       18175
DBRMControllerNode  pm1       ACTIVE            Sat Sep 23 18:00:59 2017       19865
ServerMonitor       pm1       ACTIVE            Sat Sep 23 18:01:00 2017       19898
DBRMWorkerNode      pm1       ACTIVE            Sat Sep 23 18:01:01 2017       19967
DecomSvr            pm1       ACTIVE            Sat Sep 23 18:01:05 2017       20138
PrimProc            pm1       ACTIVE            Sat Sep 23 18:01:07 2017       20235
ExeMgr              pm1       ACTIVE            Sat Sep 23 18:01:19 2017       22003
WriteEngineServer   pm1       ACTIVE            Sat Sep 23 18:01:22 2017       22171
DDLProc             pm1       ACTIVE            Sat Sep 23 18:01:26 2017       22386
DMLProc             pm1       ACTIVE            Sat Sep 23 18:01:30 2017       22550
mysqld              pm1       ACTIVE            Sat Sep 23 18:01:36 2017       23131

ProcessMonitor      pm2       ACTIVE            Sat Sep 23 18:00:41 2017       16683
ProcessManager      pm2       HOT_STANDBY       Sat Sep 23 18:01:32 2017       18230
DBRMControllerNode  pm2       COLD_STANDBY      Sat Sep 23 18:01:15 2017
ServerMonitor       pm2       ACTIVE            Sat Sep 23 18:01:00 2017       17171
DBRMWorkerNode      pm2       ACTIVE            Sat Sep 23 18:01:03 2017       17227
DecomSvr            pm2       ACTIVE            Sat Sep 23 18:01:06 2017       17261
PrimProc            pm2       ACTIVE            Sat Sep 23 18:01:09 2017       17273
ExeMgr              pm2       ACTIVE            Sat Sep 23 18:01:19 2017       18108
WriteEngineServer   pm2       ACTIVE            Sat Sep 23 18:01:23 2017       18159
DDLProc             pm2       COLD_STANDBY      Sat Sep 23 18:01:24 2017
DMLProc             pm2       COLD_STANDBY      Sat Sep 23 18:01:24 2017
mysqld              pm2       ACTIVE            Sat Sep 23 18:01:51 2017       18550

ProcessMonitor      pm3       ACTIVE            Sat Sep 23 18:00:43 2017       14673
ProcessManager      pm3       COLD_STANDBY      Sat Sep 23 18:01:03 2017
DBRMControllerNode  pm3       COLD_STANDBY      Sat Sep 23 18:01:03 2017
ServerMonitor       pm3       ACTIVE            Sat Sep 23 18:01:07 2017       15182
DBRMWorkerNode      pm3       ACTIVE            Sat Sep 23 18:01:09 2017       15207
DecomSvr            pm3       ACTIVE            Sat Sep 23 18:01:12 2017       15220
PrimProc            pm3       ACTIVE            Sat Sep 23 18:01:15 2017       15252
ExeMgr              pm3       ACTIVE            Sat Sep 23 18:01:19 2017       15282
WriteEngineServer   pm3       ACTIVE            Sat Sep 23 18:01:23 2017       15303
DDLProc             pm3       COLD_STANDBY      Sat Sep 23 18:01:24 2017
DMLProc             pm3       COLD_STANDBY      Sat Sep 23 18:01:24 2017
mysqld              pm3       ACTIVE            Sat Sep 23 18:02:03 2017       15700

ColumnStoreクライアント

[root@cs11-pm1 ~]# mcsmysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 26
Server version: 10.2.10-MariaDB-log Columnstore 1.1.3-1

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
+---------------------+
| Database            |
+---------------------+
| calpontsys          |
| columnstore_info    |
| infinidb_querystats |
| infinidb_vtable     |
| information_schema  |
| mysql               |
| performance_schema  |
| test                |
+---------------------+

文字コード設定

設定ファイル(my.cnf)は/usr/local/mariadb/columnstore/mysqlにありますので、例えば文字コードをUTF8に設定する場合は以下のように設定します。

/usr/local/mariadb/columnstore/mysql/my.cnf

[mysqld]
character-set-server=utf8

ColumnStoreを再起動するには、mcsadminでrestartSystemを実行します。

# mcsadmin restartSystem

PM1 をシャットダウンしてみる

ここで、OAM(Operations, Administration and Maintenance) PMであるPM1をシャットダウンします。

# shutdown -h now

PM2で mcsadmin getSystemStatus を実行すると、以下のようなステータスとなり、

getsystemstatus   Sat Sep 23 18:13:48 2017

System columnstore-1

System and Module statuses

Component     Status                       Last Status Change
------------  --------------------------   ------------------------
System        ACTIVE                       Sat Sep 23 18:09:26 2017

Module pm1    AUTO_DISABLED/DEGRADED       Sat Sep 23 18:08:18 2017
Module pm2    ACTIVE                       Sat Sep 23 18:09:02 2017
Module pm3    ACTIVE                       Sat Sep 23 18:01:24 2017

Active Parent OAM Performance Module is 'pm2'
Primary Front-End MariaDB ColumnStore Module is 'pm2'
MariaDB ColumnStore Replication Feature is enabled

OAM PM が pm2 に引き継がれていることが確認できます。

pm2 で df コマンドを実行すると、

[root@cs11-pm2 ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
10.128.0.3:/dbroot2   30G  3.1G   27G  11% /usr/local/mariadb/columnstore/data2
10.128.0.3:/dbroot1   30G  3.1G   27G  11% /usr/local/mariadb/columnstore/data1

pm2にDBROOT1, DBROOT2がマウントされていることが確認できます。

pm3 では以下のようになります。

[root@cs11-pm3 ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
10.128.0.4:/dbroot3   30G  3.0G   27G  10% /usr/local/mariadb/columnstore/data3