REX-Ray Multiple Storage Configuration: ScaleIO + Isilon

Confirm REX-Ray Support

At the last of this previous post, I mentioned about REX-Ray multiple storage configuration. Reading through this document, it didn't seem to me that REX-Ray supports Multiple Storage configuration.

However, in REX-Ray FAQ page, they clearly state that REX-Ray supports multiple storage configuration like below.

So there should be a proper way to configure REX-Ray with multiple storages ScaleIO and Isilon.

Set Configuration File

At the section "Defining Service Endpoints" of this libstorage document,
the following example is explained.

rexray:
  modules:
    default-docker:
      host:     unix:///run/docker/plugins/virtualbox.sock
      spec:     /etc/docker/plugins/virtualbox.spec
      libstorage:
        service: virtualbox
    scaleio-docker:
      type:     docker
      host:     unix:///run/docker/plugins/scaleio.sock
      spec:     /etc/docker/plugins/scaleio.spec
      libstorage:
        service: scaleio
libstorage:
  server:
    services:
      scaleio:
        driver: scaleio
      virtualbox:
        driver: virtualbox
scaleio:
  endpoint:             https://SCALEIO_GATEWAY/api
  insecure:             true
  userName:             SCALEIO_USER
  password:             SCALEIO_PASS
  systemName:           SCALEIO_SYSTEM_NAME
  protectionDomainName: SCALEIO_DOMAIN_NAME
  storagePoolName:      SCALEIO_STORAG_NAME
virtualbox:
  volumePath: $HOME/VirtualBox/Volumes

According to the document, you need to set modules for advanced configuration.

Modules enable a single REX-Ray instance to present multiple personalities or volume endpoints, serving hosts that require access to multiple storage platforms.

I've never set modules parameter in /etc/rexray/config.yml so far. I am not sure if it is really necessary to set for multiple storage configuration.
So I set parameters based on the example like below.

[root@sunny ~]# cat /etc/rexray/config.yml
libstorage:
  server:
    services:
      scaleio:
        driver: scaleio
      isilon:
        driver: isilon

isilon:
  endpoint: https://192.168.20.40:8080
  group: wheel
  username: root
  password: ****
  volumePath: /ifs/volumes
  nfsHost: 192.168.20.40
  dataSubnet: 192.168.20.0/24
  insecure: true

scaleio:
  insecure: true
  thinOrThick: ThinProvisioned
  password: ****
  endpoint: https://172.16.5.1/api
  systemID: 7489376b176d3545
  protectionDomainID: 1947f06b00000000
  storagePoolID: 1f63dc4100000000
  userName: admin

Then, REX-Ray started its service normally with the configuration parameter.

[root@sunny ~]# rexray start
Starting REX-Ray...SUCCESS!

  The REX-Ray daemon is now running at PID -1. To
  shutdown the daemon execute the following command:

    sudo /usr/bin/rexray stop

Docker Operation Varification

First, I tried to check docker volume list then got this error. So you need to set explicitly "which service to use" when running REX-Ray commands.

[root@sunny ~]# rexray volume ls
Oops, an error occured!

  client must specify service

Setting service flag properly, then the results below returned.

[root@sunny ~]# rexray volume ls -s isilon
ID                          Name                        Status     Size
rexray_isilon_1             rexray_isilon_1             available  0
rexray_isilon_2             rexray_isilon_2             available  0
rexray_takeum_1             rexray_takeum_1             available  0
test_at_isilon_ifs_volumes  test_at_isilon_ifs_volumes  available  0

[root@sunny ~]# rexray volume ls -s scaleio
ID                Name              Status       Size
c19deefe00000001  rexray01          unavailable  24
c19e160d00000002  rexray_scaleio_1  available    24
c19deefd00000000  vol01             attached     16

Create new volumes using each storage service.

[root@sunny ~]# rexray volume create -s isilon rexray_isilon_m1 --size=10
ID                Name              Status     Size
rexray_isilon_m1  rexray_isilon_m1  available  0

[root@sunny ~]# rexray volume create -s scaleio rexray_scaleio_m1 --size=10
ID                Name               Status     Size
c19e160e00000003  rexray_scaleio_m1  available  16

Start a new container

[root@sunny ~]# docker run -it -v  rexray_isilon_m1:/mnt centos:latest /bin/bash
[root@440a1f9a8ba9 /]# df -Th
Filesystem                                                                                          Type   Size  Used Avail Use% Mounted on
/dev/mapper/docker-253:0-134453907-0be39430bd2ca36f9b541bcf236eec0e0a03a5ac5dbffb0dc3dd7bafae913bbc xfs     10G  237M  9.8G   3% /
tmpfs                                                                                               tmpfs  1.9G     0  1.9G   0% /dev
tmpfs                                                                                               tmpfs  1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mapper/centos-root                                                                             xfs     38G  3.8G   34G  11% /mnt
shm                                                                                                 tmpfs   64M     0   64M   0% /dev/shm
tmpfs                                                                                               tmpfs  1.9G     0  1.9G   0% /sys/firmware

It seems to be working fine. Even though it is bothering to set service flag every time to run REX-Ray commands, it must be useful in some cases to operate multiple storages from one REX-Ray server.