Introduction
Global Replication Service (GRS) is now available between some DCs of Power Virtual Server.
https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-getting-started-GRS
Appendix A. of Redbook is also helpful.
GRS is a feature that allows asynchronous replication of storage volumes to another datacenter.
Currently, GRS functionality can be operated via CLI/API.
The pairs of data centers that can configure GRS are determined by the PowerVS service, and in this article I will use the pair of WDC06 and DAL12.
I will proceed with the configuration with WDC06 as the primary site(production site) and DAL12 as the secondary site(DR site).
Operation on primary site
Use the ibmcloud pi service-target command to switch the work target to the primary site (WDC06).
(1) Create a volume for the primary site.
Create a volume with the --replication-enabled option.
$ ibmcloud pi volume-create tmgw-vol-grs --type tier1 --size 10 --replication-enabled
Creating volume tmgw-vol-grs-2 under account XXX as user XXXXXXXXXX...
ID 8bd3fad5-7901-405f-b181-88c5e61f0332
Name tmgw-vol-grs
Profile tier1
Status creating
Size 10
Created 2023-11-16T06:56:35.000Z
Updated 0001-01-01T00:00:00.000Z
Shareable false
Bootable false
Storage Pool Tier1-Flash-1
Replication Enabled true
Auxiliary -
PVMInstanceIDs -
WWN -
$
(2) Create an AIX server from GUI as usual in the primary site, and use the volume created in (1) as a VG for data.
Create a file for testing to confirm that replication has taken place.
# lspv
hdisk0 00fa00d6b56a99bb rootvg active
hdisk1 00c91120d777dfdf datavg active
# lsvg -l datavg
datavg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
datalv jfs2 246 246 1 open/syncd /data
loglv00 jfs2log 1 1 1 open/syncd N/A
# ls /data
1G.dummy lost+found test.txt
#
(3) Create a volume group by specifying the volumes at the primary site.
Note that "volume group" here is a GRS term, different from AIX volume group.
$ ic pi vgc --volume-group-name vg-tmgw-01 --member-volume-ids 8bd3fad5-7901-405f-b181-88c5e61f0332
Creating volume group vg-tmgw-01 under account XXX as user XXXXXXXXXX...
Volume group vg-tmgw-01 created.
Name ID Status Status Details
vg-tmgw-01 42e368d1-3c45-43d4-bbea-2308261fb763 available -
$
Operation on secondary site
Use the ibmcloud pi service-target command to switch to the secondary site (DAL12).
(1) Creating an auxiliary volume
(1)-1
Check the CRN of the primary PowerVS workspace.
$ ic pi sl|grep -i wdc06
Listing services under account XXX as user XXXXXXXXXX...
crn:v1:bluemix:public:power-iaas:wdc06:a/039dbe6794084c7cb514a276dd2345da:e9812314-e026-48d8-8917-313dd26fb954:: Power Systems Virtual Server-WDC06
$
(1)-2
Output the properties of the primary volume in json format and check the auxVolumeName value.
$ ic pi vol 8bd3fad5-7901-405f-b181-88c5e61f0332 --json
{
"auxVolumeName": "aux_volume-tmgw-vol-grs-8bd3fad5-790128730651",
"auxiliary": false,
"bootable": false,
"consistencyGroupName": "rccg-42e3-fb763",
.
.
}
$
Execute the command for onboarding with the above two parameters.
("Onboarding" may be a peculiar term, but it is best understood as the process of creating a volume in the secondary site, based on the primary volume.)
$ ibmcloud pi volume-onboarding-create --source-crn crn:v1:bluemix:public:power-iaas:wdc06:a/039dbe6794084c7cb514a276dd2345da:e9812314-e026-48d8-8917-313dd26fb954:: --auxiliary-volume aux_volume-tmgw-vol-grs-8bd3fad5-790128730651
Creating volume onboarding under account XXX as user XXXXXXXXXX...
ID c0699c9d-5bd0-4401-b7bb-25b4c6b6c391
Description -
$
List of onboarding Processes
$ ibmcloud pi volume-onboardings
Listing volume onboarding operations under account XXX as user XXXXXXXXXX...
ID Status Description Volumes
c0699c9d-5bd0-4401-b7bb-25b4c6b6c391 SUCCESS onboard_aux_vols_LCJRDNN8 aux_volume-tmgw-vol-grs-8bd3fad5-790128730651
$
Details of the onboarding process
$ ibmcloud pi volume-onboarding c0699c9d-5bd0-4401-b7bb-25b4c6b6c391
Getting volume onboarding c0699c9d-5bd0-4401-b7bb-25b4c6b6c391 details under account XXX as user XXXXXXXXXX...
ID c0699c9d-5bd0-4401-b7bb-25b4c6b6c391
Status SUCCESS
Description onboard_aux_vols_LCJRDNN8
Volumes: aux_volume-tmgw-vol-grs-8bd3fad5-790128730651
Creation Time 2023-11-16T09:04:21.000Z
Progress 100
Onboarded Volumes aux_volume-tmgw-vol-grs-8bd3fad5-790128730651
Failures -
$
As a result of the onboarding, an auxiliary volume was created in the secondary site (DAL12), and we can see that the Auxiliary entry is true.
$ ibmcloud pi volumes
Listing volumes under account XXX as user XXXXXXXXXX...
ID Name Address
79a9d716-d9bb-4062-8cb3-243994dd9c8f aux_volume-tmgw-vol-grs-8bd3fad5-790128730651 /pcloud/v1/cloud-instances/8d08e1b20a9f41349d226d276c079dbe/volumes/79a9d716-d9bb-4062-8cb3-243994dd9c8f
$
$ ibmcloud pi volume 79a9d716-d9bb-4062-8cb3-243994dd9c8f
Getting volume 79a9d716-d9bb-4062-8cb3-243994dd9c8f under account XXX as user XXXXXXXXXX...
ID 79a9d716-d9bb-4062-8cb3-243994dd9c8f
Name aux_volume-tmgw-vol-grs-8bd3fad5-790128730651
Profile tier1
Status in-use
Size 10
Created 2023-11-16T09:06:22.000Z
Updated 2023-11-16T12:16:26.000Z
Shareable false
Bootable false
Storage Pool Tier1-Flash-3
Replication Enabled true
Auxiliary true
PVMInstanceIDs aa450365-cf50-432a-a901-30aa0ebc2b16
WWN 60050768108182530000000000003A33
$
(2)
Create an AIX server from GUI as usual in the secondary site, and use the auxiliary volume created in (1) as a VG for data.
While the replication relationship is established, the file systems on the auxiliary volume cannot be mounted.
# mount /data
Replaying log for /dev/datalv.
mount: /dev/datalv on /data: Unformatted or incompatible media
The superblock on /dev/datalv is dirty. Run a full fsck to fix.
#
The progress of replication can be checked by the Progress value of the following command.
$ ibmcloud pi volume-group-remote-copy-relationships 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting remote copy relationships for volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
Name Auxiliary Volume Name Consistency Group Name Master Volume Name Remote Copy ID State Copy Type Cycling Mode Freeze Time Primary Role Progress
rcrel97 aux_volume-tmgw-vol-grs-8bd3fad5-790128730651 rccg-42e3-fb763 volume-tmgw-vol-grs-8bd3fad5-7901 136 consistent_copying global multi 2023-11-22T03:13:55.000Z master 89
$ ibmcloud pi volume-group-remote-copy-relationships 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting remote copy relationships for volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
Name Auxiliary Volume Name Consistency Group Name Master Volume Name Remote Copy ID State Copy Type Cycling Mode Freeze Time Primary Role Progress
rcrel97 aux_volume-tmgw-vol-grs-8bd3fad5-790128730651 rccg-42e3-fb763 volume-tmgw-vol-grs-8bd3fad5-7901 136 consistent_copying global multi 2023-11-22T03:13:55.000Z master 89
$ ibmcloud pi volume-group-remote-copy-relationships 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting remote copy relationships for volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
Name Auxiliary Volume Name Consistency Group Name Master Volume Name Remote Copy ID State Copy Type Cycling Mode Freeze Time Primary Role Progress
rcrel97 aux_volume-tmgw-vol-grs-8bd3fad5-790128730651 rccg-42e3-fb763 volume-tmgw-vol-grs-8bd3fad5-7901 136 consistent_copying global multi 2023-11-22T03:22:06.000Z master 99
$ ibmcloud pi volume-group-remote-copy-relationships 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting remote copy relationships for volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
Name Auxiliary Volume Name Consistency Group Name Master Volume Name Remote Copy ID State Copy Type Cycling Mode Freeze Time Primary Role Progress
rcrel97 aux_volume-tmgw-vol-grs-8bd3fad5-790128730651 rccg-42e3-fb763 volume-tmgw-vol-grs-8bd3fad5-7901 136 consistent_copying global multi 2023-11-22T03:22:06.000Z master 99
$ ibmcloud pi volume-group-remote-copy-relationships 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting remote copy relationships for volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
Name Auxiliary Volume Name Consistency Group Name Master Volume Name Remote Copy ID State Copy Type Cycling Mode Freeze Time Primary Role Progress
rcrel97 aux_volume-tmgw-vol-grs-8bd3fad5-790128730651 rccg-42e3-fb763 volume-tmgw-vol-grs-8bd3fad5-7901 136 consistent_copying global multi 2023-11-22T03:30:38.000Z master 99
$ ibmcloud pi volume-group-remote-copy-relationships 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting remote copy relationships for volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
Name Auxiliary Volume Name Consistency Group Name Master Volume Name Remote Copy ID State Copy Type Cycling Mode Freeze Time Primary Role Progress
rcrel97 aux_volume-tmgw-vol-grs-8bd3fad5-790128730651 rccg-42e3-fb763 volume-tmgw-vol-grs-8bd3fad5-7901 136 consistent_copying global multi 2023-11-22T03:39:09.000Z master 100
$
Failover
Assuming that the primary site (WDC06) has been affected by the disaster, run the volume-group-stop command with the --allow-read-access option to allow access to the volume at the secondary site (DAL12).
Assuming that the primary site is affected and cannot be operated, we will operate in the environment of the secondary site.
$ ibmcloud pi volume-group-stop 2c3ba209-314c-4256-8bb4-5fd580bc069f --allow-read-access
Performing stop action on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
OK
Stop request on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f has been accepted.
$
Replication Status is now disabled.
$ ibmcloud pi volume-group 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
ID 2c3ba209-314c-4256-8bb4-5fd580bc069f
Name rccg-42e3-fb763
Consistency Group Name rccg-42e3-fb763
Replication Status disabled
Status available
Status Description -
$
Volume can be mounted on the secondary site, and files created in the primary site can be seen at the secondary site.
# mount /data
Replaying log for /dev/datalv.
# ls /data
1G.dummy lost+found test.txt
#
Failback
Assume that the primary site is recovered from the disaster. Please note that /data is unmounted at the primary site (WDC06).
Run the volume-group-start command with the --source auxiliary option, replicating in the direction of secondary site -> primary site, so that changes made at the secondary site during failover are reflected to the primary site.
$ ibmcloud pi volume-group-start 2c3ba209-314c-4256-8bb4-5fd580bc069f --source auxiliary
Performing start action on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
OK
Start request on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f has been accepted.
$
When the replication in the reverse direction is complete, stop the volume group again. And unmount /data at the secondary site (DAL12).
$ ibmcloud pi volume-group-stop 2c3ba209-314c-4256-8bb4-5fd580bc069f --allow-read-access
Performing stop action on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
OK
Stop request on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f has been accepted.
$
Now run volume-group-start without the --source auxiliary option to change the replication relationship to the original, primary site -> secondary site direction.
$ ibmcloud pi volume-group-start 2c3ba209-314c-4256-8bb4-5fd580bc069f
Performing start action on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
OK
Start request on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f has been accepted.
$
That's it!