What's this?
Nova instance が稼働している compute node が障害停止した際に、Nova instance を別の AZ の compute node 上に復旧(=退避)
* Nova instance を停止せずリアルタイムに別の compute node に移動する機能ではない
* あくまで shared storage 上のデータを使用して、別の compute node で起動し直す機能
[Case]
- Nova instance data is stored in Nova volume of shared storage. Cinder volume is attached to Nova instance.
- Nova instance data is stored in Cinder volume (= boot from Cinder volume), Cinder data is stored in Cinder volume of shared storage. Cinder volume is attached to Nova instance.
[条件]
- Shared storage あり
- Nova instance の全データは Shared storage にある
- Cinder volume が Nova instance に接続されている
- Hypervisor は KVM
[Case1] Nova instance data is stored in Nova volume of shared storage. Cinder volume is attached to Nova instance.
Initial state
Compute node
# nova service-list | egrep "Host|nova-compute"
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
| nova-compute | node-27 | nova | enabled | up | 2015-03-27T05:41:26.000000 | - |
| nova-compute | node-28 | nova | enabled | up | 2015-03-27T05:41:32.000000 | - |
| nova-compute | node-31 | nova2 | enabled | up | 2015-03-27T05:41:34.000000 | - |
AZ
# nova availability-zone-list | egrep -A 10 "^\| nova"
| nova | available |
| |- node-27 | |
| | |- nova-compute | enabled :-) 2015-03-27T05:41:36.000000 |
| |- node-28 | |
| | |- nova-compute | enabled :-) 2015-03-27T05:41:42.000000 |
| nova2 | available |
| |- node-31 | |
| | |- nova-compute | enabled :-) 2015-03-27T05:41:44.000000 |
+--------------------------+----------------------------------------+
Nova instance is running on node-28
# nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| ID | Name | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| 839d7887-7ea1-4efa-8d05-3543c0c08f19 | admin_vm01 | ACTIVE | active | nova | node-28 |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
Cinder volume is attached
# nova show admin_vm01 | egrep "Property|os-extended-volumes"
| Property | Value |
| os-extended-volumes:volumes_attached | [{"id": "59dda5f6-d8dd-475d-8445-3c23aa16e3b6"}] |
# cinder list | egrep "ID|59dda5f6-d8dd-475d-8445-3c23aa16e3b6"
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
| 59dda5f6-d8dd-475d-8445-3c23aa16e3b6 | in-use | admin_vol01 | 1 | None | false | 2a227a9c-801f-458b-ae67-29513e87282b |
Stop nova-compute service of node-28
Stop nova-compute service
# service openstack-nova-compute stop
# nova service-list | egrep "Host|nova-compute"
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
| nova-compute | node-27 | nova | enabled | up | 2015-03-27T05:49:56.000000 | - |
| nova-compute | node-28 | nova | enabled | down | 2015-03-27T05:48:02.000000 | - |
| nova-compute | node-31 | nova2 | enabled | up | 2015-03-27T05:49:54.000000 | - |
AZ
# nova availability-zone-list | egrep -A 10 "^\| nova"
| nova | available |
| |- node-27 | |
| | |- nova-compute | enabled :-) 2015-03-27T05:50:06.000000 |
| |- node-28 | |
| | |- nova-compute | enabled XXX 2015-03-27T05:48:02.000000 |
| nova2 | available |
| |- node-31 | |
| | |- nova-compute | enabled :-) 2015-03-27T05:50:04.000000 |
+--------------------------+----------------------------------------+
Evacuate
Execute evacuate
Evacuate Nova instance from node-28(inactive) of AZ:nova to node-31(active) of AZ:nova2
# nova evacuate --on-shared-storage 839d7887-7ea1-4efa-8d05-3543c0c08f19 node-31
Check the evacuate result
Nova instance is running on node-31(active) by using the previous data stored in shared storage
# nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| ID | Name | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| 839d7887-7ea1-4efa-8d05-3543c0c08f19 | admin_vm01 | ACTIVE | active | nova2 | node-31 |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
Cinder volume(vdb) is attached even after evacuate
$ ssh cirros@<INSTANCE> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 1G 0 disk
`-vda1 253:1 0 1011.9M 0 part /
vdb 253:16 0 1G 0 disk
[Case2] Nova instance data is stored in Cinder volume (= boot from Cinder volume), Cinder data is stored in Cinder volume of shared storage. Cinder volume is attached to Nova instance.
Initial state
Compute node
# nova service-list | egrep "Host|nova-compute"
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
| nova-compute | node-27 | nova | enabled | up | 2015-03-27T05:41:26.000000 | - |
| nova-compute | node-28 | nova | enabled | up | 2015-03-27T05:41:32.000000 | - |
| nova-compute | node-31 | nova2 | enabled | up | 2015-03-27T05:41:34.000000 | - |
AZ
# nova availability-zone-list | egrep -A 10 "^\| nova"
| nova | available |
| |- node-27 | |
| | |- nova-compute | enabled :-) 2015-03-27T05:41:36.000000 |
| |- node-28 | |
| | |- nova-compute | enabled :-) 2015-03-27T05:41:42.000000 |
| nova2 | available |
| |- node-31 | |
| | |- nova-compute | enabled :-) 2015-03-27T05:41:44.000000 |
+--------------------------+----------------------------------------+
Nova instance is running on node-28
# nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| ID | Name | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| 01983a98-7d5b-498b-b535-91941e5ac956 | admin_vm02 | ACTIVE | active | nova | node-28 |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
Cinder volume is attached
# nova show admin_vm02 | egrep "Property|os-extended-volumes|image"
| Property | Value |
| image | Attempt to boot from volume - no image supplied |
| os-extended-volumes:volumes_attached | [{"id": "75bf2311-ea23-48fd-b9c4-bb5173f3464f"}, {"id": "805fea46-28f8-4acc-84cf-e320c42d3d0d"}] |
# cinder list | egrep "ID|75bf2311-ea23-48fd-b9c4-bb5173f3464f|805fea46-28f8-4acc-84cf-e320c42d3d0d"
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
| 75bf2311-ea23-48fd-b9c4-bb5173f3464f | in-use | admin_vol02 | 1 | None | true | 4c0b84ec-3647-4bd9-b222-d07e3ce9a8db |
| 805fea46-28f8-4acc-84cf-e320c42d3d0d | in-use | admin_vol03 | 1 | None | false | 4c0b84ec-3647-4bd9-b222-d07e3ce9a8db |
Stop nova-compute service of node-28
Stop nova-compute service
# service openstack-nova-compute stop
# nova service-list | egrep "Host|nova-compute"
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
| nova-compute | node-27 | nova | enabled | up | 2015-03-27T05:49:56.000000 | - |
| nova-compute | node-28 | nova | enabled | down | 2015-03-27T05:48:02.000000 | - |
| nova-compute | node-31 | nova2 | enabled | up | 2015-03-27T05:49:54.000000 | - |
AZ
# nova availability-zone-list | egrep -A 10 "^\| nova"
| nova | available |
| |- node-27 | |
| | |- nova-compute | enabled :-) 2015-03-27T05:50:06.000000 |
| |- node-28 | |
| | |- nova-compute | enabled XXX 2015-03-27T05:48:02.000000 |
| nova2 | available |
| |- node-31 | |
| | |- nova-compute | enabled :-) 2015-03-27T05:50:04.000000 |
+--------------------------+----------------------------------------+
Evacuate
Execute evacuate
Evacuate Nova instance from node-28(inactive) of AZ:nova to node-31(active) of AZ:nova2
# nova evacuate --on-shared-storage 01983a98-7d5b-498b-b535-91941e5ac956 node-31
Check the evacuate result
Nova instance is running on node-31(active) by using the previous data stored in Cinder volume of shared storage
# nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| ID | Name | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| 01983a98-7d5b-498b-b535-91941e5ac956 | admin_vm02 | ACTIVE | active | nova2 | node-31 |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
Cinder volume(vdb) is attached even after evacuate
$ ssh cirros@<INSTANCE> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 1G 0 disk
`-vda1 253:1 0 1011.9M 0 part /
vdb 253:16 0 1G 0 disk
Environment
- Openstack : Juno
- Ceph : Firefly
Reference
Openstack - Nova Evacuate with Shared Storage
Openstack - Nova Evacuate with Shared Storage with Cinder
Openstack - Nova Evacuate with Shared Storage with Cinder between AZ
Openstack - Nova Evacuate without Shared Storage
Openstack - Nova Host-Evacuate with Shared Storage
Openstack - Nova Host-Evacuate with Shared Storage between AZ