LoginSignup
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

@idzzy

Openstack - Nova Evacuate with Shared Storage

Last updated at Posted at 2015-12-13

What's this?

Nova instance が稼働している compute node が障害停止した際に、Nova instance を別の compute node 上に復旧(=退避)

  • Nova instance を停止せずリアルタイムに別の compute node に移動する機能ではない
  • あくまで shared storage 上のデータを使用して、別の compute node で起動し直す機能

[Case]

  1. Nova instance data is stored in Nova volume of shared storage
  2. Nova instance data is stored in Cinder volume (= boot from Cinder volume), Cinder data is stored in Cinder volume of shared storage

[条件]

  • Shared storage あり
  • Nova instance の全データは Shared storage にある
  • 追加 Volume としての Cinder 未使用
  • Hypervisor は KVM

[Case1] Nova Evacuate (Nova instance data is stored in Nova volume of shared storage)

initial state

Compute node

# nova service-list | egrep "Host|nova-compute"
| Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
| nova-compute     | node-27 | nova     | enabled | up    | 2015-03-17T08:36:27.000000 | -               |
| nova-compute     | node-28 | nova     | enabled | up    | 2015-03-17T08:36:24.000000 | -               |
| nova-compute     | node-31 | nova2    | enabled | up    | 2015-03-17T08:36:24.000000 | None            |

AZ

# nova availability-zone-list | egrep -A 10 "^\| nova"
| nova                     | available                              |
| |- node-27               |                                        |
| | |- nova-compute        | enabled :-) 2015-03-17T08:36:37.000000 |
| |- node-28               |                                        |
| | |- nova-compute        | enabled :-) 2015-03-17T08:36:34.000000 |
| nova2                    | available                              |
| |- node-31               |                                        |
| | |- nova-compute        | enabled :-) 2015-03-17T08:36:34.000000 |
+--------------------------+----------------------------------------+

Nova instance is running on node-28

# nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| ID                                   | Name       | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| 322d9761-c264-4b33-aaed-5175cba464d2 | admin_vm01 | ACTIVE | active               | nova                         | node-28                              |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+

Stop nova-compute service of node-28

Stop nova-compute service

# service openstack-nova-compute stop

# nova service-list | egrep "Host|nova-compute"
| Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
| nova-compute     | node-27 | nova     | enabled | up    | 2015-03-17T08:38:17.000000 | -               |
| nova-compute     | node-28 | nova     | enabled | down  | 2015-03-17T08:37:14.000000 | -               |
| nova-compute     | node-31 | nova2    | enabled | up    | 2015-03-17T08:38:14.000000 | None            |

AZ

# nova availability-zone-list | egrep -A 10 "^\| nova"
| nova                     | available                              |
| |- node-27               |                                        |
| | |- nova-compute        | enabled :-) 2015-03-17T08:38:17.000000 |
| |- node-28               |                                        |
| | |- nova-compute        | enabled XXX 2015-03-17T08:37:14.000000 |
| nova2                    | available                              |
| |- node-31               |                                        |
| | |- nova-compute        | enabled :-) 2015-03-17T08:38:14.000000 |
+--------------------------+----------------------------------------+

Evacuate

Execute evacuate

Evacuate Nova instance from node-28(inactive) to node-27(active)

nova evacuate --on-shared-storage 322d9761-c264-4b33-aaed-5175cba464d2 node-27

Nova log

node-27 nova-compute 28104 AUDIT nova.compute.manager [req-501c487c-4648-4c4e-a757-489783eb60ba None] [instance: 322d9761-c264-4b33-aaed-5175cba464d2] Rebuilding instance
node-27 nova-compute 28104 INFO nova.compute.manager [req-501c487c-4648-4c4e-a757-489783eb60ba None] disk on shared storage, recreating using existing disk
node-27 nova-compute 28104 INFO nova.virt.libvirt.driver [req-501c487c-4648-4c4e-a757-489783eb60ba None] [instance: 322d9761-c264-4b33-aaed-5175cba464d2] Creating image
node-27 nova-compute 28104 INFO nova.compute.manager [-] Lifecycle event 0 on VM 322d9761-c264-4b33-aaed-5175cba464d2
node-27 nova-compute 28104 INFO nova.virt.libvirt.driver [-] [instance: 322d9761-c264-4b33-aaed-5175cba464d2] Instance spawned successfully.
node-27 nova-compute 28104 INFO nova.compute.manager [req-82a47a47-acf8-48a4-9ea4-5b2333ee557d None] [instance: 322d9761-c264-4b33-aaed-5175cba464d2] During sync_power_state the instance has a pending task. Skip.
node-27 nova-compute 28104 INFO nova.compute.manager [req-501c487c-4648-4c4e-a757-489783eb60ba None] bringing vm to original state: 'active'

Check the evacuate result

Nova instance is running on node-27(active) by using the previous data stored in shared storage

# nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| ID                                   | Name       | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| 322d9761-c264-4b33-aaed-5175cba464d2 | admin_vm01 | ACTIVE | active               | nova                         | node-27                              |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+

[Case2] Nova instance data is stored in Cinder volume (= boot from Cinder volume), Cinder data is stored in Cinder volume of shared storage

initial state

Compute node

# nova service-list | egrep "Host|nova-compute"
| Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
| nova-compute     | node-27 | nova     | enabled | up    | 2015-03-23T05:29:26.000000 | -               |
| nova-compute     | node-28 | nova     | enabled | up    | 2015-03-23T05:29:21.000000 | -               |
| nova-compute     | node-31 | nova2    | enabled | up    | 2015-03-23T05:29:25.000000 | None            |

AZ

# nova availability-zone-list | egrep -A 10 "^\| nova"
| nova                     | available                              |
| |- node-27               |                                        |
| | |- nova-compute        | enabled :-) 2015-03-23T05:29:26.000000 |
| |- node-28               |                                        |
| | |- nova-compute        | enabled :-) 2015-03-23T05:29:31.000000 |
| nova2                    | available                              |
| |- node-31               |                                        |
| | |- nova-compute        | enabled :-) 2015-03-23T05:29:25.000000 |
+--------------------------+----------------------------------------+

Nova instance is running on node-28

# nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
+--------------------------------------+-------------+--------+----------------------+------------------------------+--------------------------------------+
| ID                                   | Name        | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
+--------------------------------------+-------------+--------+----------------------+------------------------------+--------------------------------------+
| 69546468-97d3-487b-a91c-7b93cd502929 | admin_vm11  | ACTIVE | active               | nova                         | node-28                              |
+--------------------------------------+-------------+--------+----------------------+------------------------------+--------------------------------------+

All Nova data is stored in Cinder volume (= boot from Cinder volume)

# nova show 69546468-97d3-487b-a91c-7b93cd502929 | egrep "Property|image|volumes_attached"
| Property                             | Value                                                    |
| image                                | Attempt to boot from volume - no image supplied          |
| os-extended-volumes:volumes_attached | [{"id": "9d3ac7f8-db69-41e4-87be-1d94f2358144"}]         |

Stop nova-compute service of node-28

Stop nova-compute service

# /etc/init.d/openstack-nova-compute stop

# nova service-list | egrep "Host|nova-compute"
| Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
| nova-compute     | node-27 | nova     | enabled | up    | 2015-03-23T05:32:46.000000 | -               |
| nova-compute     | node-28 | nova     | enabled | down  | 2015-03-23T05:31:31.000000 | -               |
| nova-compute     | node-31 | nova2    | enabled | up    | 2015-03-23T05:32:55.000000 | None            |

AZ

# nova availability-zone-list | egrep -A 10 "^\| nova"
| nova                     | available                              |
| |- node-27               |                                        |
| | |- nova-compute        | enabled :-) 2015-03-23T05:32:56.000000 |
| |- node-28               |                                        |
| | |- nova-compute        | enabled XXX 2015-03-23T05:31:31.000000 |
| nova2                    | available                              |
| |- node-31               |                                        |
| | |- nova-compute        | enabled :-) 2015-03-23T05:32:55.000000 |
+--------------------------+----------------------------------------+

Evacuate

Execute evacuate

Evacuate Nova instance from node-28(inactive) to node-27(active)

nova evacuate --on-shared-storage 69546468-97d3-487b-a91c-7b93cd502929 node-27

Nova log

node-27 nova-compute 26063 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-27
node-27 nova-compute 26063 AUDIT nova.compute.manager [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] [instance: 69546468-97d3-487b-a91c-7b93cd502929] Rebuilding instance
node-27 nova-compute 26063 INFO nova.compute.manager [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] disk on shared storage, recreating using existing disk
node-27 nova-compute 26063 INFO urllib3.connectionpool [req-a41f2a0e-d557-488a-807f-40c23d6a5988 ] Starting new HTTP connection (1): 10.30.0.2
node-27 nova-compute 26063 AUDIT nova.virt.block_device [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] [instance: 69546468-97d3-487b-a91c-7b93cd502929] Booting with volume 9d3ac7f8-db69-41e4-87be-1d94f2358144 at vda
node-27 nova-compute 26063 INFO urllib3.connectionpool [req-a41f2a0e-d557-488a-807f-40c23d6a5988 ] Starting new HTTP connection (1): 10.30.0.2
node-27 nova-compute 26063 WARNING nova.virt.libvirt.utils [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] systool is not installed
node-27 nova-compute 26063 WARNING nova.virt.libvirt.utils [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] systool is not installed
node-27 nova-compute 26063 INFO urllib3.connectionpool [req-a41f2a0e-d557-488a-807f-40c23d6a5988 ] Starting new HTTP connection (1): 10.30.0.2
node-27 nova-compute 26063 INFO urllib3.connectionpool [req-a41f2a0e-d557-488a-807f-40c23d6a5988 ] Starting new HTTP connection (1): 10.30.0.2
node-27 nova-compute 26063 INFO nova.virt.libvirt.driver [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] [instance: 69546468-97d3-487b-a91c-7b93cd502929] Creating image
node-27 nova-compute 26063 INFO nova.compute.manager [-] Lifecycle event 0 on VM 69546468-97d3-487b-a91c-7b93cd502929
node-27 nova-compute 26063 INFO nova.virt.libvirt.driver [-] [instance: 69546468-97d3-487b-a91c-7b93cd502929] Instance spawned successfully.
node-27 nova-compute 26063 INFO nova.compute.manager [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] bringing vm to original state: 'active'

Check the evacuate result

Nova instance is running on node-27(active) by using the previous data stored in Cinder volume of shared storage

# nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| ID                                   | Name       | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
| 69546468-97d3-487b-a91c-7b93cd502929 | admin_vm11 | ACTIVE | active               | nova                         | node-27                              |
+--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+

Environment

  • Openstack : Juno
  • Ceph : Firefly

Reference

Openstack - Nova Evacuate with Shared Storage
Openstack - Nova Evacuate with Shared Storage with Cinder
Openstack - Nova Evacuate with Shared Storage with Cinder between AZ
Openstack - Nova Evacuate without Shared Storage
Openstack - Nova Host-Evacuate with Shared Storage
Openstack - Nova Host-Evacuate with Shared Storage between AZ

0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?