Various nova instance migration techniques.

In this article, I am going to list the various nova instances techniques. I have used my packstack all-in-one setup and two extra compute nodes to show these tests. I am using the local storage

  • Offline storage migration : Downtime required.

As my instances ephemeral disks are configured on local storage hence the first migration which comes to our mind is the offline migration :

[root@allinone6 ~(keystone_admin)]# nova migrate test-instance1 –poll

Above command will not give the option to specify the destination host on which we want to run the instance, scheduler will choose the destination host for you.

Once the migration is completed successfully, you will see the instance is running (ACTIVE) on other compute node.

I have seen the below state of instance during the migration.

ACTIVE –> RESIZE –> VERIFY_RESIZE –> ACTIVE

If I am checking the instance action list I can see that it has performed the migrate and resize operations both.

[root@allinone6 ~(keystone_admin)]# nova instance-action-list test-instance1
+—————+——————————————+———+—————————-+
| Action        | Request_ID                               | Message | Start_Time                 |
+—————+——————————————+———+—————————-+
| create        | req-93d78dbe-8914-46b9-9605-0e9ff7ed76e8 | –       | 2016-03-06T02:13:57.000000 |
| migrate       | req-f0c214d7-d5ed-4633-a147-056dad6611a2 | –       | 2016-03-07T05:04:26.000000 |
| confirmResize | req-6d97c3cf-a509-4e6c-a016-457569ca46b3 | –       | 2016-03-07T05:05:14.000000 |
+—————+——————————————+———+—————————-+

Even thou it’s performing the resize but flavor will remain the same. Only reason which I can find for resize is that migrate and resize are sharing the same code.

Below are the nova-api.log from the controller node.

[root@allinone6 ~(keystone_admin)]# grep ‘req-f0c214d7-d5ed-4633-a147-056dad6611a2’ /var/log/nova/nova-api.log
2016-03-07 00:04:26.198 3819 DEBUG nova.api.openstack.wsgi [req-f0c214d7-d5ed-4633-a147-056dad6611a2 None] Action: ‘action’, calling method: <bound method AdminActionsController._migrate of <nova.api.openstack.compute.contrib.admin_actions.AdminActionsController object at 0x3aff8d0>>, body: {“migrate”: null} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:934
2016-03-07 00:04:26.228 3819 DEBUG nova.compute.api [req-f0c214d7-d5ed-4633-a147-056dad6611a2 None] [instance: c8cb4bcc-2b4e-4478-9a7c-61f5170fb177] flavor_id is None. Assuming migration. resize /usr/lib/python2.7/site-packages/nova/compute/api.py:2559
2016-03-07 00:04:26.229 3819 DEBUG nova.compute.api [req-f0c214d7-d5ed-4633-a147-056dad6611a2 None] [instance: c8cb4bcc-2b4e-4478-9a7c-61f5170fb177] Old instance type m1.tiny,  new instance type m1.tiny resize /usr/lib/python2.7/site-packages/nova/compute/api.py:2578
2016-03-07 00:04:26.305 3819 INFO oslo.messaging._drivers.impl_rabbit [req-f0c214d7-d5ed-4633-a147-056dad6611a2 ] Connecting to AMQP server on 192.168.122.234:5672
2016-03-07 00:04:26.315 3819 INFO oslo.messaging._drivers.impl_rabbit [req-f0c214d7-d5ed-4633-a147-056dad6611a2 ] Connected to AMQP server on 192.168.122.234:5672
2016-03-07 00:04:26.563 3819 INFO nova.osapi_compute.wsgi.server [req-f0c214d7-d5ed-4633-a147-056dad6611a2 None] 192.168.122.234 “POST /v2/618cb39791784d7fb7a80d17eb99b306/servers/c8cb4bcc-2b4e-4478-9a7c-61f5170fb177/action HTTP/1.1” status: 202 len: 209 time: 0.4026990

It’s a offline operation, we can confirm the same using uptime of instance.

[root@allinone6 ~(keystone_admin)]# ip netns exec qdhcp-0b9572fb-29fe-4705-b50c-74aa00acb983 ssh cirros@10.10.3.15
cirros@10.10.3.15’s password:
$ uptime
22:05:31 up 0 min,  1 users,  load average: 0.12, 0.04, 0.01

  • Evacuating the instance from failed compute node.

This makes sense while using the shared storage.

Instance is running on compute26 and I shutdown the same node, instance remains in the ACTIVE state but I am not able to ping the instance. Actually the instance is down.
[root@allinone6 ~(keystone_admin)]# nova service-list
+—-+——————+———–+———-+———+——-+—————————-+—————–+
| Id | Binary           | Host      | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+—-+——————+———–+———-+———+——-+—————————-+—————–+
| 1  | nova-consoleauth | allinone6 | internal | enabled | up    | 2016-03-07T05:20:44.000000 | –               |
| 2  | nova-scheduler   | allinone6 | internal | enabled | up    | 2016-03-07T05:20:44.000000 | –               |
| 3  | nova-conductor   | allinone6 | internal | enabled | up    | 2016-03-07T05:20:44.000000 | –               |
| 5  | nova-compute     | allinone6 | nova     | enabled | up    | 2016-03-07T05:20:39.000000 | –               |
| 6  | nova-cert        | allinone6 | internal | enabled | up    | 2016-03-07T05:20:44.000000 | –               |
| 7  | nova-compute     | compute26 | nova     | enabled | down  | 2016-03-07T05:19:19.000000 | –               |
| 8  | nova-compute     | compute16 | nova     | enabled | up    | 2016-03-07T05:20:44.000000 | –               |
+—-+——————+———–+———-+———+——-+—————————-+—————–+

Started the evacuation of instance[,s] from the failed node using below command, in this case we can specify the destination compute node.

[root@compute16 ~(keystone_admin)]# nova host-evacuate –target_host allinone6 compute26
+————————————–+——————-+—————+
| Server UUID                          | Evacuate Accepted | Error Message |
+————————————–+——————-+—————+
| c8cb4bcc-2b4e-4478-9a7c-61f5170fb177 | True              |               |
+————————————–+——————-+—————+

Below is the state of instances which I noticed in nova list output.

ACTIVE — REBUILD — ACTIVE

In below command, we can see the evacuate task is inserted.

[root@allinone6 ~(keystone_admin)]# nova instance-action-list test-instance1
+—————+——————————————+———+—————————-+
| Action        | Request_ID                               | Message | Start_Time                 |
+—————+——————————————+———+—————————-+
| create        | req-93d78dbe-8914-46b9-9605-0e9ff7ed76e8 | –       | 2016-03-06T02:13:57.000000 |
| migrate       | req-f0c214d7-d5ed-4633-a147-056dad6611a2 | –       | 2016-03-07T05:04:26.000000 |
| confirmResize | req-6d97c3cf-a509-4e6c-a016-457569ca46b3 | –       | 2016-03-07T05:05:14.000000 |
| migrate       | req-43e92f8e-04d6-4379-98c1-8ce72094766f | –       | 2016-03-07T05:17:09.000000 |
| confirmResize | req-4a11404d-e448-4692-86f0-063a0dfd2d4a | –       | 2016-03-07T05:17:42.000000 |
| evacuate      | req-1b22f414-36c1-487e-9560-359e2ecd2800 | –       | 2016-03-07T05:22:18.000000 |
+—————+——————————————+———+—————————-+

We can see the below nova-api.log corresponding to “evacuate” operation.

[root@allinone6 ~(keystone_admin)]# grep ‘req-1b22f414-36c1-487e-9560-359e2ecd2800’ /var/log/nova/nova-api.log
2016-03-07 00:22:18.162 3819 DEBUG nova.api.openstack.wsgi [req-1b22f414-36c1-487e-9560-359e2ecd2800 None] Action: ‘action’, calling method: <bound method Controller._evacuate of <nova.api.openstack.compute.contrib.evacuate.Controller object at 0x3b01f10>>, body: {“evacuate”: {“host”: “allinone6”, “onSharedStorage”: false}} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:934
2016-03-07 00:22:18.209 3819 DEBUG nova.compute.api [req-1b22f414-36c1-487e-9560-359e2ecd2800 None] [instance: c8cb4bcc-2b4e-4478-9a7c-61f5170fb177] vm evacuation scheduled evacuate /usr/lib/python2.7/site-packages/nova/compute/api.py:3258
2016-03-07 00:22:18.219 3819 DEBUG nova.servicegroup.drivers.db [req-1b22f414-36c1-487e-9560-359e2ecd2800 None] Seems service is down. Last heartbeat was 2016-03-07 05:19:19. Elapsed time is 179.219082 is_up /usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py:75
2016-03-07 00:22:18.270 3819 INFO nova.osapi_compute.wsgi.server [req-1b22f414-36c1-487e-9560-359e2ecd2800 None] 192.168.122.41 “POST /v2/618cb39791784d7fb7a80d17eb99b306/servers/c8cb4bcc-2b4e-4478-9a7c-61f5170fb177/action HTTP/1.1” status: 200 len: 225 time: 0.1504130

  • Live-migration [block migration]

In this case we are doing the live migration of instance despite of not having the shared storage configured for the instance. It will perform the scp of disk from source to destination compute node.

ACTIVE  –> MIGRATING –> ACTIVE

Below is the command for the live-migration.

[root@allinone6 ~(keystone_admin)]# nova live-migration –block-migrate test-instance1 compute16

In instance-action-list, you will not be able to see anything.

[root@allinone6 ~(keystone_admin)]# nova instance-action-list test-instance1
+——–+——————————————+———+—————————-+
| Action | Request_ID                               | Message | Start_Time                 |
+——–+——————————————+———+—————————-+
| create | req-93d78dbe-8914-46b9-9605-0e9ff7ed76e8 | –       | 2016-03-06T02:13:57.000000 |
+——–+——————————————+———+—————————-+

You can see the below logs in nova-api.log file while doing the live migration.

From : /var/log/nova/nova-api.log

~~~
2016-03-06 23:55:42.463 3820 INFO nova.osapi_compute.wsgi.server [req-22ca9202-8a90-4998-a0d2-6705e7bbfa71 None] 192.168.122.234 “GET /v2/618cb39791784d7fb7a80d17eb99b306/servers/c8cb4bcc-2b4e-4478-9a7c-61f5170fb177 HTTP/1.1” status: 200 len: 1903 time: 0.1395051
2016-03-06 23:55:42.468 3818 DEBUG nova.api.openstack.wsgi [req-6fbe4aa6-2e40-403d-b573-18c6f50669f5 None] Action: ‘action’, calling method: <bound method AdminActionsController._migrate_live of <nova.api.openstack.compute.contrib.admin_actions.AdminActionsController object at 0x3aff8d0>>, body: {“os-migrateLive”: {“disk_over_commit”: false, “block_migration”: true, “host”: “compute16”}} _process_stack /u
sr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:934
2016-03-06 23:55:42.505 3818 DEBUG nova.compute.api [req-6fbe4aa6-2e40-403d-b573-18c6f50669f5 None] [instance: c8cb4bcc-2b4e-4478-9a7c-61f5170fb177] Going to try to live migrate instance to compute16 live_migrate /usr/lib/python2.7/site-packages/nova/compute/api.py:3234
2016-03-06 23:55:42.765 3818 INFO nova.osapi_compute.wsgi.server [req-6fbe4aa6-2e40-403d-b573-18c6f50669f5 None] 192.168.122.234 “POST /v2/618cb39791784d7fb7a80d17eb99b306/servers/c8cb4bcc-2b4e-4478-9a7c-61f5170fb177/action HTTP/1.1” status: 202 len: 209 time: 0.2986290
2016-03-06 23:55:49.325 3819 DEBUG keystoneclient.session [-] REQ: curl -i -X GET http://192.168.122.234:35357/v3/auth/tokens -H “X-Subject-Token: TOKEN_REDACTED” -H “User-Agent: python-keystoneclient” -H “Accept: application/json” -H “X-Auth-Token: TOKEN_REDACTED” _http_log_request /usr/lib/python2.7/site-packages/keystoneclient/session.py:155
~~~

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s