"Host key verification failed" error while resizing instance Openstack

Setting allow_resize_to_same_host=true will enable the resizing in All in one model of installation . but if we have mutiple controller and compute node's we will be getting an authentication error as following

"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4907,
in migrate_disk_and_power_off\n utils.execute(\'ssh\', dest, \'mkdir\', \'-p\', inst_base)\n', '
File "/usr/lib/python2.6/site-packages/nova/utils.py", line 165, in execute\n return processutils.execute(*cmd, **kwargs)\n', '
File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 193, in execute\n cmd=\' \'.join(cmd))\n',
"ProcessExecutionError: Unexpected error while running command.\nCommand: ssh 10.1.15.44
mkdir -p /var/lib/nova/instances/5d5ced81-6fb1-4028-97cd-686e450d1bab\nExit code: 255\nStdout: ''\nStderr:
'Host key verification failed.\\r\\n'\n"]

This is due to a bug in nova as told in below bug report which can be solved by enabling password less authentication between the compute and controller node's nova user .

https://bugzilla.redhat.com/show_bug.cgi?id=975014#c3

By default the nova user does not have a shell so we need to enable the shell and do steps to enable password less authentication between all the server .

Enable shell for nova user

In all the controller and compute server’s enable nova user.

sed -i "s/\/var\/lib\/nova:\/sbin\/nologin/\/var\/lib\/nova:\/bin\/bash/g" /etc/passwd

cat /etc/passwd |grep nova

nova:x:162:162:OpenStack Nova Daemons:/var/lib/nova:/bin/bash

Enable password less authentication

Between all the compute and controller nodes create password less authenticatioin for user nova.

 

su - nova

ssh-keygen

ssh-copy-id

Creating Multiple Network in Openstack – Icehouse

Adding first network

On Network Node

#Add the integration bridge:
ovs-vsctl add-br br-int
#Add the external bridge:
ovs-vsctl add-br br-ex
#Add a port to the external bridge that connects to the physical external network interface:
#Replace INTERFACE_NAME with the actual interface name. For example, eth2 or ens256.
ovs-vsctl add-port br-ex eth2

On Controller Node

[root@controller1 ~]# source /root/admin-openrc.sh
[root@controller1 ~]# neutron net-create ext-net --shared --router:external=True
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 2fdfff06-5837-4ab8-b971-c08696165e9d |
| name | ext-net |
| provider:network_type | gre |
| provider:physical_network | |
| provider:segmentation_id | 1 |
| router:external | True |
| shared | True |
| status | ACTIVE |
| subnets | |
| tenant_id | 8c9fb6577a1e45879f35e3e43b34de58 |
+---------------------------+--------------------------------------+
[root@controller1 ~]#
[root@controller1 ~]# neutron subnet-create ext-net --name ext-subnet --allocation-pool start=192.168.10.129,end=192.168.10.254 --disable-dhcp --gateway 192.168.10.1 192.168.10.128/24
Created a new subnet:
+------------------+------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------+
| allocation_pools | {"start": "192.168.10.129", "end": "192.168.10.254"} |
| cidr | 192.168.10.0/24 |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | 192.168.10.1 |
| host_routes | |
| id | 09ec246e-4b1d-4fcb-babc-fdfd73ad13d0 |
| ip_version | 4 |
| name | ext-subnet |
| network_id | 2fdfff06-5837-4ab8-b971-c08696165e9d |
| tenant_id | 8c9fb6577a1e45879f35e3e43b34de58 |
+------------------+------------------------------------------------+
[root@controller1 ~]# neutron net-create demo-net
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | df86ab38-57f2-432b-8ea8-ef30ceb72607 |
| name | demo-net |
| provider:network_type | gre |
| provider:physical_network | |
| provider:segmentation_id | 2 |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | 8c9fb6577a1e45879f35e3e43b34de58 |
+---------------------------+--------------------------------------+
[root@controller1 ~]# neutron subnet-create demo-net --name demo-subnet --gateway 10.0.0.1 10.0.0.0/24
Created a new subnet:
+------------------+--------------------------------------------+
| Field | Value |
+------------------+--------------------------------------------+
| allocation_pools | {"start": "10.0.0.2", "end": "10.0.0.254"} |
| cidr | 10.0.0.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.0.0.1 |
| host_routes | |
| id | 7f103f72-1d77-48d9-a4eb-fdb10c6dc11d |
| ip_version | 4 |
| name | demo-subnet |
| network_id | df86ab38-57f2-432b-8ea8-ef30ceb72607 |
| tenant_id | 8c9fb6577a1e45879f35e3e43b34de58 |
+------------------+--------------------------------------------+
[root@controller1 ~]# neutron router-create demo-router
Created a new router:
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| admin_state_up | True |
| external_gateway_info | |
| id | d2e481dd-a624-4447-8ea7-54fac383ec52 |
| name | demo-router |
| status | ACTIVE |
| tenant_id | 8c9fb6577a1e45879f35e3e43b34de58 |
+-----------------------+--------------------------------------+
[root@controller1 ~]# neutron router-interface-add demo-router demo-subnet
uter demo-router
neutron router-gateway-set demo-router ext-net
Added interface 628fbec4-625b-4b03-a2c5-92666cbc72af to router demo-router.
[root@controller1 ~]# #Set gateway for router demo-router
[root@controller1 ~]# neutron router-gateway-set demo-router ext-net
Set gateway for router demo-router
On Network Node

Edit /etc/neutron/l3_agent.ini and add following details
handle_internal_only_routers = True
gateway_external_network_id = ****.*****.*****.*****.****
external_network_bridge = br-ex

As per above example
handle_internal_only_routers = True
gateway_external_network_id = 2fdfff06-5837-4ab8-b971-c08696165e9d
external_network_bridge = br-ex
Adding the Second Network

On Network Node

ovs-vsctl add-br br-ex-2
ovs-vsctl add-port br-ex-2 eth3
On Controller Node

[root@controller1 ~]# source /root/admin-openrc.sh
[root@controller1 ~]# neutron net-create ext-net-2 --shared --router:external=True
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 19b85183-9637-4bea-9b26-92caf2a5cb99 |
| name | ext-net-2 |
| provider:network_type | gre |
| provider:physical_network | |
| provider:segmentation_id | 3 |
| router:external | True |
| shared | True |
| status | ACTIVE |
| subnets | |
| tenant_id | 8c9fb6577a1e45879f35e3e43b34de58 |
+---------------------------+--------------------------------------+
[root@controller1 ~]# neutron subnet-create ext-net-2 --name ext-subnet-2 --allocation-pool start=192.168.11.50,end=192.168.11.254 --disable-dhcp --gateway 192.168.11.1 192.168.11.0/24
Created a new subnet:
+------------------+-----------------------------------------------+
| Field | Value |
+------------------+-----------------------------------------------+
| allocation_pools | {"start": "192.168.11.50", "end": "192.168.11.254"} |
| cidr | 192.168.11.0/24 |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | 192.168.11.1 |
| host_routes | |
| id | 92dcc66d-00a3-4678-a8ac-9d72a94613ed |
| ip_version | 4 |
| name | ext-subnet-2 |
| network_id | 19b85183-9637-4bea-9b26-92caf2a5cb99 |
| tenant_id | 8c9fb6577a1e45879f35e3e43b34de58 |
+------------------+-----------------------------------------------+
[root@controller1 ~]# neutron router-create demo-router-2
Created a new router:
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| admin_state_up | True |
| external_gateway_info | |
| id | e9e194bb-dafe-4e41-b867-0a64c2e74e29 |
| name | demo-router-2 |
| status | ACTIVE |
| tenant_id | 8c9fb6577a1e45879f35e3e43b34de58 |
+-----------------------+--------------------------------------+
[root@controller1 ~]# neutron router-gateway-set demo-router-2 ext-net-2
Set gateway for router demo-router-2
[root@controller1 ~]# neutron net-create demo-net-2
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | f6238c45-9de4-4869-ad3e-cf3d1a647285 |
| name | demo-net-2 |
| provider:network_type | gre |
| provider:physical_network | |
| provider:segmentation_id | 4 |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | 8c9fb6577a1e45879f35e3e43b34de58 |
+---------------------------+--------------------------------------+
[root@controller1 ~]# neutron subnet-create demo-net-2 --name demo-subnet-2 --gateway 10.1.0.1 10.1.0.0/24
Created a new subnet:
+------------------+--------------------------------------------+
| Field | Value |
+------------------+--------------------------------------------+
| allocation_pools | {"start": "10.1.0.2", "end": "10.1.0.254"} |
| cidr | 10.1.0.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.1.0.1 |
| host_routes | |
| id | 30bcb484-e7e7-42a5-9a08-b7364f94180a |
| ip_version | 4 |
| name | demo-subnet-2 |
| network_id | f6238c45-9de4-4869-ad3e-cf3d1a647285 |
| tenant_id | 8c9fb6577a1e45879f35e3e43b34de58 |
+------------------+--------------------------------------------+
[root@controller1 ~]# neutron router-interface-add demo-router-2 demo-subnet-2
Added interface daa4f64d-117c-49ee-a5b6-430c837a59d5 to router demo-router-2.
[root@controller1 ~]#

On Network Node

Edit /etc/neutron/l3_agent.ini and add following details
handle_internal_only_routers = False
gateway_external_network_id = ****.*****.*****.*****.****
external_network_bridge = br-ex

As per above example
handle_internal_only_routers = False
gateway_external_network_id = 19b85183-9637-4bea-9b26-92caf2a5cb99
external_network_bridge = br-ex-2

[root@neutron1 ~]# cp /etc/init.d/neutron-l3-agent /etc/init.d/neutron-l3-agent-2

Make the change in the following Lines to get the difference command as below
[root@neutron1 ~]# diff -n /etc/init.d/neutron-l3-agent /etc/init.d/neutron-l3-agent-2
d3 1
a3 1
# neutron-l3-agent-2 OpenStack Neutron Layer 3 Agent
d18 1
a18 1
"/etc/$proj/l3_agent-2.ini" \
d21 1
a21 1
pidfile="/var/run/$proj/$prog-2.pid"
d23 1
a23 1
[ -e /etc/sysconfig/$prog-2 ] && . /etc/sysconfig/$prog-2
d25 1
a25 1
lockfile=/var/lock/subsys/$prog-2
d32 2
a33 3
echo -n $"Starting $prog-2: "
daemon --user neutron --pidfile $pidfile "$exec --log-file /var/log/$proj/$plugin-2.log --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/l3_agent-2.ini --config-file /etc/neutron/fwaas_driver.ini &>/dev/null & echo \$! > $pidfile"

d41 2
a42 2
echo -n $"Stopping $prog-2: "
killproc -p $pidfile $prog-2
[root@neutron1 ~]#

[root@neutron1 ~]# chkconfig neutron-l3-agent-2 on
[root@neutron1 ~]# openstack-service restart
Stopping neutron-dhcp-agent: [ OK ]
Starting neutron-dhcp-agent: [ OK ]
Stopping neutron-l3-agent: [ OK ]
Starting neutron-l3-agent: [ OK ]
Stopping neutron-l3-agent-2: [ OK ]
Starting neutron-l3-agent-2: [ OK ]
Stopping neutron-metadata-agent: [ OK ]
Starting neutron-metadata-agent: [ OK ]
Stopping neutron-openvswitch-agent: [ OK ]
Starting neutron-openvswitch-agent: [ OK ]
[root@neutron1 ~]#

Openstack - Error While resizing the instance

Error While resizing the instance

=========
2014-08-12 09:38:08.449 13602 WARNING nova.scheduler.utils [req-7cf9b53a-24ed-46e7-adce-6c5e83c2839b 2ca1137e67d741e0839e67e
4959a8eea 8c9fb6577a1e45879f35e3e43b34de58] Failed to compute_task_migrate_server: No valid host was found.
=========
Solution

make sure following settings are enables in all the nova.conf

"allow_resize_to_same_host=true"
"allow_migrate_to_same_host=true"

After enabling it reboot all the openstack services and try the resizing from dashboard

We will be asked for an conformation for resizing, if we don't select any option resizing will fail

By command promt
[root@controller1 ~]# nova list
+--------------------------------------+------------------+--------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------+--------+------------+-------------+-------------------+
| 0c86c7bd-e815-4616-9c01-30cb8eb09414 | Test Instance 00 | ACTIVE | - | Running | demo-net=10.0.0.9 |
+--------------------------------------+------------------+--------+------------+-------------+-------------------+

List the available flavors with the following command:

$ nova flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 1 | m1.tiny | 512 | 0 | 0 | | 1 | 1.0 | True |
| 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True |
| 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True |
| 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True |
| 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
To resize the server, pass the server ID or name and the new flavor to the nova resize command. Include the --poll parameter to report the resize progress.

$ nova resize myCirrosServer 4 --poll
Instance resizing... 100% complete
Finished
Show the status for your server:

$ nova list

[root@controller1 ~]# nova list
+--------------------------------------+------------------+--------+------------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------+--------+------------------+-------------+-------------------+
| 0c86c7bd-e815-4616-9c01-30cb8eb09414 | Test Instance 00 | RESIZE | resize_migrating | Running | demo-net=10.0.0.9 |
+--------------------------------------+------------------+--------+------------------+-------------+-------------------+

When the resize completes, the status becomes VERIFY_RESIZE.

Confirm the resize:

$ nova resize-confirm 0c86c7bd-e815-4616-9c01-30cb8eb094145
The server status becomes ACTIVE.

If the resize fails or does not work as expected, you can revert the resize:

$ nova resize-revert 0c86c7bd-e815-4616-9c01-30cb8eb09414
The server status becomes ACTIVE.