Enabling Instance resizing In Openstack Juno

Editing Configuration
#Run on ALL Compute Server and Contoller Server

sed -i "s/#allow_resize_to_same_host.*/allow_resize_to_same_host=true/g" /etc/nova/nova.conf
sed -i "s/#allow_migrate_to_same_host.*/allow_migrate_to_same_host=true/g" /etc/nova/nova.conf

Configure the nova user
usermod -s /bin/bash nova

And enable password less authentication between Nova user’s in all server’s.

To Create Public and Private key for the user
ssh-keygen

To Copy the Public key to other users
ssh-copy-id <To all Server’s>

Add the Following Configruation file under Nova User in Every Server which has a Nova User
su - nova
cat << EOF > ~/.ssh/config
Host *
    StrictHostKeyChecking no
    UserKnownHostsFile=/dev/null
EOF

Openstack Recovering Data from Failed Instances Disk

Openstack Recovering Data from Failed Instances Disk

****************************
Qemu-nbd tools in Ubuntu
****************************

In some scenarios, instances are running but are inaccessible through SSH and do not respond to any command. The VNC console could be displaying a boot failure or kernel panic error messages. This could be an indication of file system corruption on the VM itself. If you need to recover files or inspect the content of the instance, qemu-nbd can be used to mount the disk.

We can get the path of the Instance by greping the Instance name from the common instance path.

>>egrep -i "Instance-name" /var/lib/nova/instances/*/*.xml

To access the instance's disk (/var/lib/nova/instances/xxxx-instance-uuid-xxxxxx/disk), use the following steps:
1.)Suspend the instance using the virsh command.
2.)Connect the qemu-nbd device to the disk.
3.)Mount the qemu-nbd device.
4.)Unmount the device after inspecting.
5.)Disconnect the qemu-nbd device.
6.)Resume the instance.

If you do not follow steps 4 through 6, OpenStack Compute cannot manage the instance any longer. It fails to respond to any command issued by OpenStack Compute, and it is marked as shut down.

Once you mount the disk file, you should be able to access it and treat it as a collection of normal directories with files and a directory structure. However, we do not recommend that you edit or touch any files because this could change the access control lists (ACLs) that are used to determine which accounts can perform what operations on files and directories. Changing ACLs can make the instance unbootable if it is not already.

Suspend the instance using the virsh command, taking note of the internal ID:

# virsh list
Id Name                 State
----------------------------------
1 instance-00000981    running
2 instance-000009f5    running
30 instance-0000274a    running

# virsh suspend 30
Domain 30 suspended
Connect the qemu-nbd device to the disk:

# cd /var/lib/nova/instances/instance-0000274a
# ls -lh
total 33M
-rw-rw---- 1 libvirt-qemu kvm  6.3K Jan 15 11:31 console.log
-rw-r--r-- 1 libvirt-qemu kvm   33M Jan 15 22:06 disk
-rw-r--r-- 1 libvirt-qemu kvm  384K Jan 15 22:06 disk.local
-rw-rw-r-- 1 nova         nova 1.7K Jan 15 11:30 libvirt.xml
# qemu-nbd -c /dev/nbd0 `pwd`/disk
Mount the qemu-nbd device.

The qemu-nbd device tries to export the instance disk's different partitions as separate devices. For example, if vda is the disk and vda1 is the root partition, qemu-nbd exports the device as /dev/nbd0 and /dev/nbd0p1, respectively:

# mount /dev/nbd0p1 /mnt/
You can now access the contents of /mnt, which correspond to the first partition of the instance's disk.

To examine the secondary or ephemeral disk, use an alternate mount point if you want both primary and secondary drives mounted at the same time:

# umount /mnt
# qemu-nbd -c /dev/nbd1 `pwd`/disk.local
# mount /dev/nbd1 /mnt/
# ls -lh /mnt/
total 76K
lrwxrwxrwx.  1 root root    7 Jan 15 00:44 bin -> usr/bin
dr-xr-xr-x.  4 root root 4.0K Jan 15 01:07 boot
drwxr-xr-x.  2 root root 4.0K Jan 15 00:42 dev
drwxr-xr-x. 70 root root 4.0K Jan 15 11:31 etc
drwxr-xr-x.  3 root root 4.0K Jan 15 01:07 home
lrwxrwxrwx.  1 root root    7 Jan 15 00:44 lib -> usr/lib
lrwxrwxrwx.  1 root root    9 Jan 15 00:44 lib64 -> usr/lib64
drwx------.  2 root root  16K Jan 15 00:42 lost+found
drwxr-xr-x.  2 root root 4.0K Feb  3  2012 media
drwxr-xr-x.  2 root root 4.0K Feb  3  2012 mnt
drwxr-xr-x.  2 root root 4.0K Feb  3  2012 opt
drwxr-xr-x.  2 root root 4.0K Jan 15 00:42 proc
dr-xr-x---.  3 root root 4.0K Jan 15 21:56 root
drwxr-xr-x. 14 root root 4.0K Jan 15 01:07 run
lrwxrwxrwx.  1 root root    8 Jan 15 00:44 sbin -> usr/sbin
drwxr-xr-x.  2 root root 4.0K Feb  3  2012 srv
drwxr-xr-x.  2 root root 4.0K Jan 15 00:42 sys
drwxrwxrwt.  9 root root 4.0K Jan 15 16:29 tmp
drwxr-xr-x. 13 root root 4.0K Jan 15 00:44 usr
drwxr-xr-x. 17 root root 4.0K Jan 15 00:44 var
Once you have completed the inspection, unmount the mount point and release the qemu-nbd device:

# umount /mnt
# qemu-nbd -d /dev/nbd0
/dev/nbd0 disconnected
Resume the instance using virsh:

# virsh list
Id Name                 State
----------------------------------
1 instance-00000981    running
2 instance-000009f5    running
30 instance-0000274a    paused

# virsh resume 30
Domain 30 resumed


****************************
Libguestfs  tools in Centos7
****************************

sudo yum install libguestfs-tools      # Fedora/RHEL/CentOS
sudo apt-get install libguestfs-tools  # Debian/Ubuntu


[boris@icehouse1 Downloads]$  guestfish --rw -a disk files

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

> run
> list-filesystems
/dev/sda1: ext4
> mount /dev/sda1 /
> ls /



****************************
Guestmount tools in Centos7
****************************

[root@compute ea1aeb3xxxxxxxxxxxxxxxxx3157a9b81621]# ls
console.log  disk  disk.info  libvirt.xml
[root@compute ea1aeb3xxxxxxxxxxxxxxxxx3157a9b81621]# ls -al
total 13790864
drwxr-xr-x.  2 nova nova        3864 Mar 18 15:07 .
drwxr-xr-x. 20 nova nova        3864 Mar 19 11:01 ..
-rw-rw----.  1 root root           0 Mar 19 11:01 console.log
-rw-r--r--.  1 qemu qemu 14094106624 Mar 19 12:09 disk
-rw-r--r--.  1 nova nova          79 Mar 18 15:07 disk.info
-rw-r--r--.  1 nova nova        2603 Mar 19 10:59 libvirt.xml
[root@compute ea1aeb3xxxxxxxxxxxxxxxxx3157a9b81621]# guestmount -a disk -i /mnt
[root@compute ea1aeb3xxxxxxxxxxxxxxxxx3157a9b81621]# ll /mnt/
total 136
dr-xr-xr-x.  2 root root  4096 Mar 18 15:44 bin
dr-xr-xr-x.  4 root root  4096 Apr 16  2014 boot
drwxr-xr-x. 10 root root  4096 Mar 19 10:22 cgroup
drwxr-xr-x.  2 root root  4096 Apr 16  2014 dev
drwxr-xr-x. 80 root root  4096 Mar 19 11:00 etc
drwxr-xr-x.  3 root root  4096 Mar 18 15:08 home
dr-xr-xr-x.  8 root root  4096 Apr 16  2014 lib
dr-xr-xr-x. 11 root root 12288 Mar 18 15:44 lib64
drwx------.  2 root root 16384 Apr 16  2014 lost+found
drwxr-xr-x.  2 root root  4096 Sep 23  2011 media
drwxr-xr-x.  2 root root  4096 Sep 23  2011 mnt
drwxr-xr-x.  2 root root  4096 Sep 23  2011 opt
drwxr-xr-x.  2 root root  4096 Apr 16  2014 proc
dr-xr-x---.  4 root root 24576 Mar 19 10:59 root
dr-xr-xr-x.  2 root root 12288 Mar 18 15:45 sbin
drwxr-xr-x.  2 root root  4096 Apr 16  2014 selinux
drwxr-xr-x.  2 root root  4096 Sep 23  2011 srv
drwxr-xr-x.  2 root root  4096 Apr 16  2014 sys
drwxrwxrwt.  3 root root  4096 Mar 19 11:00 tmp
drwxr-xr-x. 13 root root  4096 Apr 16  2014 usr
drwxr-xr-x. 19 root root  4096 Mar 19 10:14 var
[root@compute ea1aeb3xxxxxxxxxxxxxxxxx3157a9b81621]# guestunmount /mnt/
[root@compute ea1aeb3xxxxxxxxxxxxxxxxx3157a9b81621]#

NovaException: Unexpected vif_type=binding_failed In Openstack Juno Migration

Sample Error
=============
ERROR nova.compute.manager [req-] [instance: ******-******-******-*******] Setting instance vm_state to ERROR
TRACE nova.compute.manager [instance: ******-******-******-*******] Traceback (most recent call last):
TRACE nova.compute.manager [instance: ******-******-******-*******]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5596, in _error_out_instance_on_exception
TRACE nova.compute.manager [instance: ******-******-******-*******]     yield
TRACE nova.compute.manager [instance: ******-******-******-*******]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3459, in resize_instance
TRACE nova.compute.manager [instance: ******-******-******-*******]     block_device_info)
TRACE nova.compute.manager [instance: ******-******-******-*******]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4980, in migrate_disk_and_power_off
TRACE nova.compute.manager [instance: ******-******-******-*******]     utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
TRACE nova.compute.manager [instance: ******-******-******-*******]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 165, in execute
TRACE nova.compute.manager [instance: ******-******-******-*******]     return processutils.execute(*cmd, **kwargs)
TRACE nova.compute.manager [instance: ******-******-******-*******]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 193, in execute
TRACE nova.compute.manager [instance: ******-******-******-*******]     cmd=' '.join(cmd))
TRACE nova.compute.manager [instance: ******-******-******-*******] ProcessExecutionError: Unexpected error while running command.
TRACE nova.compute.manager [instance: ******-******-******-*******] Command: ssh 10.5.2.20 mkdir -p /var/lib/nova/instances/******-******-******-*******
TRACE nova.compute.manager [instance: ******-******-******-*******] Exit code: 255
TRACE nova.compute.manager [instance: ******-******-******-*******] Stdout: ''
TRACE nova.compute.manager [instance: ******-******-******-*******] Stderr: 'Host key verification failed.\r\n'
TRACE nova.compute.manager [instance: ******-******-******-*******]
ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Unexpected error while running command.
Command: ssh 10.5.2.20 mkdir -p /var/lib/nova/instances/******-******-******-*******
Exit code: 255
Stdout: ''
Stderr: 'Host key verification failed.\r\n'

Things Need to be checked

Configure the nova user
First things first, let's make sure our nova user has an appropriate shell set:

cat /etc/passwd | grep nova
Verify that the last entry is /bin/bash.

If not, let's modify the user and make it so:

usermod -s /bin/bash nova


After doing this the next steps are all run as the nova user.
SSH Configuration
su - nova
We need to generate an SSH key:

ssh-keygen

Next up we need to configure SSH to not do host key verification, unless you want to manually SSH to all compute nodes that exist and accept the key (and continue to do so for each new compute node you add).

cat << EOF > ~/.ssh/config
Host *
    StrictHostKeyChecking no
    UserKnownHostsFile=/dev/null
EOF

Make Password less Authentication with all Nova user's.

Integrating Docker into Juno Nova Service as a Hypervisor

Installing Python Modules Needed for Docker
===========================================
yum install -y python-six
yum install -y python-pbr
yum install -y python-babel
yum install -y python-openbabel
yum install -y python-oslo-*
yum install -y python-docker-py

Installing Latest Version of Docker
==================================
yum install wget
wget http://cbs.centos.org/kojifiles/packages/docker/1.2.0/4.el7.centos/x86_64/docker-1.2.0-4.el7.centos.x86_64.rpm
wget http://cbs.centos.org/kojifiles/packages/docker/1.2.0/4.el7.centos/x86_64/docker-devel-1.2.0-4.el7.centos.x86_64.rpm
wget http://cbs.centos.org/kojifiles/packages/docker/1.2.0/4.el7.centos/x86_64/docker-pkg-devel-1.2.0-4.el7.centos.x86_64.rpm
yum install docker-*

Starting the Docker Service
===========================
systemctl start docker
systemctl status docker
systemctl enable docker


Installing and configuring Nova-Docker Driver
=============================================
yum install -y python-pip git
pip install -e git+https://github.com/stackforge/nova-docker#egg=novadocker
cd src/novadocker/
python setup.py install


Install and configure Neutorn Service In Docker Server
======================================================
http://www.adminz.in/2014/10/openstack-juno-part-6-neutron.html

Inatall and configure Nova Service to use Docker
======================================================
Installing Packages
yum install openstack-nova-compute -y ; usermod -G docker nova


openstack-config --set /etc/nova/nova.conf DEFAULT compute_driver novadocker.virt.docker.DockerDriver


openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend rabbit
openstack-config --set /etc/nova/nova.conf DEFAULT rabbit_host controller
openstack-config --set /etc/nova/nova.conf DEFAULT rabbit_password guest

openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone

openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_uri http://controller:5000/v2.0
openstack-config --set /etc/nova/nova.conf keystone_authtoken identity_uri http://controller:35357
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_tenant_name service
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_user nova
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_password mar4nova

#On Controller1 #Public IP on contreller server. Hostname don't work. configure the my_ip option to use the management interface IP address of the controller node
openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 10.1.15.144
openstack-config --set /etc/nova/nova.conf DEFAULT vnc_enabled True
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen 0.0.0.0
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address 10.1.15.144
openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://10.1.15.140:6080/vnc_auto.html

openstack-config --set /etc/nova/nova.conf glance host controller

[DEFAULT]
compute_driver = novadocker.virt.docker.DockerDriver

systemctl enable openstack-nova-compute.service
systemctl start openstack-nova-compute.service

Conufigure Glance to Include Docker Images
==========================================
On Controller server
# Supported values for the 'container_format' image attribute
container_formats=ami,ari,aki,bare,ovf,ova,docker

systemctl restart openstack-glance-api

Creating Custom Rootwrap Filters. On Docker Server
=================================
mkdir /etc/nova/rootwrap.d/
cat << EOF >> /etc/nova/rootwrap.d/docker.filters
# nova-rootwrap command filters for setting up network in the docker driver
# This file should be owned by (and only-writeable by) the root user
[Filters]
# nova/virt/docker/driver.py: 'ln', '-sf', '/var/run/netns/.*'
ln: CommandFilter, /bin/ln, root
EOF

chgrp nova /etc/nova/rootwrap.d -R
chmod 640 /etc/nova/rootwrap.d -R

systemctl restart openstack-nova-compute

If you face an time out issue with Nova try the fix in following URL

http://www.adminz.in/2014/11/docker-nova-time-out-error.html

On Docker Server Adding Docker Image
docker pull tutum/wordpress
docker save tutum/wordpress | glance image-create --is-public=True --container-format=docker --disk-format=raw --name tutum/wordpress

Docker + Nova Time Out Error

http://paste.openstack.org/show/131728/

Sample Error
==========
    out = f(*args, **kwds)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 468, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/novadocker/virt/docker/client.py", line 36, in wrapper
    out = f(*args, **kwds)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
 File "/usr/lib/python2.7/site-packages/novadocker/virt/docker/client.py", line 36, in wrapper
    out = f(*args, **kwds)
 File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 299, in _make_request
    timeout_obj = self._get_timeout(timeout)
 File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 279, in _get_timeout
    return Timeout.from_float(timeout)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/timeout.py", line 152, in from_float
    return Timeout(read=timeout, connect=timeout)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/timeout.py", line 95, in __init__
    self._connect = self._validate_timeout(connect, 'connect')
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/timeout.py", line 125, in _validate_timeout
    "int or float." % (name, value))
ValueError: Timeout value connect was Timeout(connect=10, read=10, total=None), but it must be an int or float.

To fix the problem i have to modify directly "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py"

def _get_timeout(self, timeout):                                            

    """ Helper that always returns a :class:`urllib3.util.Timeout` """      

    if timeout is _Default:                                                           

        return self.timeout.clone()                                         

    if isinstance(timeout, Timeout): <========================== Timeout is not a urllib3 timeout

        return timeout.clone()                                              

    else:                                                                   

        # User passed us an int/float. This is for backwards compatibility, 

        # can be removed later                                                                                                 

        return Timeout.from_float(timeout._connect ) <======================= manually entered _connect

I

Removing Nova and Neutron Services from Mysql

Some times we need to remove the services listed in the Nova or neutron as they are duplicated or they are removed from the entire system. So we can do it in the following way.

Removing Nova Service from Mysql Database. 

>>nova service-list
>>nova hypervisor-list

mysql> use nova;
mysql> SELECT id, created_at, updated_at, hypervisor_hostname FROM compute_nodes;

mysql> DELETE FROM compute_node_stats WHERE compute_node_id='1';
mysql> DELETE FROM compute_nodes WHERE hypervisor_hostname='compute1';
mysql> DELETE FROM services WHERE host='compute1';



Removing Nneutron  Service from Mysql Database. 

>>neutron agent-list

mysql> use neutorn
mysql> DELETE FROM agents WHERE host='compute1';

Docker with Openstack Giving Error "ova.openstack.common.threadgroup ValueError: Timeout value connect was Timeout"

When I try to integrate Docker to Openstack Juno, I am not able to start the nova service in the compute node. I followed https://wiki.openstack.org/wiki/Docker .

When I remove or comment out #compute_driver = novadocker.virt.docker.DockerDriver from nova configuration, the service is able to start but the pid gets killed soon.

I am getting following error while trying to start the nova service.

Complete Error.
http://paste.openstack.org/show/128805/

Sample Error
****2014-11-03 14:14:08.138 5264 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/timeout.py", line 125, in _validate_timeout
2014-11-03 14:14:08.138 5264 TRACE nova.openstack.common.threadgroup     "int or float." % (name, value))
2014-11-03 14:14:08.138 5264 TRACE nova.openstack.common.threadgroup ValueError: Timeout value connect was Timeout(connect=10, read=10, total=None), but it must be an int or float.
2014-11-03 14:14:08.138 5264 TRACE nova.openstack.common.threadgroup****


The issue has been fixed , I didn't installed the docker requirement . Once i installed it and rebooted the server its working fine now .

For testing I have used * for installation ,we just need to install the correct packages. https://github.com/stackforge/nova-do...

yum install *pbr*
yum install *six*
yum install *babel*
yum install *oslo*
yum install docker-py

Openstack Juno -Part 3 -Compute service Nova

  Creating Nova Database 
create database nova;
 GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'localhost' IDENTIFIED BY 'mar4nova';
GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%' IDENTIFIED BY 'mar4nova';
GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'10.0.0.200' IDENTIFIED BY 'mar4nova';
GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'10.0.0.202' IDENTIFIED BY 'mar4nova';
flush privileges;

Configuring User's in keystone
source admin-openrc.sh
keystone user-create --name nova --pass mar4nova --email EMAIL_ADDRESS
keystone user-role-add --user nova --tenant service --role admin
keystone service-create --name nova --type compute --description "OpenStack Compute"
keystone endpoint-create --service-id $(keystone service-list | awk '/ compute / {print $2}') --publicurl http://controller:8774/v2/%\(tenant_id\)s --internalurl http://controller:8774/v2/%\(tenant_id\)s --adminurl http://controller:8774/v2/%\(tenant_id\)s --region regionOne

#On Controller

Installing packages 
yum -y install openstack-nova-api openstack-nova-cert openstack-nova-conductor openstack-nova-console openstack-nova-novncproxy openstack-nova-scheduler python-novaclient

Configuring Service 
openstack-config --set /etc/nova/nova.conf database connection mysql://nova:mar4nova@controller/nova

openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend rabbit
openstack-config --set /etc/nova/nova.conf DEFAULT rabbit_host controller
openstack-config --set /etc/nova/nova.conf DEFAULT rabbit_password guest

#On Controller1 #Public IP on contreller server. Hostname don't work. configure the my_ip option to use the management interface IP address of the controller node
openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 10.1.15.142
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen 10.1.15.142
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address 10.1.15.142

openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_uri http://controller:5000/v2.0
openstack-config --set /etc/nova/nova.conf keystone_authtoken identity_uri http://controller:35357
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_user nova
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_tenant_name service
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_password mar4nova

openstack-config --set /etc/nova/nova.conf glance host controller

#Populate the database 

su -s /bin/sh -c "nova-manage db sync" nova

Database changed
MariaDB [nova]> show tables;
+--------------------------------------------+
| Tables_in_nova                             |
+--------------------------------------------+
| agent_builds                               |
| aggregate_hosts                            |
| aggregate_metadata                         |
| aggregates                                 |
| block_device_mapping                       |
| bw_usage_cache                             |
| cells                                      |
| certificates                               |
| compute_nodes                              |
| console_pools                              |
| consoles                                   |
| dns_domains                                |
| fixed_ips                                  |
| floating_ips                               |
| instance_actions                           |
| instance_actions_events                    |
| instance_extra                             |
| instance_faults                            |
| instance_group_member                      |
| instance_group_policy                      |
| instance_groups                            |
| instance_id_mappings                       |
| instance_info_caches                       |
| instance_metadata                          |
| instance_system_metadata                   |
| instance_type_extra_specs                  |
| instance_type_projects                     |
| instance_types                             |
| instances                                  |
| iscsi_targets                              |
| key_pairs                                  |
| migrate_version                            |
| migrations                                 |
| networks                                   |
| pci_devices                                |
| project_user_quotas                        |
| provider_fw_rules                          |
| quota_classes                              |
| quota_usages                               |
| quotas                                     |
| reservations                               |
| s3_images                                  |
| security_group_default_rules               |
| security_group_instance_association        |
| security_group_rules                       |
| security_groups                            |
| services                                   |
| shadow_agent_builds                        |
| shadow_aggregate_hosts                     |
| shadow_aggregate_metadata                  |
| shadow_aggregates                          |
| shadow_block_device_mapping                |
| shadow_bw_usage_cache                      |
| shadow_cells                               |
| shadow_certificates                        |
| shadow_compute_nodes                       |
| shadow_console_pools                       |
| shadow_consoles                            |
| shadow_dns_domains                         |
| shadow_fixed_ips                           |
| shadow_floating_ips                        |
| shadow_instance_actions                    |
| shadow_instance_actions_events             |
| shadow_instance_extra                      |
| shadow_instance_faults                     |
| shadow_instance_group_member               |
| shadow_instance_group_policy               |
| shadow_instance_groups                     |
| shadow_instance_id_mappings                |
| shadow_instance_info_caches                |
| shadow_instance_metadata                   |
| shadow_instance_system_metadata            |
| shadow_instance_type_extra_specs           |
| shadow_instance_type_projects              |
| shadow_instance_types                      |
| shadow_instances                           |
| shadow_iscsi_targets                       |
| shadow_key_pairs                           |
| shadow_migrate_version                     |
| shadow_migrations                          |
| shadow_networks                            |
| shadow_pci_devices                         |
| shadow_project_user_quotas                 |
| shadow_provider_fw_rules                   |
| shadow_quota_classes                       |
| shadow_quota_usages                        |
| shadow_quotas                              |
| shadow_reservations                        |
| shadow_s3_images                           |
| shadow_security_group_default_rules        |
| shadow_security_group_instance_association |
| shadow_security_group_rules                |
| shadow_security_groups                     |
| shadow_services                            |
| shadow_snapshot_id_mappings                |
| shadow_snapshots                           |
| shadow_task_log                            |
| shadow_virtual_interfaces                  |
| shadow_volume_id_mappings                  |
| shadow_volume_usage_cache                  |
| shadow_volumes                             |
| snapshot_id_mappings                       |
| snapshots                                  |
| task_log                                   |
| virtual_interfaces                         |
| volume_id_mappings                         |
| volume_usage_cache                         |
| volumes                                    |
+--------------------------------------------+
108 rows in set (0.00 sec)

MariaDB [nova]>

systemctl enable openstack-nova-api.service
systemctl enable openstack-nova-cert.service
systemctl enable openstack-nova-consoleauth.service
systemctl enable openstack-nova-scheduler.service
systemctl enable openstack-nova-conductor.service
systemctl enable openstack-nova-novncproxy.service
systemctl start openstack-nova-api.service
systemctl start openstack-nova-cert.service
systemctl start openstack-nova-consoleauth.service
systemctl start openstack-nova-scheduler.service
systemctl start openstack-nova-conductor.service
systemctl start openstack-nova-novncproxy.service



On compute Node

Installing Packages
yum install openstack-nova-compute -y

Configuring Service
openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend rabbit
openstack-config --set /etc/nova/nova.conf DEFAULT rabbit_host controller
openstack-config --set /etc/nova/nova.conf DEFAULT rabbit_password guest

openstack-config --set /etc/nova/nova.conf DEFAULT auth_uri = http://controller:5000/v2.0
openstack-config --set /etc/nova/nova.conf DEFAULT identity_uri = http://controller:35357
openstack-config --set /etc/nova/nova.conf DEFAULT admin_tenant_name = service
openstack-config --set /etc/nova/nova.conf DEFAULT admin_user = nova
openstack-config --set /etc/nova/nova.conf DEFAULT admin_password = mar4nova


#On Controller1 #Public IP on contreller server. Hostname don't work. configure the my_ip option to use the management interface IP address of the controller node
openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 10.1.15.144
openstack-config --set /etc/nova/nova.conf DEFAULT vnc_enabled True
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen 0.0.0.0
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address 10.1.15.144
openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://controller:6080/vnc_auto.html

openstack-config --set /etc/nova/nova.conf glance host controller

Determine whether your compute node supports hardware acceleration for virtual machines:
$ egrep -c '(vmx|svm)' /proc/cpuinfo
If this command returns a value of one or greater, your compute node supports hardware acceleration which typically requires no additional configuration.
If this command returns a value of zero, your compute node does not support hardware acceleration and you must configure libvirt to use QEMU instead of KVM.
Edit the [libvirt] section in the /etc/nova/nova.conf file as follows:
[libvirt]
...
virt_type = qemu

openstack-config --set /etc/nova/nova.conf libvirt virt_type qemu


systemctl enable libvirtd.service
systemctl start libvirtd.service
systemctl enable openstack-nova-compute.service
systemctl start openstack-nova-compute.service


#Verify operation

$ nova service-list
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host       | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| 1  | nova-conductor   | controller | internal | enabled | up    | 2014-09-16T23:54:02.000000 | -               |
| 2  | nova-consoleauth | controller | internal | enabled | up    | 2014-09-16T23:54:04.000000 | -               |
| 3  | nova-scheduler   | controller | internal | enabled | up    | 2014-09-16T23:54:07.000000 | -               |
| 4  | nova-cert        | controller | internal | enabled | up    | 2014-09-16T23:54:00.000000 | -               |
| 5  | nova-compute     | compute1   | nova     | enabled | up    | 2014-09-16T23:54:06.000000 | -               |
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+

[root@controller ~]# nova image-list
+--------------------------------------+---------------------+--------+--------+
| ID                                   | Name                | Status | Server |
+--------------------------------------+---------------------+--------+--------+
| e54cb5b2-4717-4139-8258-2a0366216b92 | cirros-0.3.3-x86_64 | ACTIVE |        |
+--------------------------------------+---------------------+--------+--------+
[root@controller ~]#

Delete a VM in an Error State in Openstack

Delete a VM in an Error State

mysql -u$MYSQL_USER -p$MYSQL_PASSWD -h$MYSQL_HOST -e 'USE nova; DELETE FROM security_group_instance_association WHERE instance_id IN (SELECT id FROM instances WHERE vm_state = "error");'
mysql -u$MYSQL_USER -p$MYSQL_PASSWD -h$MYSQL_HOST -e 'USE nova; DELETE FROM block_device_mapping WHERE instance_id IN (SELECT id FROM instances WHERE vm_state = "error");'
mysql -u$MYSQL_USER -p$MYSQL_PASSWD -h$MYSQL_HOST -e 'USE nova; DELETE FROM instance_info_caches WHERE instance_id IN (SELECT uuid FROM instances WHERE vm_state = "error");'
mysql -u$MYSQL_USER -p$MYSQL_PASSWD -h$MYSQL_HOST -e 'USE nova; UPDATE fixed_ips SET allocated = 0 WHERE instance_id IN (SELECT id FROM instances WHERE vm_state = "error");'
mysql -u$MYSQL_USER -p$MYSQL_PASSWD -h$MYSQL_HOST -e 'USE nova; DELETE FROM instances WHERE vm_state = "error";'

Resetting the Instance state once it’s in a stuck state in Openstack

Resetting the Instance state once it’s in a stuck state

$ nova list
+--------------------------------------+----------------+---------+------------------------------------------.  -------------+-------------+----------------------------------+
| ID                                                                 | Name         | Status        |   Task State  | Power State | Networks                                      |
+--------------------------------------+----------------+---------+-------------------------------------------  -------------+-------------+----------------------------------+
| ca9496e9-0bd2-4734-9cf9-eb4e264628f7 | www            | SHUTOFF | powering-on | Shutdown    | fsf-lan=10.0.3.18, 93.20.168.177  |
+--------------------------------------+----------------+---------+----------------------------------------------------------+-------------+----------------------------------+

   
Setting the fields for the instance directly in the database will allow operations on the instance (nova start or nova volume-detach for instance):

$ mysql -e "update instances set task_state = NULL, \
           vm_state = 'stopped', \
           power_state = 4 \
           where deleted = 0 and hostname = 'www' and \
           uuid = 'ca9496e9-0bd2-4734-9cf9-eb4e264628f7'" nova

Nova issue with deleting Vm

We can delete the nova instance which are in error/deleting state by following the steps mentioned below.

Get the instance ID from command “nova list ” and reset the state using “nova reset-state <ID>” command

nova list

[root@controller1 ~]# nova list
+--------------------------------------+-----------------+--------+------------+-------------+----------------------------------------------+
| ID                                   | Name            | Status | Task State | Power State | Networks                                     |
+--------------------------------------+-----------------+--------+------------+-------------+----------------------------------------------+
| 0ccfa148-97de-4be9-b85d-4283037746b1 | Ha-Porxy-F5     | ACTIVE | -          | Running     | Trusted-Internal-NIC=10.0.0.157, 10.1.15.140 |
| e61a759d-528c-423b-bb24-dcf7e3a5618e | Ha-Porxy-Mysql  | ACTIVE | -          | Running     | Trusted-Internal-NIC=10.0.0.158, 10.1.15.139 |

nova reset-state 0ccfa148-97de-4be9-b85d-4283037746b1


You can also use the --active parameter to force the instance back to an active state instead of an error state. For example:

$ nova reset-state --active c6bbbf26-b40a-47e7-8d5c-eb17bf65c485

"Host key verification failed" error while resizing instance Openstack

Setting allow_resize_to_same_host=true will enable the resizing in All in one model of installation . but if we have mutiple controller and compute node's we will be getting an authentication error as following

"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4907,
in migrate_disk_and_power_off\n utils.execute(\'ssh\', dest, \'mkdir\', \'-p\', inst_base)\n', '
File "/usr/lib/python2.6/site-packages/nova/utils.py", line 165, in execute\n return processutils.execute(*cmd, **kwargs)\n', '
File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 193, in execute\n cmd=\' \'.join(cmd))\n',
"ProcessExecutionError: Unexpected error while running command.\nCommand: ssh 10.1.15.44
mkdir -p /var/lib/nova/instances/5d5ced81-6fb1-4028-97cd-686e450d1bab\nExit code: 255\nStdout: ''\nStderr:
'Host key verification failed.\\r\\n'\n"]

This is due to a bug in nova as told in below bug report which can be solved by enabling password less authentication between the compute and controller node's nova user .

https://bugzilla.redhat.com/show_bug.cgi?id=975014#c3

By default the nova user does not have a shell so we need to enable the shell and do steps to enable password less authentication between all the server .

Enable shell for nova user

In all the controller and compute server’s enable nova user.

sed -i "s/\/var\/lib\/nova:\/sbin\/nologin/\/var\/lib\/nova:\/bin\/bash/g" /etc/passwd

cat /etc/passwd |grep nova

nova:x:162:162:OpenStack Nova Daemons:/var/lib/nova:/bin/bash

Enable password less authentication

Between all the compute and controller nodes create password less authenticatioin for user nova.

 

su - nova

ssh-keygen

ssh-copy-id

Openstack Live Migration failure: operation failed

Error in the error log nova/compute

2014-06-25 01:32:50.752 2703 ERROR nova.virt.libvirt.driver [-]
[instance: fc118bff-77a3-4300-ab27-371a314b819f] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://compute2/system

Try updating the libvirt configurations. Modify /etc/libvirt/libvirtd.conf. To see all of the available options

1. 
   

   before : #listen_tls = 0
   after : listen_tls = 0
   
   before : #listen_tcp = 1
   after : listen_tcp = 1
   
   add: auth_tcp = "none"

   
   

Openstack+ Shared Storage(NFS) +Permission denied

While setting up Openstack with shared storage , If we get following error while creating a instance ensure that the server is in permissive selinux mode with

getenfonce

and if that too doesn't work try giving 755 permission to the mounted directory here its /var/lib/nova

2014-06-24 18:58:20.642 5119 TRACE nova.compute.manager [instance: a7996f1f-9af2-4410-8351-139d43f00786] libvirtError: internal error Process exited while reading console log output: qemu-kvm: -chardev file,id=charserial0,path=/var/lib/nova/instances/a7996f1f-9af2-4410-8351-139d43f00786/console.log: Could not open '/var/lib/nova/instances/a7996f1f-9af2-4410-8351-139d43f00786/console.log': Permission denied

 

At last if nothing else worked , tell libvirtd/qemu to use root user to access datas.
[root@compute nova]# cat /etc/libvirt/qemu.conf |grep root
user = "root"
#group = "root"
[root@compute nova]#

Openstack Icehouse - VNC console not connecting to server

Make sure that the setting in the controller and compute node are correct and also double check the IP's.  And replace the host-name with the IP.

controller - 192.168.216.130

running:
nova-consoleauth
nova-novncproxy

nova.conf:
novncproxy_host=0.0.0.0
novncproxy_port=6080
novncproxy_base_url=http://192.168.216.130:6080/vnc_auto.html

compute - 192.168.216.140

running:
nova-compute

nova.conf:
vnc_enabled=True
novncproxy_base_url=http://192.168.216.130:6080/vnc_auto.html
vncserver_listen=0.0.0.0
vncserver_proxyclient_address=192.168.216.140

Openstack Icehouse Part 3 NOVA

COMPUTE SERVER CONFIGURATION On Controller

yum -y install openstack-nova-api openstack-nova-cert openstack-nova-conductor openstack-nova-console openstack-nova-novncproxy openstack-nova-scheduler python-novaclient

openstack-config --set /etc/nova/nova.conf database connection mysql://nova:nova4mar@controller/nova
openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend qpid
openstack-config --set /etc/nova/nova.conf DEFAULT qpid_hostname controller

openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 192.168.216.130
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen 192.168.216.130
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address 192.168.216.130

mysql -u root -p
CREATE DATABASE nova;
GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'localhost' IDENTIFIED BY 'nova4mar';
GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%' IDENTIFIED BY 'nova4mar';
exit

#Create the Compute service tables:

su -s /bin/sh -c "nova-manage db sync" nova



keystone user-create --name=nova --pass=nova4mar --email=nova@example.com
keystone user-role-add --user=nova --tenant=service --role=admin

openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_uri http://controller:5000
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_host controller
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_protocol http
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_port 35357
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_user nova
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_tenant_name service
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_password nova4mar
keystone service-create --name=nova --type=compute --description="OpenStack Compute"
keystone endpoint-create --service-id=$(keystone service-list | awk '/ compute / {print $2}') --publicurl=http://controller:8774/v2/%\(tenant_id\)s --internalurl=http://controller:8774/v2/%\(tenant_id\)s --adminurl=http://controller:8774/v2/%\(tenant_id\)s

service openstack-nova-api start
service openstack-nova-cert start
service openstack-nova-consoleauth start
service openstack-nova-scheduler start
service openstack-nova-conductor start
service openstack-nova-novncproxy start
chkconfig openstack-nova-api on
chkconfig openstack-nova-cert on
chkconfig openstack-nova-consoleauth on
chkconfig openstack-nova-scheduler on
chkconfig openstack-nova-conductor on
chkconfig openstack-nova-novncproxy on

nova image-list



Add a rule to the default Nova Security Group to allow SSH access and Ping to instances:
nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0

NOVA ON COMPUTE NODE

Install the Compute packages:

yum -y install openstack-nova-compute

Edit the /etc/nova/nova.conf configuration file:

openstack-config --set /etc/nova/nova.conf database connection mysql://nova:NOVA_DBPASS@controller/nova
openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_uri http://controller:5000
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_host controller
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_protocol http
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_port 35357
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_user nova
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_tenant_name service
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_password nova4mar

openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend qpid
openstack-config --set /etc/nova/nova.conf DEFAULT qpid_hostname controller

#Configure Compute to provide remote console access to instances.

openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 192.168.216.140
openstack-config --set /etc/nova/nova.conf DEFAULT vnc_enabled True
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen 0.0.0.0
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address 192.168.216.140
openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://controller:6080/vnc_auto.html

#Specify the host that runs the Image Service.

openstack-config --set /etc/nova/nova.conf DEFAULT glance_host controller

#You must determine whether your system's processor and/or hypervisor support hardware acceleration for virtual machines.

Run the following command:
$ egrep -c '(vmx|svm)' /proc/cpuinfo
If this command returns a value of one or greater, your system supports hardware acceleration which typically requires no additional configuration.
If this command returns a value of zero, your system does not support hardware acceleration and you must configure libvirt to use QEMU instead of KVM.
Run the following command:
# openstack-config --set /etc/nova/nova.conf libvirt virt_type qemu
Start the Compute service and configure it to start when the system boots:

service libvirtd start
service messagebus start
chkconfig libvirtd on
chkconfig messagebus on
service openstack-nova-compute start
chkconfig openstack-nova-compute on