ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

Kelcey Damage | 11/24/2013

Foreword

We are going to take a quick look at the networking schema changes introduced in 4.2 for KVM agents, as well as the upgrade process existing users.

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

Agenda

1.) KVM agent networking schema changes

2.) Known upgrade issues & the common upgrade method

3.) Troubleshooting and work-arounds

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

KVM networking schema changes

There are two key areas of interest in regards to the changes made to networking with KVM and Cloudstack

Schema Changes

Impact On Production

The KVM Agent

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

Bridge SchemaACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

bridge name bridge id STP enabled interfaces

brbondg-1205 8000.0023aef93c7c no bondg.1205 vnet7brbondg-1213 8000.0023aef93c7c no bondg.1213

vnet12cloudVirBr1205 8000.0023aef93c7c no vnet5cloudVirBr1213 8000.0023aef93c7c no vnet9

cloud0 8000.fe00a9fe014f no vnet0vnet4

vnet8cloudbrguest 8000.0023aef93c7c yes bondgcloudbrman 8000.0026b9fdb774 yes bondm vnet1 vnet3cloudbrpub 8000.0023aef93c80 yes bondp vnet2 vnet6virbr0 8000.525400368385 yes virbr0-nic

KVM Agent Schema as of 4.1.0+

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

InterfacesWhat’s Changed:

Interfaces now require names shorter them 8 characters.

Why:

Because the new distributed port-group names are concatenations of the VLAN ID and the interface name.

“br{interface_name}-{VLAN#}”

What this replaces:

“cloudVirBr{VLAN#}”

Important note on interfaces

• Longer names will get truncated.

• Routing will break and traffic may be shared within VLANs.

Example:

• Naming a bond as ‘bondguest’ with a VLAN of 1204 will create the port-group ‘brbondguest-120’. VLANs 1200 – 1209 will all get assigned to this port-group

Effect on production systems

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

If using any name other then the default:

“eth/em/p1p”…

… such as bonds or aliases then the bond or alias name ..//

//.. must be shorter then 8 characters.

Upgrading to 4.2+ and beyond

There are two key areas of interest in the upgrade process for anyone moving to the latest version.

Documentation

The New System VM

The Upgrade Process

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

In the official release notes, the steps for upgrading from 4.0 forward to 4.2+ were missing the section on how and where to download the latest templates.

We now use a virtual-IO socket for communication with the system VM, instead of the prior ‘patchdisk’ method of passing cmdline and authorized_keys.

Documentation The New System VM1 2

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

Major changes in the templates require updating them to the

latest version prior to upgrading the CloudStack.

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

KVM

Name: systemvm-kvm-4.2 Description: systemvm-kvm-4.2 URL: http://download.cloud.com/templates/4.2/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2 Zone: Choose the zone where this hypervisor is used Hypervisor: KVM Format: QCOW2 OS Type: Debian GNU/Linux 7.0 (32-bit) (or the highest Debian release number available in the dropdown) Extractable: no Password Enabled: no Public: no Featured: no

Oh No! I Broke My Cloud

If you’ve gone ahead and upgraded your CloudStack version before upgrading your templates, there are ways to fix your cloud.

The Database Method

The Java Method

The Fix

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

The Database Method

o Step 1): Mount your secondary storage to your management server

mount -t nfs {ip_of_storage_server}:[path_to_secondary_storage] /mnt

o Step 2): Download the latest version of the templates:

/usr/share/cloudstack-common/scripts/storage/secondary/cloud- install-sys- tmplt -m /mnt -u http://download.cloud.com/templates/4.2/systemvmtemplate- 2013-06-12-master-kvm.qcow2.bz2 -h kvm -F

o Step 3): Find the name of the old template in the database:

USE cloud; SELECT install_path FROM template_host_ref WHERE template_id=3;

o Step 4): write down the name of the .qcow2 file that your given in the previous step.

o Step 5): from the management server locate the new template on the mounted secondary storage:

cd /mnt/template/tmpl/1/3/

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

The Database Method

o Step 6): rename the .qcow2 file in that folder to the name we copied from the database.

o Step 7): edit the template.properties file in the same folder and change both instances of the old name to the new one.

o Step 8): we need to reset the cached template in the database:

UPDATE template_spool_ref SET download_pct='0',download_state='NOT_DOWNLOADED',state='NULL',local_path='NULL‘,install_path='NULL',template_size='0' WHERE template_id='3‘;

o Step 9): Unmount your secondary storage from the management server:

umount /mnt

o Step 10): disable the zone from the management UI.

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

The Database Method

o Step 11): update the database records for your system VMs to be ‘Stopped’. You will need to do this for both the Secondary Storage VM and the Console Proxy. The ID of the system VM is the number in it’s name, for example; s-34-VM,, would have an ID of ’34 .′

UPDATE vm_instance SET state='Stopped' where id='{id_of_system_vm}';

o Step 12): From the management UI, destroy both the system VMs.

o Step 13): Once both system VMs have been destroyed, re-enable the zone.

o Step 14): Tail the management log and watch for the VMs to start.

tail -f /var/log/cloudstack/management/management-server.log

o Step 15): Enjoy your 4.2+ Cloud!

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

IYSE Method *(If Your Smart Enough)

For those in the USR tract that understand gobble-de-gook below is the JAVA method that some prefer.

(1) register a new template (such as systemvmtemplate-4.2) on UI or in Upgrade*.java file

(2) Change template type from User to System in Upgrade*.java file.

(3) cloud-install-sys-tmplt download the system-vm template

(4) cloudstack-sysvmadm re-deploy system-vms

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading

Referencehttps://issues.apache.org/jira/browse/CLOUDSTACK-4826

http://cloud.kelceydamage.com/cloudfire/blog/2013/10/08/conquering-the-cloudstack-4-2-dragon-kvm/

Get Involved

Web: http://cloudstack.apache.org/

Mailing Lists: cloudstack.apache.org/mailing-lists.html

IRC:  irc.freenode.net: 6667 #cloudstack

Twitter:  @cloudstack

LinkedIn: www.linkedin.com/groups/CloudStack-Users-Group-3144859

If it didn’t happen on the mailing list, it didn’t happen.

ACS 4.1.0+ | KVM Agent Networking Schema And Upgrading