Alex Huang

Architect, Cloud Platforms Group, Citrix Systems Inc.

Apache CloudStack

Co-founded VMOps (Cloud.Com) Software architect for CloudStack. Responsible for CloudStack’s overall architecture, performance,

and scalability. Currently a committer and PPMC member on Apache for

CloudStack. MS from Stanford and BS in Computer Science from Berkeley.

Apache CloudStack (Incubating) An Introduction

Alex Huang

Apache CloudStack Committer

• Create VMs, disks

networks, network services

• Self service

• Meter usage

Use CloudStack to build IaaS clouds (like EC2)

• Java based

• Scalable

• Many vendor integrations

• Native and EC2 API

How did Amazon build EC2?

Commodity

Servers

Commodity

Storage Networking

Open Source Xen Hypervisor

Amazon Orchestration Software

AWS API (EC2, S3, …)

Amazon eCommerce Platform

How can you build your cloud?

Servers Storage Networking

Open Source Xen Hypervisor

Amazon Orchestration Software

AWS API (EC2, S3, …)

Amazon eCommerce Platform

ESXi, KVM, XenServer/XCP, OVM

CloudStack Orchestration Software

Your Portal (Optional)

CloudStack or AWS API

Project history

• 2008/2009: closed-source development

• First deployments in late 2009

• May 2010: ~98% open source as GPLv3 (open core)

• August 2011: 100% open source GPLv3

• April 2012: Switch to Apache License v2

• Submit code to Apache Software Foundation

Project current state

• In incubation within Apache Software Foundation

• 4.0 released!

• Bugs and wiki moved to ASF infra

• Mailing list traffic moved to ASF infra

• Many non-Citrix contributors, committers, and PPMC

members

Yes, the ASF is great

Enter ASF

IaaS Cloud Concepts

Cloud

Built for traditional enterprise

apps & client-server compute • Scale-up (pool-based resourcing)

• IT management-centric

• 1 administrator for 100’s of servers

• Proprietary vendor stack

Designed around big data,

massive scale & next-gen apps • Scale-out (horizontal resourcing)

• Autonomic management

• 1 administrator for 1,000’s of servers

• Open, value-added stack

Virtualization alone does not make a cloud

Server Virtualization

Design for failure

Self-service recovery

Multi-site redundancy

Ephemeral resources

Cloud Workload

Think Amazon Web Services

Expect reliability

Back-up everything

HA, Fault tolerance

Admin control recovery

Traditional Workload

Think Server Virtualization

Clouds must reliably run all types of workloads

Object Storage

vSphere

ESXi Cluster

ESXi Cluster

ESXi Cluster

Enterprise Networking (e.g., VLAN)

Enterprise Storage (e.g., SAN)

Cloud-era Availability

Zone

Cloud-era Availability

Zone

Cloud-era Availability

Zone

Traditional Zone CloudStack Mgmt

Server

Cloud-era Workloads Traditional Workloads

Embrace traditional and extend to Cloud-era

Cloud-era Availability

Zone

Cloud-era Availability

Zone

Traditional Availability

Zone

Apache CloudStack Management Server

Traditional Availability

Zone

Traditional Availability

Zone

Deployment and Software Architecture

Zone1

Data Center 1

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 3

Zone 4

Management Servers

• Single Management Server can manage multiple zones

• Zones can be geographically distributed but low latency links are expected for better performance

• Single MS node can manage up to 10K hosts.

• Multiple MS nodes can be deployed as cluster for scale or redundancy

Router

L3 Core Switch

Top of Rack Switch

… … … … … Availability Zone 1

Servers

Primary Mgmt Server Cluster

Object Store

Pod 1 Pod 2 Pod 3 Pod N

Primary MySQL

Load Balancer

Admin Internet

Availability Zone 2

Backup MySQL

Standby Mgmt Server Cluster Cloud-era zone deployment

10Gbps

Storage

& Mgmt

1Gbps

Guest

10Gbps

Storage

& Mgmt

1Gbps

Guest

10Gbps

Storage

& Mgmt

1Gbps

Guest

…

Load Balancer

Core Switch

Aggregation

Switch

TOR Switch

Compute

Nodes

NFS

Primary

Storage

Object

Store

Pod 1

Pod 2

Pod 200

Internet

Traditional zone

deployment

Management Server

XenServer

ESX

vCenter

KVM

Agent

OVM

Agent

XAPI HTTP

• XS 5.6, 5.6FP1, 5.6 SP2, 6.0.2, XCP 1.1

• Incremental Snapshots • VHD • NFS, iSCSI, FC & Local disk • Storage over-

provisioning: NFS

• ESX 4.1, 5.0

• Full Snapshots • VMDK • NFS, iSCSI, FC & Local disk • Storage over-provisioning:

NFS, iSCSI

• RHEL 6.0, 6.1, 6.2, Ubuntu 12.04

• Full Snapshots (not live) • QCOW2 • NFS, iSCSI & FC • Storage over-

provisioning: NFS

• OVM 2.2 • No Snapshots • RAW • NFS & iSCSi • No storage over-

provisioning

XCP

Mgm

t S

erv

er

CP

U U

til.

Se

co

nd

s to

de

plo

y

25,000 …. to …. 30,000 VMs 0 …. to …. 30,000 VMs

• Simulator developed to test massive scale

• Four Management Servers can manage 30,000 hosts

• Scale to hundreds of thousands of hosts possible with

multiple management server clusters (regions)

Features

Compute

XCP/XS VMware KVM Oracle VM Bare metal

Hypervisor

Storage

Local Disk iSCSI NFS Fiber

Channel Object Stores

Block & Object

Network

Network Type

Isolation Load

balancer Firewall VPN

Network & Network Services

Users

Start

Stop

Restart

Destroy

VM Operations Console Access

• CPU Utilized

• Network Read

• Network Writes

VM Status Change

Service Offering

2 CPUs 1 GB RAM 20 GB 20 Mbps

4 CPUs 4 GB RAM 200 GB 100 Mbps

Volume

VM 1 Add / Delete Volumes

Schedule

Snapshots

Hourly Daily

Weekly Monthly

Now

Create Templates

from Volumes

Volume

Template

View Snapshot

History 12/2/2012 7.30 am

…. 2/2/2012 7.30 am

CPU Cores

CPU (MHz)

Memory (MB)

Name

Compute

Specify Resource Levels

Custom Disk Size

Disk Size (GB)

Storage Tag

Storage Tag

Public

Name

Disk

Network Rate

Redundant VR

Public

Name

Network

Firewall

Load balancer

CPU Cap

Host Tag

Enable HA

Configure Properties

Public

Define Scope

• Domain is a unit of isolation that represents a customer org, business unit or a reseller

• Domain can have arbitrary levels of sub-domains

• A Domain can have one or more accounts

• An Account represents one or more users and is the basic unit of isolation

• Admin can limit resources at the Account or Domain levels

Admin

Org A

Admin

Reseller A

Domain

Domain

Admin

Org C

Sub-Domain

User 1

User 2

Group B

Account

Group A

Account

VMs, IPs, Snapshots…

VMs, IPs, Snapshots…

Resources

Resources

• Create Networks and attach

VMs

• Acquire public IP address for

NAT & load balancing

• Control traffic to VM using

ingress and egress firewall

rules

• Set up rules to load balance

traffic between VMs

Zone N

• Provides cloud operator

defined service features • Isolation

• Load Balancing

• VPN

• Firewall

• Supports Physical Devices • NetScaler

• F5 BIG-IP

• Juniper SRX

Network offering

Pod N Pod 1

Zone 1

Pod 1

Public Network 65.11.0.0/16

65.11.1.2

Guest VM 1

Guest VM 2

Guest VM 3

Guest VM 4

Public Network/Internet

Physical Load

Balancer

Network Services Managed Externally Network Services Managed by CS

65.11.1.3

65.11.1.4

65.11.1.5

DHCP, DNS

CS Virtual Router

Security Group 1

Security Group 2

65.11.1.2

Guest VM 1

Guest VM 2

Guest VM 3

Guest VM 4

65.11.1.3

65.11.1.4

65.11.1.5

DHCP, DNS

CS Virtual Router

Security Group 1

Security Group 2

EIP, ELB

Public Network/Internet

Guest Virtual Network 10.0.0.0/8 VLAN 100

Gateway address 10.1.1.1

DHCP, DNS NAT Load Balancing VPN

6.37..1.11

10.1.1.1

Guest VM 1

10.1.1.3

Guest VM 2

10.1.1.4

Guest VM 3

10.1.1.5

Guest VM 4

CS Virtual Router

Public Network/Internet

Guest Virtual Network 10.0.0.0/8 VLAN 100

Private IP 10.1.1.112

DHCP, DNS

Public IP 6.37.1.11

10.1.1.1

Guest VM 1

10.1.1.3

Guest VM 2

10.1.1.4

Guest VM 3

10.1.1.5

Guest VM 4

Physical Load

Balancer

Private IP 10.1.1.111

Public IP 6.37.1.12

Juniper SRX

Firewall

CS Virtual Router provides Network Services External Devices provide Network Services

CS Virtual Router

Layer-2 Layer-3 Isolation VLAN/SDN Security Groups

Performance Better Better

Network setup Moderate Easy

Support broadcast Yes No

Scalability Good Best

Interoperability with physical servers

Good Poor

Pod 1

Host 2

Cluster 1

Host 1

Primary Storage

L3 switch

Secondary

Storage

L2 switch

CloudStack storage

• Configured at Cluster-level. Close to hosts

for better performance

• Stores all disk volumes for VMs in a cluster

• Cluster can have one or more primary

storages

• Local disk, iSCSI, FC or NFS

Primary Storage

• Configured at Zone-level

• Stores all Templates, ISOs and Snapshots

• Zone can have one or more secondary

storages

• NFS, OpenStack Swift, others coming

Secondary Storage