2 architectural at cloudstack developer day
Post on 19-Oct-2014
2.378 views
DESCRIPTION
2 architectural at CloudStack Developer Day By Alex Huang Architect, Cloud Platforms Group, Citrix Systems Inc.TRANSCRIPT
Apache CloudStack
Architecture
Alex Huang
Software Architect, Citrix Systems
Compute
Self-Service Access via CloudStack UI
Network
Storage
Admin
Users
Org A
Admin
Users
Org B
Users
End User
Admin
Deployment Architecture
• Hosts • Servers onto which services will be provisioned
• Primary Storage • VM disk storage
• Cluster • A grouping of hosts and their associated storage
• Pod • Collection of clusters in the same failure boundary
• Network • Logical network associated with service offerings
• Secondary Storage • Template, snapshot and ISO storage
• Zone • Collection of pods, network offerings and
secondary storage
• Management Server Farm • Management and provisioning tasks
Components
Zone
CloudStack Pod
Cluster
Host
Host
Network
Primary Storage
VM
VM
CloudStack Pod
Cluster Secondary
Storage
Pod 1
Host 2
Cluster 1
Host 1
Primary Storage
L3 switch
Secondary Storage
L2 switch
Two Types of Storage
• Stores disk volumes for VMs in a cluster • Configured at Cluster-level. • Close to hosts for better performance • Cluster have at least one primary storage • Requires high IOPs (can be expensive)
Primary Storage
• Stores all Templates, ISOs and Snapshots • Configured at Zone-level • Zone can have one or more secondary
storages • High capacity, low cost commodity
storage
Secondary Storage
Pod 1
….
Cluster N
L2
Host 2
Cluster 1
Deployment Architecture
Host 1
Hypervisor is the basic unit of scale.
Cluster consists of one ore more hosts of same hypervisor
All hosts in cluster have access to shared (primary) storage
Pod is one or more clusters, usually with L2 switches.
Availability Zone has one or more pods, has access to secondary storage.
One or more zones represent cloud
Primary Storage
Zone 1
….
L3
Secondary Storage
Pod N
Management Server Cluster
Internet
Management Server Cluster
Replica
Infrastructure Resources
User API
Admin API
Load Balancer
Management Server
Management Server
MySQL
MS is stateless. MS can be deployed as physical server or VM
Single MS node can manage up to 10K hosts. Multiple nodes can be deployed for scale or redundancy
RHEL 5.4+, Ubuntu 10.0.4, Fedora 16
Replication
Managing Complexity
The Three C’s of Complexity
• Control
• Choice
• Compliance
Compute
Giving Control Brings Complexity
Network
Storage
Admin
Users
Org A
Admin
Users
Org B
Users
End User
Admin
VM Ware
Xen Server
KVM
NFS
iSCSI
FC
Net Scaler
F5
Jun. SRX
Local Disk
Cisco ASA
Swift
HDFS
Hyper-V
• ACL • Limits • Governance
Oracle VM
Bare Metal
Guest Virtual Layer-2 Network
Guest 1 VM 1
Guest 1 VM 2
Guest 1 VM 3
Guest 1 Virtual Network 10.1.1.0/24
Gateway 10.1.1.1
Guest 10.1.1.2
Guest 10.1.1.3
Guest 10.1.1.4
Guest 1 Virtual Router
Guest 2 VM 1
Guest 2 VM 2
Guest 2 VM 3
Guest 2 Virtual Network 10.1.1.0/24
Gateway 10.1.1.1
Guest 10.1.1.2
Guest 10.1.1.3
Guest 10.1.1.4
Guest 2 Virtual Router
Public IP 65.37.141.24 65.37.141.80
Public IP 65.37.141.11 65.37.141.36
Internet
Multi-tier Network
Private IP 10.1.1.112
DHCP, DNS User-data
Public IP 65.37.141.112
10.1.1.1
Web VM 1
10.1.1.3
Web VM 2
10.1.1.4
Web VM 3
10.1.1.5
Web VM 4
Netscaler Load
Balancer
Private IP 10.1.1.111
Public IP 65.37.141.111
Juniper SRX
Firewall
Virtual Router
Virtual Network 10.1.1.0/24 VLAN 100
Virtual Network 10.1.2.0/24 VLAN 1001
10.1.2.21
10.1.2.18
10.1.2.38
10.1.2.39
10.1.2.31
App VM 1 10.1.3.21
Virtual Network 10.1.3.0/24 VLAN 141
10.1.2.24
App VM 2 10.1.3.45
10.1.3.24
DB VM 1
DHCP, DNS, User-data
DHCP, DNS User-data, Source-NAT, VPN
Public IP 65.37.141.115
Virtual Router
Virtual Router
Unified Multi-tier Network
10.1.1.1
Web VM 1
10.1.1.3
Web VM 2
10.1.1.4
Web VM 3
10.1.1.5
Web VM 4
Virtual Network 10.1.1.0/24 VLAN 100
Virtual Network 10.1.2.0/24 VLAN 1001
10.1.2.31
App VM 1
Virtual Network 10.1.3.0/24 VLAN 141
10.1.2.24
App VM 2
10.1.3.24
DB VM 1
Virtual Router
Customer Premises
IPSec or SSL site-to-site VPN
Internet
Monitoring VLAN
Virtual Router Services • IPAM • DNS • LB [intra] • S-2-S VPN • Static Routes • ACLs • NAT, PF • FW [ingress & egress] • BGP
Load Balancer
Other Topologies
Guest Virtual Network 10.1.1.0/24 VLAN 100
Gateway address 10.1.1.1
10.1.1.1
Guest VM 1
10.1.1.3
Guest VM 2
10.1.1.4
Guest VM 3
10.1.1.5
Guest VM 4
Guest Virtual Network 10.1.1.0/24 VLAN 100
DHCP, DNS User-data
10.1.1.1
Guest VM 1
10.1.1.3
Guest VM 2
10.1.1.4
Guest VM 3
10.1.1.5
Guest VM 4
No services [Static IPs] Dedicated VLAN with DHCP and DNS User can request specific IP[s] for NIC
Core switch
Gateway address 10.1.1.1
Core switch
Virtual Router
Other Topologies
Guest Virtual Network 10.1.1.0/24 VLAN 100
Gateway address 10.1.1.1
10.1.1.100
Guest VM 1
10.1.1.200
Guest VM 2
10.1.1.101
Guest VM 3
10.1.1.115
Guest VM 4
Guest Virtual Network 10.1.1.0/24 VLAN 100
DHCP, DNS User-data
10.1.1.1
Guest VM 1
10.1.1.3
Guest VM 2
10.1.1.4
Guest VM 3
10.1.1.5
Guest VM 4
MPLS Use Case Shared VLAN with DHCP and DNS
CS Virtual Router
Core switch
Gateway address 10.1.1.1
Core switch
MPLS VLAN 100
DHCP, DNS User-data
CS Virtual Router
…
DB Security Group
Web Security Group
Layer 3 Networking (Amazon Style)
… …
Web VM
Web VM
Web VM
Web VM
DB VM
Web VM
DB VM
Web VM
Software Architecture
Management Server
Orchestration Engine - Drives long running VM
operations - Syncs between resources
managed and DB - Generates events
Resource Management
Cluster Management
Job Management
DB
UI Cloud Portal
CLI Other Clients
Deployment Planning
Network Gurus
Network Elements
Hypervisor Gurus
Database Access
Alert & Event Management
Plu
gin
AP
I
Resource API
Hypervisor Resources
Network Resources
Storage Resources
Image Resources
Snapshot Resources
REST API
OAM&P API End User API EC2 API Pluggable Service API Engine Other APIs
Security Adapters
Account Management Connectors
ACL & Authentication - Accounts, Domains, and Projects - ACL, limits checking
Services API
Serv
ices
AP
I
Console Proxy Management
Template Access
HA
Usage Calculations
Additional Services
Event Bus
Message Bus Usage Server
Orchestration Engine
• Understands how to orchestrate long running processes (i.e. VM starts, Snapshot copies, Template propagation)
• Well defined process steps
• Calls Plugin API to execute functionalities that it needs
Plugins
• Various ways to add more capability to CloudStack
• Implements clearly defined interfaces
• All operations must be idempotent
• All calls are at transaction boundaries
• Compiles only against the Plugin API module
Anatomy of a Plugin
ServerResource - Optional. Required if Plugin needs to be co-
located with the resource - Implements translation layer to talk to
resource - Communicates with server component via
JSON
Rest API - Optional. Required only if needs to expose
configuration API to admin.
Plu
gin
AP
I
Data Access Layer
Implementation
• Can be two jars: server component to be deployed on management server and an optional ServerResource component to be deployed co-located with the resource
• Server component can implement multiple Plugin APIs to add its feature
• Can expose its own API through Pluggable Service so administrators can configure the plugin
• As an example, OVS plugin actually implements both NetworkGuru and NetworkElement
Plugin Interfaces Available
• NetworkGuru – Implements various network isolation and ip address technologies
• NetworkElement – Facilitate network services on network elements to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)
• DeploymentPlanner – Different algorithms to place a VM and volumes.
• Investigator – Ways to find out if a host is down or VM is down. • Fencer – Ways to fence off a VM if the state is unknown • UserAuthenticator – Methods of authenticating a user • SecurityChecker – ACL access • HostAllocator – Provides different ways to allocate host • StoragePoolAllocator – Provides different ways to allocate volumes
Separating Data and Control
Data Center 1
Cloud
Data Center 2
Data Center 3
Management
Server
Management Servers control all resources, both virtual and physical
SSVMs deployed to transfer data between zones
CPVMs deployed to transfer VNC console traffic
VR deployed for traffic into public internet
Management Server is never in the data path
SSVM
SSVM
SSVM Transfer of Templates,
ISOs, Snapshots
CPVM CPVM
CPVM
VR
VR
VR
Internet
Kernel
Sequence Flow for VM Creation
End User Rest API
Security Checkers
User VM Mgr
Network Mgr
Storage Mgr
Job Scheduling
VirtualMachine Mgr
Network Guru
Deploy VM
ACL Checks
Allocate Entity in CS
Allocate VM
Allocate NIC
Allocate Volume
Allocate IP
Schedules Deploy Job
Returns with job id, VM id
Query Job Result
Returns with job status
Sequence Flow for VM Creation
Job Threads Network Element
User VM Mgr
Network Mgr
Storage Mgr
VirtualMachine Mgr
Network Guru
Start VM
Start VM
Prepare Nics
Notify that Nic is about to be started in network
Reserve resources for Nic
Services API Server
Resources
Start User VM
Agent Calls
Prepare Volumes
Template Mgr
Deployment
Planner
Get a Deployment Plan (Host and StoragePool)
Prepare template on Primary Storage
Agent Calls
Agent Start VM Call
Stores job result
Conclusion
Design Goals for CloudStack
• Design for complexity
– Clear interfaces
• Design for scalability
– Separate out data path and control paths
– Design to maximize the use of database connections
• Design against failure
– Provide clear boundaries (process and compilation)
– Utilize cloud administrator to give guidance
More Information
28
• http://cloudstack.org
• Apache mailing lists
• Thank you