lessions from building a high available cloud foudry on top of open stack
TRANSCRIPT
Lessions from Building a High Available Cloud
Foudry on top of OpenStack
1
Hello World!Jiang Yi Tao
IBM CDL Bluemix IaaS PaaS
2
Agenda• Open Stack
• Nova Neutron Cinder Swift
• Cloud Foundry
• LB Gorouter
• Database
• Backup
3
Releases
• CF 233
• OpenStack Kilo
4
OpenStack
5
OpenStack Intergration
6
• Image
• Router
• network, subnet
• Flavor
• Keypair
• Security Group
OpenStack Create VM
7
How to Deploy CF1.Target a bosh director using cli
2.Upload a stemcell
3.Upload a release
4.Gererate a manifest
5.bosh deploy
5.1.prepare deployment
5.2.complie the packages
5.3.create and bind vm
5.4.pull in job configurations
5.5.create needed job instances
8
OpenStack-CPI
• OpenStack CPI is an implementation of the BOSH CPI
• Leverages the Fog Ruby gem for OpenStack
http://fog.io
9
OpenStack Ready?
• Fog Version?
• bosh openstack cpi
• Validate OpenStack
• https://docs.cloudfoundry.org/deploying/openstack/validate_openstack.html
10
OpenStack Ready?(cont.)gem install XXX -v XXX
require 'fog'
EXCON_DEBUG=true
nova=Fog::Compute.new({:provider => 'openstack', :openstack_tenant => 'jiangytcn', :openstack_api_key => 'XXXX', :openstack_username => 'jiangytcn', :openstack_auth_url => 'https://XXX.cn.ibm.com:35357/v3/auth/tokens', :openstack_domain_name => 'default', :connection_options => { :ssl_verify_peer => false}})
nova.list_servers
nova.servers.create(name = 'trusty', image_ref = image, flavor_ref = flavor, key_name = my_key)
Similar for Volumes, Networks, Images, etc.
11
Nova• QEMU KVM
• Over Commit
• Flavor
• ephemeral >= 2* ram If has ephemeral
• Security Groups
• Only open required ports, remote CIDR/SG
• VM runs OK?
• ping 127.0.0.1
• (Compute Node) watch 'tail /sys/kernel/mm/ksm/*'
12
Resource Pool• Scheduler hints
• AZ Affinity-Group Anti-affinity-group
• No SPOF
• Need more physical resource
13
Neutron• Overlap
• Vlan gre vxlan
• Isolated
• IP translation, controlled
• Flat
• Flat-dhcp
• Shared
• Directly to switch✤MTU Selection
PMTU
ping -D -g <minsize> -G <maxsize> -h <sizeincr> <host>
✤ JUMBO FRAMES
✤meta-data/user-data
14
Cinder• Backend
• LVM CEPH ...
• Disk Readonly *
15
Swift - BlobStore• user-uploaded applications, buildpacks, droplets, and
application resources
• Tempurl
• X-Account-Meta-Temp-URL-Key
• runner download packages using tempurl
• Ratelimit(Optional. If no account or container limits are provided there will be no rate limiting)
• NTP
16
Keystone
• Tenant/Project with Large Quotas
• Day one & Day two
17
Deployment• keystone
• v2 or v3
• network type
• manual VS dynamic VS static
18
Cloud Foundry?
19
CF push workflow
20
Grouter & Load Balancer• Domains
• system, apps
• Gorouter only support single certs
• TLS Pass-Through
• Performant and a single TLS connection
• TLS Termination at Load Balancer
• Less performant option
• Multiple certificates to be used
• Internal
• Haproxy
• External LB
• Third party
• LBaaS
21
BlobStore• NFS
• Webdav
• Swift
22
Cloud Controller• resources clean up
app_events:
cutoff_age_in_days: 36
app_usage_events:
cutoff_age_in_days: 36
audit_events:
cutoff_age_in_days: 36
failed_jobs:
cutoff_age_in_days: 7
23
Database• Cluster
• Postgresql
• Mysql
• Backup
• Transaction Logs
• Full backup
24
DB Encrypt/Decrypt
25
CCDB.apps
DB Encrypt/Decrypt• Cloud Controller
• db_encryption_key
• row level encrypt, salt
26
def encrypt(input, salt)
return nil unless input
Base64.strict_encode64(run_cipher(make_cipher.encrypt, input, salt))
end
def decrypt(encrypted_input, salt)
return nil unless encrypted_input
run_cipher(make_cipher.decrypt, Base64.decode64(encrypted_input), salt)
end
def make_cipher
OpenSSL::Cipher::Cipher.new(ALGORITHM)
end
def run_cipher(cipher, input, salt)
cipher.pkcs5_keyivgen(db_encryption_key, salt)
cipher.update(input).tap { |result| result << cipher.final }
end
27
DB Encrypt/Decrypt
28
Buildpacks• application runtime
• installed buildpack
• system build pack will be override every update
29
Backing up Cloud Foundry• Platform
• BOSH DB • configuration files • CCDB UAADB • Blobstore • Credentials
• User • apps • service instance
30
Bosh DB• DNS
• for internal use
• VM and Volume mapping
31
Configuration• deployment manifest
bosh download manifest <DEPLOYMENT> cf.yml
kept it
• key pairs, certs
32
33
You can find me at: • @jiangytcn • [email protected]
Validate OpenStack
core component vm metadata service
Cloud Foundry Network Type Data encrypt/decrypt
Backup DB services manifest