operating your openstack private cloud.pdf
DESCRIPTION
trueTRANSCRIPT
October 12, 2012
Operating your OpenStack Private Cloud
Ryan RichardOpenStack Engineer
[email protected]@rackninja
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Monitoring and Reporting
Where we were -‐ April 2012
Basic CDM
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Monitoring and Reporting
Where we were -‐ April 2012
Basic CDMNow
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Monitoring and Reporting
Where we were -‐ April 2012
Basic CDMNow
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Tools
There is no good way to get the following info:
I need a list of instances on a host and their IPs
I need to gracefully start/stop all instances on a host
Some tools needs hostname, some need id (decimal or hex), some need uuid
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Tools
There is no good way to get the following info:
I need a list of instances on a host and their IPs
I need to gracefully start/stop all instances on a host
Some tools needs hostname, some need id (decimal or hex), some need uuid
SELECT instances.id,instances.hostname,instances.project_id,fixed_ips.address as fixed_address,floating_ips.address as floating_address FROM instances LEFT JOIN fixed_ips ON instances.id=fixed_ips.instance_id LEFT JOIN floating_ips ON floating_ips.fixed_ip_id=fixed_ips.id WHERE instances.deleted="NULL" AND instances.host="<hostname of physical machine>" ORDER BY instances.id;
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4
Tools
WE NEED BETTER OPS TOOLS!
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4
Tools
Pulsar
https://github.com/rsoprivatecloud/pulsar
“nova swiss army knife”
requires direct nova database access
WE NEED BETTER OPS TOOLS!
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4
Tools
Pulsar
https://github.com/rsoprivatecloud/pulsar
“nova swiss army knife”
requires direct nova database access
WE NEED BETTER OPS TOOLS!
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Tools
Holland (opensource database backup framework)
Written by Rackspace DBAs
http://wiki.hollandbackup.org/
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Tools
dsh
dsh -Mcg compute uname-a
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Tools
dsh
dsh -Mcg compute uname-a
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Tools
dsh
dsh -Mcg compute uname-a
bashfoo
for i in `knife node list | grep cpu`; do knife node run_list add $i "role[single-‐compute]"; done
for k in `seq 1 20`; do for i in {compute,network}; do nova-‐manage service disable computevm0$k nova-‐$i; done; done
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Performance and Scale Considerations
Disk IO
For high performance use remote block storage
For “local” disk IO, raw image type is only slightly faster than qcow2
IO will degrade while Glance copies images between machines
scheduler=cfq, KVM cache=none
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Performance and Scale Considerations
Disk IO
For high performance use remote block storage
For “local” disk IO, raw image type is only slightly faster than qcow2
IO will degrade while Glance copies images between machines
scheduler=cfq, KVM cache=none
0" 200" 400" 600" 800" 1000" 1200" 1400" 1600"
compute/host"
compute/host"(no"ht)"
compute/host"(deadline)"
Rs/speed/test13"(deadline"cache=none)"
Rs/speed/test13"(cfq,"cache=none,"no"ht)"
Rs/speed/test12"(cfq"cache=none)"
Rs/speed/test12"(noop"cache=none)"
rs/speed/test13"(cfq,"cache=writeback)"
Rs/speed/test13"(noop,"cache=writeback)"
rs/speed/test12"(cfq,"host"deadline,"cache=none)"
Async&Random&IO&
randW"(direct)"
randR"(direct)"
randW"
randR"
0"
2000"
4000"
6000"
8000"
10000"
12000"
14000"
randR" randW" randR"(direct)"
randW"(direct)"
seqR" seqW" seqR"(direct)"
seqw"(direct)"
Host&vs.&Instance&
compute/host"
Rs/speed/test12"(cfq"cache=none)"
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Performance and Scale Considerations
Glance chunk Size
200Mb chunk size
1GB chunk size
5GB chunk size
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Performance and Scale Considerations
Glance chunk Size
200Mb chunk size
1GB chunk size
5GB chunk size
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Performance and Scale Considerations
Glance chunk Size
200Mb chunk size
1GB chunk size
5GB chunk size
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Performance and Scale Considerations
Glance chunk Size
200Mb chunk size
1GB chunk size
5GB chunk size
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Performance and Scale Considerations
Swift disk usage with different chunk sizes
5 zones -‐ 4 x 1TB disks per zone
20TB raw -‐ 6.67TB usable
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Performance and Scale Considerations
Swift disk usage with different chunk sizes
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Performance and Scale Considerations
Glance chunk size
Too high and swift can become unbalanced
What are the downsides to being too low?
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Glance
Disk Tuning (swift)
read ahead on your block device(s) -‐ no noticeable gain
deadline scheduler -‐ no noticeable gain
Best thing for glance performance -‐ Caching
Performance and Scale Considerations
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Glance
Disk Tuning (swift)
read ahead on your block device(s) -‐ no noticeable gain
deadline scheduler -‐ no noticeable gain
Best thing for glance performance -‐ Caching
Image Size Not Cached Cached
1.4GB 20secs 1sec
16.4GB 2min 21secs 1sec
Performance and Scale Considerations
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Glance
Disk Tuning (swift)
read ahead on your block device(s) -‐ no noticeable gain
deadline scheduler -‐ no noticeable gain
Best thing for glance performance -‐ Caching
Image Size Not Cached Cached
1.4GB 20secs 1sec
16.4GB 2min 21secs 1sec
*times from “creating image” to “qemu-‐img
create”
Performance and Scale Considerations
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Performance and Scale Considerations
Scheduler
What we use by default:
scheduler tasks are not processed in parallel
Adding additional schedulers helps provide HA but they don’t speed up overall times to complete requests
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Automated Config Management
Chef: http://github.com/rcbops/chef-‐cookbooks
time to stand up
controller -‐ less than 20 minutes
compute node -‐ less than 2 min
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Day to Day tasks
Dealing with new issues
resize -‐ all nova-‐compute processes need to be able to log into all other compute nodes via ssh keys
Hardware failures
We’re still managing infrastructure, failures happen
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Lessons Learned
We need better Operations tools!
Network Design can be confusing for people used to “the old way”
OpenStack is still relatively new, help your organization understand it.
It’s easy to forget we’re working with Linux machines
It’s not you, it’s a bug :)
Thursday, October 18, 12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
But....
But this is a design summit also
Open to discussions/thoughts/questions
Thursday, October 18, 12
RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM
RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM
Rackspace is hiring
www.rackertalent.com
Thursday, October 18, 12