jeff kramer (hp helion) scaling hadoop in & out of … hadoop.pdfmaking chocolate marshmallows...

20
SCALING HADOOP IN & OUT OF THE PRIVATE CLOUD JEFF KRAMER (HP HELION)

Upload: others

Post on 27-Sep-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

S C A L I N G H A D O O P I N & O U T O F T H E P R I VAT E C L O U D

J E F F K R A M E R ( H P H E L I O N )

Page 2: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

P R I VA T E C L O U D S A R E L I K E …

Flickr Photo by John Lustig https://www.flickr.com/photos/john_lustig/2614212165/

Page 3: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

S O , O P E N S TA C K

Page 4: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

R A T H E R …

Page 5: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

O P E N S TA C K P R I VA T E C L O U D O F F E R I N G S

• HP • IBM • Canonical • Red Hat • Blue Box • EMC • Cisco • or Roll Your Own!

Page 6: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

Y O U R P R I VA T E C L O U D

Flickr Photo by Tristan Schmurr https://www.flickr.com/photos/kewl/8475764430

Page 7: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

W E L C O M E T O T H E S A VA N N A S A H A R A

… O R N O T ?Flickr Photo by John Y. Can https://www.flickr.com/photos/yavuzcan/8175834112

Page 8: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

B R I N G Y O U R D A TA

Flickr Photo by Joel Franusic https://www.flickr.com/photos/jpf/152611490

… O R N O T !

Page 9: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

I T ’ S A L L A B O U T P L A C E M E N T

Page 10: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

E L E P H A N T S TA C K

Flickr Photo by Heather Katsoulis https://www.flickr.com/photos/hlkljgk/3504681812

Page 11: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

Flickr Photo by Natalia Wilson https://www.flickr.com/photos/moonrat/6654701075

H A R D A N D S Q U I S H Y B I T S …

Page 12: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

Flickr Photo by Quinn Dombrowski https://www.flickr.com/photos/quinnanya/5203889651

H A D O O P A S A S Q U I S H Y W O R K L O A D

Page 13: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

D O HFlickr Photo by Thomas Quine https://www.flickr.com/photos/quinet/11062763654

Page 14: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

C H O C O L A T E M A R S H M A L L O W S !

Flickr Photo by Bryan Ochalla https://www.flickr.com/photos/bochalla/5479673618

Page 15: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending
Page 16: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

M A K I N G C H O C O L A T E M A R S H M A L L O W S

Cloud FoundryCluster

Hadoop Cluster

1. Place Scale-Up Order2. Order Set to Pending

Nova Plugin6. CF Cluster Views Now-Open Scale-Up Order7. Creates New VMs8. Acknowledges Order

3. Plugin Notes Pending Order4. Plugin Creates Scale-Down Order5. Sahara Views Order & Scales Down 1 Instance

Page 17: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

T H E N O - Q U O TA , P R I O R I T Y B A L A N C E D , U T O P I A N P R I VAT E C L O U D F U T U R E

• Quotas are a Pain

• Workloads are Opaque

• Workloads are Spiky

• Workloads Know Themselves

• Cloud Managers Need More Knobs

• Bindings Need to be Loose

Flickr Photo by Eugene Peretz https://www.flickr.com/photos/peretzpup/2253718230

Page 18: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

O T H E R S T U F F

• Whither Mesos?

• CloudBreak

• DockerDockerDocker

Flickr Photo by Eugene Peretz https://www.flickr.com/photos/peretzpup/2253718230

Page 19: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

Flickr Photo by Eli Christman https://www.flickr.com/photos/gammaman/7144684205

Page 20: JEFF KRAMER (HP HELION) SCALING HADOOP IN & OUT OF … Hadoop.pdfMAKING CHOCOLATE MARSHMALLOWS Cloud Foundry Cluster Hadoop Cluster 1. Place Scale-Up Order 2. Order Set to Pending

github.com/HPCloud/os-workloads

@jeffk