![Page 1: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/1.jpg)
Big Data In the
Cloud@natishalom
![Page 2: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/2.jpg)
2
About GigaSpaces
Managing Big Data on the Cloud
100s of Enterprise Customers
![Page 3: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/3.jpg)
My Data Out of My Hands..
No Way!
![Page 4: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/4.jpg)
4
The Reality of Big Data..
2.7 ZB
0.5 Petabytes
66%
Global Digital Data
Two years tweets
Plan to use Big Data/Cloud
43% think that data
analytics could be improved in their organization if data analytics was part of
cloud services
![Page 5: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/5.jpg)
Large ISV Case Study
• Application– Call Center surveillance
• Background– Previously – voice data
• Goal for a new system– Monitor data & voice– Multiple data sources – Advanced correlations
![Page 6: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/6.jpg)
The Challenges...
Ever Growing Data
Deeper Correlation
Tight Performance
![Page 7: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/7.jpg)
A Classic Case for...
![Page 8: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/8.jpg)
A Typical Big Data System…
![Page 9: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/9.jpg)
The Challenge
Cost Business Impact
Lower Margins
Competiveness
Time to Market
Customer Satisfaction
Infrastructure
Operational
![Page 10: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/10.jpg)
The Solution Big Data
in the Cloud
![Page 11: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/11.jpg)
Big Data in the Cloud - 3 Reasons
• Skills– Do you really need/want this all in-
house?• Huge amounts of external data – Does it make sense to move and
manage all this data behind your firewall?
• Focus on the value of your data– Instead of big data management
Holger Kisker
![Page 12: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/12.jpg)
Managing Big Data on the
Cloud
• Auto start VMs• Install and configure
app components • Monitor • Repair • (Auto) Scale• Burst…
![Page 13: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/13.jpg)
Big Data in the Cloud...
Reduce the Infrastructure Cost
Choose the Right Cloud for the Job
Use Eucalyptus for private data , AWS for sporadic workloads..
![Page 14: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/14.jpg)
Big Data in the Cloud...
Reducing the Operational Complexity
• Consistent Management
• Automation Through the Entire Stack
![Page 15: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/15.jpg)
Let’s Take a Closer Look…
Consistent Management
Portability
Automation
![Page 16: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/16.jpg)
16
Consistent ManagementRecipes consistent description for running any app:
What middleware services to run Dependencies between services How to install services Where application and service binaries are When to spawn or terminate instances How to monitor each of the services
![Page 17: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/17.jpg)
The Right Cloud for the Job (Cloud
Portability)
![Page 18: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/18.jpg)
18
Choosing the Right Cloud for the Jobcompute { template "SMALL_LINUX"}
SMALL_LINUX : template imageId "us-east-1/ami-76f0061f“ remoteDirectory "/home/ec2-user/gs-files“ machineMemoryMB 1600 hardwareId "m1.small" locationId "us-east-1" localDirectory "upload" keyFile "myKeyFile.pem"
options ([ "securityGroups" : ["default"]as
String[], "keyPair" : "myKeyFile"])
overrides (["jclouds.ec2.ami-query":"",
"jclouds.ec2.cc-ami-query":""])privileged true
}
SMALL_LINUX : template{ imageId linuxImageId remoteDirectory "/home/user/gs-files" machineMemoryMB 1600 hardwareId “m1.medium” locationId “us-west-1” localDirectory "upload“ keyFile “myEucaKeyFile.pem” username "user" options ([ "securityGroups" : ["default"] as String[], "keyPair" : keyPair ]) overrides ([“endpoint” : “http://communitycloud.eucalyptus.com”]) privileged true}
![Page 19: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/19.jpg)
Automation across the stack1 Upload your recipe.
2 Cloudify creates VM’s & installs agents
3 Agents install and manage your app
4 Cloudify automate the scaling
![Page 20: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/20.jpg)
Big Data Apps, on Any Cloud, Your Way
Open Source (Apache2)
![Page 21: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/21.jpg)
Big Data On Demand with CloudifyRelational DB Clusters NoSQL Clusters Hadoop
MySQL MongoDB Hadoop (Hive, Pig,..)
Postgress Cassandra Storm
Couchbase ZooKeeper
ElasticSearch
![Page 22: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/22.jpg)
® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
22
Demo Time: Storm Cluster
![Page 23: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/23.jpg)
Large ISV Case Study
• Application– Call Center surveillance system
• Background– Previously – voice data
• Goal for a new systemMonitor data & voiceMultiple data sources Advanced correlations Mission
Accomplished
![Page 24: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/24.jpg)
Additional Benefits
• True Cloud Economics
• One product -> Any Customer Environment
• Increased Agility
![Page 25: Scaling and Managing Big Data Apps in the Cloud](https://reader033.vdocument.in/reader033/viewer/2022061218/54b6ed254a7959fd508b4594/html5/thumbnails/25.jpg)
Thank You!
References: http://www.cloudifysource.org http://github.com/CloudifySource