Cloud Computing Development

Cloud Computing Development Cloud Computing Development Shallow IntroductionIntroduction

What is the cloud computingIs it computing while in flight?

Image Courtesy SevensHeaven.nlWhat is the cloud computingIs it computing while in flight?

NO

Image Courtesy SevensHeaven.nlWhat is the cloud computingWhat is it about then?

What is the cloud computingWhat is it about then?

Cloud computing is consumption of computing resources without worrying about specifics. What is the cloud computingWhat is it about then?

As well as ability to add or remove resources according to the demand.

What is the cloud computingWhat is it about then?

Similar to the power grid andtelephone network.

What is the cloud computingWhat is it about then?

Similar to the power grid andtelephone network.

How does it work? Consumer signs up for the service. (Same as if you get a mobile phone plan) Consumer uses services according to their needs Provider sends the bill at the end of the cycle Consumer pays Provider ModelsSoftware As A Service

SAASEmailCRMOffice AppsProvider ModelsSoftware As A Service

SAASEmailCRMOffice AppsPlatform As A Service

PAASApplication ServersDatabasesMiddlewareProvider ModelsSoftware As A Service

SAASEmailCRMOffice AppsPlatform As A Service

PAASApplication ServersDatabasesMiddlewareInfrastructure As A Service

IAASBare Hardware (Sort of )ProvidersSoftware As A Service

SAASGoogle (GMail)Salesforce Microsoft (Office Live)ProvidersSoftware As A Service

SAASGoogle (GMail)Salesforce Microsoft (Office Live)Platform As A Service

PAASGoogle App EngineHeroku / Engine Yard (Rails)Windows Azure (.NET)ProvidersSoftware As A Service

SAASGoogle (GMail)Salesforce Microsoft (Office Live)Platform As A Service

PAASGoogle App EngineHeroku / Engine Yard (Rails)Windows Azure (.NET)Infrastructure As A Service

IAASAmazon AWSRackspaceGoGridProvider: Windows Azure Platform as a service Windows based Storage provided through blob storage, drives, SQL Azure State is stored and propagated with Queues and Tables Integrated with Visual Studio Eclipse plug-in for PHP

Slide courtesy Vlad Vinogradsky from MicrosoftProvider: Google App Engine Platform as a service Python or Java based Storage provided through BigTable Automatically scales web nodesProvider: Rackspace Infrastructure as a service Very Basic just a few Linux or Windows images Provides storage with CloudFiles Very Cheap Open source API Relatively NewProvider: Amazon AWS Oldest on the market Many services / Images / Third party providers Provides computation through EC2 / EMR Provides state / storage through S3, SQS, RDS, SimpleDB Multiple APIsSample PricesAmazonCompute $0.10+ VM/Hr Storage $0.15+ GB/Month $0.15+ GB/XFer RackspaceCompute $0.02+ VM/Hr Storage $0.15+ GB/Month $0.22+ GB/XFer Microsoft

Compute $0.12 VM/HrStorage $0.15 GB/moBandwidh $0.15 GB/XFerDevelopmentPractical ConsiderationsCloud Development is slightly different from traditional in house model.

Practical ConsiderationsCloud Development is slightly different from traditional in house model.Everything is virtualized (most of the time)Everything is distributed Per instance reliability is much lowerOverall reliability is much higher

Cloud Programming Model

Cloud Programming ModelCompute and Interface nodes are not reliable, they can crash and disappear at any time.Storage and State are reliable and heavily distributed.At any time we can start more compute or interface nodes and shut them down when demand subsides.

Cloud Programming Model on Azure Compute : Worker Nodes State: Tables / Queues / SQL Storage: SQL / Tables / Blobs / Drives Client Inteface: Web Nodes

Cloud Programming Model on AWSCompute : EC2 InstancesState: S3 / Queues / SimpleDB / RDSStorage: S3 / SimpleDB / RDS Client Inteface: S3 / EC2 / CloudFront AWS Details: S3S3 = Simple Storage ServiceGuaranteed to be reliableSimple {Key, Value} storageKeys are stored within bucketsValues could be as large as 5GBDefault Storage Mechanism for AWSAWS Details: Simple DBSchema less databaseMain storage unit is domain ( similar to table )Each record can have many attributes, new attributes could be added at any timeSimilar to LISP / Scheme attributesCan query domain for records containing particular attributeNo Joins / Unions with other domainsEventual ConsistencyAWS Details: RDS RDS = Relational Data Storage MySQL in a cluster mode Preferred to simply running DB server within instance (ask me why for details)AWS Details: SQSSQS = Simple Queue SystemMassively scalableAllows to put message in the queue and retrieve later onRetrieving the message hides it from the other usersWhen message is processed it is deleted from the queueIf message is not deleted before the timeout it is returned backAWS Details: EC2EC2 = Elastic Compute CloudAllows to run arbitrary virtual machinesProvided they are compatible with Amazons modified XenKernels and Startup Disks are stored in S3Also have large local storageMachines are not exactly like physical machinesLocal storage is not persistentWhen machine is shut down all local data disappears.Hardware TCP [No packet layer / No Broadcast ]Can launch many copies of the machine at the same timeLots of preconfigured machinesAWS Details: Other ServicesEMR = Elastic Map ReduceLets run Hadoop jobs on EC2 CloudFront Content Delivery NetworkELB = Elastic Load BalancerEBS = Elastic Block StorageS3 backed persistent storagePublic Data Sets - Lots of publicly available data Census ( 1980 , 1990, 2000 ), Wikipedia logs, Freebase dumps, Genetic and Chemistry dataStarting UpAmazon AccountCredentials KeyID : SecretKeyX509 CeriticateHelpful ToolsS3 Fox - Firefox extension for browsing S3Elastic Fox - Firefox extension for operating EC2Transmit - Mac utility for S3 ($)Right Scale - Web based platform for managing everything ( Free / $ )LibrariesOfficial Amazon Libraries (Java)Unofficial Libraries - .Net / Ruby / PerlAWS4C - C/C++/Objective C Boto - Very popular Python library (official Hadoop/EC2 library)DemoDemo

Running Hadoop on EC2Questions

????