1. topics is cloud computing the way to go? arc abm review configuration basics setting up the arc...
TRANSCRIPT
1
TopicsIs Cloud Computing
the way to go?ARC ABM ReviewConfiguration BasicsSetting up the ARC
Cloud-Based ABMHardware
ConfigurationSoftware
ConfigurationRun TimesInputs and OutputsClient User InterfaceFuture WorkQuestions 2
Cloud ComputingOn-demand use of remote computer resourcesExamples:
Yahoo MailGoogle Apps (such as Calendar, Docs, etc)Cube Cloud ControllerAmazon Elastic Compute Cloud (Ec2)
Key characteristics of cloud computing:ScalableFee structure usually on-demand or
subscription-basedSupports multiple users/instances
3
ARC ABM Review Based on the CT-RAMP family of ABMs
developed, or being developed, in Columbus, Atlanta, the San Francisco Bay Area, San Diego, Phoenix, Chicago, Miami, and other regions
ARC model implemented with Cube-based networks, assignments and skimming, GUI and ancillary models (external model, truck model, etc)
Population Size: 1.7 million households in 2005, 2.7 million in 2030
Uses Java Parallel Processing Framework and Cube Cluster to thread and distribute work across multiple machines
Base year runs 3 feedback loops in 16 hours on the ARC cluster using 3 Windows 64bit machines with 8 processors and 32 GB of RAM each
4
Hall
Fulton
Carroll
Bartow
Cobb
Coweta
Henry
Gwinnett
Walton
Cherokee
DeKalb
Newton
Paulding
Forsyth
Fayette
Douglas
Spalding
Barrow
Clayton
Rockdale
¯
ARC ABM System Architecture
5
Configuration BasicsGoal: Setup the ABM in the cloud and see how it
performsTwo basic approaches to the cloud-based setup:
System is open so the user can configure and use custom remote machines Example: rent a few instances (i.e. remote machines) from
Amazon E2C and configure them as needed Advantage: flexibility Disadvantage: complexity
System is configured ahead of time and a client user interface is developed that sits between the user and the remote machines Example: Cube Cloud Controller, which wraps a user
interface around Amazon EC2 and allows the user to upload files, run models, and get results
Advantage: ease-of-use Disadvantage: lack of flexibility
6
Setting Up a Cloud-Based ABMSelect and Configure Machine Instances
Standard Instances (1.7GB RAM, 1 Core, 160GB HD, 32bit) Large Instances (7.5GB RAM, 4 Core, 850GB HD, 64bit) Extra Large Instances (15GB RAM, 8 Core, 1690GB HD, 64bit)
High-Memory Extra Large Instances (17GB RAM, 6.5 Core, 420GB HD, 64bit) High-Memory Double Extra Large Instances (34GB RAM, 13 Core, 850GB HD,
64bit) High-Memory Quadruple Extra Large Instances (68GB RAM, 26 Core, 1690GB
HD, 64bit)
Cluster Compute Quadruple Extra Large Instances (23GB RAM, 33.5 Core, 1690GB HD, 64bit, 10GBit Ethernet)
Windows Server or various Linux versionsVarious database options and web server options as wellEstimate overall price tag:
http://aws.amazon.com/calculator
7
Setting Up a Cloud-Based ABMOn-Demand Instance Pricing
8
Setting Up a Cloud-Based ABMReserved Instance Pricing
9
Setting Up a Cloud-Based ABMARC ABM Example Computing Cost (which
does not include data transfer costs)Configure 3 High-Memory Double Extra Large
InstancesOn-Demand
16 hour model run - $60Reserved Instances for 3 years
Initial fee - $12,000 16 hour model run - $23 Need to run the model 324 times in 3 years to
justify initial fee
10
Setting Up a Cloud-Based ABMManage your instances with Management ConsoleRemote desktop to instances and install Java and CubeWe used Amazon EC2 instances configured by Citilabs
so they already had a Cube site license (no hardware key version) installed on them
However, most modeling packages require a hardware keyThis is not possible with Amazon EC2Possible future solutions to this include:
Remote license checkout Purchasing a one time use key for each model run instead of
purchasing an unlimited use hardware keyThe cloud machines are only used for running the
model, and network editing is still done on a local machine
11
Run TimesTwo runs tested at this point:
3 machines with 6 feedback loops6 machines with 6 feedback loops
12
IterationRun Times (hh:mm)
3 machine 6 machine1 3:11 3:032 6:11 2:503 7:38 4:074 7:16 4:375 7:32 5:046 11:37 7:30
Total Java Run Time 43:25 27:11Total Model Run Time 48:10 31:25
The 3 machine cloud run time is comparable to the ARC cluster run time
The move from 3 to 6 machines resulted in a ~35% reduction in run times
More test runs are planned, including with more instances, as well as with some performance tuning
Another AppproachMore model runs (32-cores machines, 64, 128, 256,
512):Increase in computing instances resulted in significant
non-linear reductions in the ABM run time. Doubling the number of cores from 32 to 64 reduced
the run time by 37 percent. Doubling it again reduced the run time relative to the
32 core run by 55 percent. The 256 and 512 core runs show little improvement
beyond the 128 core run. ARC ABM computing power sweet spot is somewhere
around 128 cores.
13
Inputs and OutputsSeparate cost to transfer data “in” and “out”
of Amazon’s Ec2 cloud
14
Inputs and OutputsARC ABM Example Data Transfer Cost
250MB for zipped complete model setup – $0.025 However, only key inputs for a scenario are likely to
be uploaded so the cost is usually less15 GB of output for a compute model run –
$2.25 However, only key outputs are likely to be
downloaded so the cost is usually less
15
Client User InterfaceAmazon EC2 is designed for software
developers so it is not really that easy to useAs a result, a client web-based user interface
would need to be developed that allows the user to easily use the cloud-based ABM
An example client user interface is Cube Cloud Controller, which allows users to:Upload model inputsRun and manage scenariosDownload model outputs
16
Future WorkExperiment with Cube Cloud Controller
for managing runs in the cloudEase-of-useCostMultiple-user capabilities
Develop system to allow multiple users to run the ABM simultaneously or in a more coordinated fashion - we need a cluster is busy light
ARC review the costs of in-house runs versus cloud-runs and develop a work program that is most efficient for the agency
17