rpi talk foster september 2011

www.ci.anl.govwww.ci.uchicago.edu

Accelerating data-intensive science by outsourcing the mundane

Ian Foster

The data deluge

1330 molec. bio databases Nucleic Acids Research (96 in Jan 2001)

Genomic sequencing output x2 every 9 month>300 public centers

100,000 TB

MACHO et al.: 1 TB

Palomar: 3 TB2MASS: 10 TB

GALEX: 30 TBSloan: 40 TB

Pan-STARRS: 40,000 TB

Climate model intercomparisonproject (CMIP) of the IPCC

2004: 36 TB

2012: 2,300 TB


4

Big science has achieved big successes

All build on NSF OCI (& DOE)-supported Globus Toolkit software

LIGO: 1 PB data in last science run, distributed worldwide

ESG: 1.2 PB climate datadelivered to 23,000 users; 600+ pubs

OSG: 1.4M CPU-hours/day, >90 sites, >3000 users, >260 pubs in 2010

Robust production solutionsSubstantial teams and expenseSustained, multi-year effortApplication-specific solutions, built on common technology


5

But small science is struggling

More data, more complex dataAd-hoc solutionsInadequate software, hardwareData plan mandates


6

Medium-scale science struggles too!• Dark Energy Survey

receives 100,000 files each night in Illinois

• They transmit files to Texas for analysis … then move results back to Illinois

• Process must be reliable, routine, and efficient

• The cyberinfrastructure team is not large

Image credit: Roger Smith/NOAO/AURA/NSF

Blanco 4m on Cerro Tololo


7

The challenge of staying competitive

"Well, in our country," said Alice … "you'd generally get to somewhere else — if you run very fast for a long time, as we've been doing.”

"A slow sort of country!" said the Queen. "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!"


8

Current approaches are unsustainable

• Small laboratories– PI, postdoc, technician, grad students– Estimate 5,000 across US university community– Average ill-spent/unmet need of 0.5 FTE/lab?

• Medium-scale projects– Multiple PIs, a few software engineers– Estimate 500 across US university community– Average ill-spent/unmet need of 3 FTE/project?

• Total 4000 FTE: at ~$100K/FTE => $400M/yr Plus computers, storage, opportunity costs, …


9

And don’t forget administrative costs

42% of the time spent by an average PI on a federally funded research project was reported to be expended on administrative tasks related to that project rather than on research — Federal Demonstration Partnership faculty burden survey, 2007


10

You can run a company from a coffee shop


11

Because businesses outsource their IT

Web presence Email (hosted Exchange) Calendar Telephony (hosted VOIP) Human resources and payroll Accounting Customer relationship mgmt

Software as a Service

(SaaS)


12

And often their large-scale computing too

Web presence Email (hosted Exchange) Calendar Telephony (hosted VOIP) Human resources and payroll Accounting Customer relationship mgmt Data analytics Content distribution

Infrastructure as a Service

(IaaS)

Software as a Service

(SaaS)


13

Let’s rethink how we provide research IT

Accelerate discovery and innovation worldwide by providing research IT as a service

Leverage software-as-a-service to• provide millions of researchers with

unprecedented access to powerful tools; • enable a massive shortening of cycle times in

time-consuming research processes; and• reduce research IT costs dramatically via

economies of scale


14

Time-consuming tasks in science

• Run experiments• Collect data• Manage data• Move data• Acquire computers• Analyze data• Run simulations• Compare experiment

with simulation• Search the literature

• Communicate with colleagues

• Publish papers• Find, configure, install

relevant software• Find, access, analyze

relevant data• Order supplies• Write proposals• Write reports• …


15

Time-consuming tasks in science

• Run experiments• Collect data• Manage data• Move data• Acquire computers• Analyze data• Run simulations• Compare experiment

with simulation• Search the literature

• Communicate with colleagues

• Publish papers• Find, configure, install

relevant software• Find, access, analyze

relevant data• Order supplies• Write proposals• Write reports• …


16

A B

Data movement can be surprisingly difficult


17

A B

Discover endpoints, determine available protocols, negotiate firewalls, configure software,

manage space, determine required credentials, configure protocols, detect and respond to failures, determine expected performance, determine actual performance, identify diagnose and correct network misconfigurations, integrate with file systems, …

Data movement can be surprisingly difficult

It took 2 weeks and much help from many people to move 10 TB between California and Tennessee.

(2007 BES report)


18

Globus Online’s SaaS/Web 2.0 architecture

Fire-and-forget data movementAutomatic fault recoveryHigh performanceNo client software installAcross multiple security domains

Web interface

HTTP REST interfacePOST https://transfer.api.globusonline.org/ v0.10/transfer <transfer-doc>

Command line interfacels alcf#dtn:/scp alcf#dtn:/myfile \ nersc#dtn:/myfile

GridFTP serversFTP servers

Other protocols:HTTP, WebDAV, SRM, …

Globus Connecton local computers

(Hosted on)

(Operate) OpenIDOAuthShibboleth


19

Example application: UC sequencing facility

Sequencing instrument

Mac using Globus Connect

iBi File Server

iBi general-purpose compute cluster

Sequencing-specific compute cluster

Mount drive

Delivery of data to customer


20

Statistics and user feedback

• Launched November 2010>1700 users registered>500 TB user data moved>30 million user files moved>150 endpoints registered

• Widely used on TeraGrid/XSEDE; other centers & facilities; internationally

• >20x faster than SCP• Faster than hand-tuned

“Last time I needed to fetch 100,000 files from NERSC, a graduate student babysat the process for a month.”

“I expected to spend four weeks writing code to manage my data transfers; with Globus Online, I was up and running in five minutes.”

“Transferred 28 MB in 20 minutes instead of 61 hours. Makes these global climate simulations manageable.”


21

Moving 586 Terabytes in two weeks


22

Monitoring provides deep visibility

Terabyte

Gigabyte

Megabyte

Kilobyte

20 Terabytes in less than one day

20 Gigabyes in more than two days


24

Common research data management steps

• Dark Energy Survey• Galaxy genomics• LIGO observatory

• SBGrid structural biology consortium• NCAR climate data applications• Land use change; economics


25

We have choices of where to compute

• Campus systems– First target for many researchers

• XSEDE supercomputers– 220,000 cores, peer-reviewed awards– Optimized for scientific computing

• Open Science Grid– 60,000 cores; high throughput

• Commercial cloud providers– Instant access for small tasks– Expensive for big projects

Users insist that they need everything connected


26

Towards “research IT as a service”


27

Research data management as a service• GO-User

– Credentials and other profile information

• GO-Transfer– Data movement

• GO-Team– Group membership

• GO-Collaborate– Connect to collaborative

tools: Jira, Confluence, …

• GO-Store– Access to campus, cloud,

XSEDE storage• GO-Catalog

– On-demand metadata catalogs

• GO-Compute– Access to computers

• GO-Galaxy– Share, create, run

workflows

Today

Fall

Prototype


28

SaaS services in action: The XSEDE vision

XUAS


29

Data analysis as a service: Early steps

Securely and reliably:1. Assemble code2. Find computers3. Deploy code4. Run program5. Access data6. Store data7. Record workflow8. Reuse workflow

[3, 4]

VM imageApp codeWorkflowGalaxyCondor

Data store

[5, 6]

We have built such systems for biological, environmental, and economics researchers

[1, 2]

[7, 8]


30

SaaS economics: A quick tutorial

• Lower per-user cost (x10?) via aggregation onto common infrastructure– $400M/yr $40M/yr?

• Initial “cost trough” due to fixed costs

• Per-user revenue permits positive return to scale

• Further reduce per-user cost over time

$

Time0

X10 reduction in per-user cost: $50K $5K/yr per lab $300K $30K/yr per project


31

A national cyberinfrastructure strategy?

LL

LL

L

LL

L

LL

L

LL

L

LL

L

LL

L

LL

L

LL

L

LP P P P

Research data management Collaboration, computationResearch administration

• To providemore capability formore people at less cost …

• Create infrastructure – Robust and universal– Economies of scale– Positive returns to scale

• Via the creative use of– Aggregation (“cloud”)– Federation (“grid”)

Small and medium laboratories and projects

aaS

P


32

Acknowledgments

• Colleagues at UChicago and Argonne– Steve Tuecke, Ravi Madduri, Kyle Chard, Tanu Malik,

and others listed at www.globusonline.org/about/goteam/

• Carl Kesselman and other colleagues at other institutions

• Participants in the recent ICiS workshop on “Human-Computer Symbiosis: 50 Years On”

• NSF OCI and MPS; DOE ASCR; and NIH for support


33

For more information

• www.globusonline.org; @globusonline: Twitter• Foster, I. Globus Online: Accelerating and

democratizing science through cloud-based services. IEEE Internet Computing(May/June):70-73, 2011.

• Allen, B., Bresnahan, J., Childers, L., Foster, I., Kandaswamy, G., Kettimuthu, R., Kordas, J., Link, M., Martin, S., Pickett, K. and Tuecke, S. Globus Online: Radical Simplification of Data Movement via SaaS. Communications of the ACM, 2011.

http://www.globusonline.org/


Thank you!

[email protected]

www.globusonline.org@globusonline

rpi talk foster september 2011

Technology

science run

pb data

relevant data

strugglingmore data

big science

data delugemacho

funded research project

small science