application skeletons: constructing and using abstract...
Post on 05-Oct-2020
1 Views
Preview:
TRANSCRIPT
www.ci.anl.govwww.ci.uchicago.edu
www.ci.anl.govwww.ci.uchicago.edu
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
DanielS.Katz1,AndreMerzky2,ZhaoZhang3,ShantenuJha2
1Computation Institute,UniversityofChicago&ArgonneNationalLaboratory2RADICALLaboratory,RutgersUniversity3AMPLab,UniversityofCalifornia,Berkeley
d.katz@ieee.org, andre.merzky.net, zhaozhang@eecs.berkeley.edu,shantenu.jha@rutgers.edu
www.ci.anl.govwww.ci.uchicago.edu
2
Motivation
• Computer scientistswhobuildtoolsandsystemsneedtoworkonrealscientificapplications toprovetheeffectiveness oftheirtoolsandsystems– Andoftenvarythem– changeproblemsize,etc.
• However, accessingandbuilding realapplicationscanbehard(andisn’treally thecoreoftheirwork)– Someapplications(source)areprivatelyaccessible– Somedataisdifficulttoaccess– Someapplicationsuselegacycodeandaredependentonout-of-date
libraries– Someapplicationsarehardtounderstandwithoutdomainscience
expertise– Realapplicationsmaybedifficulttoscaleormodifytodemonstratesystem
trendsandcharacteristics
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
3
PartoftheAIMESProjectAIMES:AbstractionsandIntegrativeMiddlewareforExtremeScales• Goal
– Improveprinciplesandpracticeofdistributeddynamicresourcefederation
• Motivation– Identifyabstractions;implement tostudy&support federation– Improvedynamicanddistributedexecution onheterogeneous distributed
computinginfrastructure (DCI)o “Howwillmyapplicationperformon thisDCI?”o “HowcanIbestadaptmyapplication toaDCI?”o “HowcanthesetofresourcesDCIbestadapttomyapplication?”o “Whydid thesystemallocatethisDCItomyapplication?o “What“variables”mattermost…matterleast?
– Understand howdistributedworkloadexecution canbemanagedo Hypothesis: Usingmiddlewarethatsupports theintegrationofapplication-level
andresource-levelinformationo Questions:Whataretherelevantdecisions, andatwhatlevelshould theybe
made?
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
4
AIMESApproach• Useabstractionsthatcanbeflexiblycomposedandsupporta
rangeofexperiments,including– Skeletons representprimaryDCIapplication characteristics– ResourceBundles provide real-time infoonstateofdiverse
resources– PilotJobsenabledynamical
distributed resourcefederation&management
– Execution strategy:temporally orderedsetofdecisions thatneedtobemadewhenexecutingagivenworkload
• Experimenttounderstandrequirements&trade-offs
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
5
SkeletonTarget
• Wewanttobuildatoolsothat– Userscanquicklyandeasilyproduceasyntheticdistributedapplication
thatrepresents thekeydistributedcharacteristics ofarealapplicationo Thesyntheticapplicationshouldhavetasktype(serialorparallel),runtime,I/O
buffer,I/Oquantity,computationandI/Ointerleavingpatternandintertaskcommunicationthataresimilartothoseoftherealapplication
– Thesyntheticapplicationiseasytoruninadistributedenvironment:grids,clusters,andclouds
– Thesyntheticapplicationshouldbeexecutablewithcommondistributedcomputingmiddleware (e.g., SwiftandPegasus)aswellastheubiquitousUnixshell
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
6
ClassesofDistributedApplications
• BagofTasks:asetofindependenttasks• MapReduce:asetofdistributedapplicationwithkey-valuepairs
asintermediatedataformat• IterativeMapReduce:MapReduceapplicationwithiteration
requirement• Campaign:aniterativeapplicationwithavaryingsetoftasks
thatmustberuntocompletionineachiteration• Multi-stageWorkflow:asetofdistributedapplicationswith
multiplestagesthatusePOSIXfilesasintermediatedataformat• ConcurrentTasks:asetoftasksthathavetobeexecutedatthe
sametime(notsupportedbycurrentwork)
• Notethatmost/allofthesearemany-taskapplications
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
7
Contributions
• Anapplicationabstractionthatgivesusersgoodexpressivenessandeaseofprogrammingtocapturethekeyperformanceelementsofdistributedapplications
• AversatileSkeletontaskimplementationthatisconfigurable(numberoftasks,serialorparalleltasks,amountofI/Oandcomputation,I/Obuffersize,computationandI/Ointerleavingoptions)
• AninteroperableSkeletonimplementationthatworkswithmainstreamworkflowframeworksandsystems(Swift,Pegasus,andShell)
• TheusageofSkeletonapplicationstosimplifysystemoptimizationimplementationandhighlighttheirimpacts
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
8
Challenge
• Balance the ease of programming and usage with the performance gap between Skeleton applications and real applications
Eas
e of
Pro
gram
min
g
Performance DifferenceSmall Large
Eas
yH
ard
Real App
Skeleton App
Simple Model
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
9
AnMulti-StageApplicationExample
• Applicationshavestages• Eachstagehastasks
– Taskshavetypes(serial/parallel)– Taskshavecomputationlengths– Input/Output fileshavesizes– I/Oisthroughbuffers– Inputfilescanbe(pre)existingfilesor
Outputfilesfrompreviousstages– ComputationandI/Ocanbe
interleaved
• Eachstagehasinput/output files– Inputfilesmaptotaskswithpatterns
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
10
SkeletonAbstraction
• ApplicationSkeletonsabstractanapplicationusingatop-downapproach:anapplicationiscomposedofstages,eachofwhichiscomposedoftasks.
• Anapplicationcanbedefinedbyaconfigurationfilecontaining:– Numberofstages– Foreachstage
o Tasktypes(serial/parallel)o Tasks(numberandcomputationlength)o Numberofprocessesforeachtasko Inputfiles(number,sizes,andmappingtotasks)o Outputfiles(number,sizes)o I/Obuffersizeo ComputationandI/Ointerleavingoption
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
11
SkeletonToolDesign
• TheSkeletontoolisimplementedasaparser.
ConfigurationFile
SkeletonTool
PreparationScripts
Executables
Application(PegasusDAG,SwiftScript,
Shellcommands)
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
12
TaskExecutable
• AlltasksimplementedasonestandaloneCprogramviaparameters
• CprogramcanbecompiledasserialwithGCC,asparallelwithMPICCcompiler.
• Anexecutionexample:– task serial 1 5 65536 65536 1 1 0
Stage_1_Input/Stage_1_Input_0_1 Stage_1_Output/Stage_1_Output_0_1 4200000
– Path_to_Task Task_TypeNum_Processes Task_LengthRead_BufferWrite_BufferNum_InputNum_OutputInterleave_Option [Input_File][Output_FileOutput_Size]
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
13
ABagofTaskApplicationExample
1. Num_Stage =12.3. Stage_Name =Bag4. Task_Type =serial5. Num_Processes =16. Num_Tasks =47. Task_Length =uniform58. Read_Buffer =655369. Write_Buffer =6553610. Input_Files_Each_Task =111. Input_1.Source=filesystem12. Input_1.Size=uniform210000013. Output_Files_Each_Task =114. Output_1.Size=uniform420000015. Interleave_Option =0
https://github.com/applicationskeleton/Skeleton/blob/master/src/sample-input/bag.inputApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
14
ABagofTaskApplicationExample
• Otheroptions– Task_Type canbeparallel
o Task_Type =parallel– Task_Length canbeastatisticaldistribution
o Task_Length =normal [20,3]– Task_Length canbeapolynomialfunctionofinputfilesizeo Task_Length =polynomial [20,3]Input_1o 20*Input_1.Size^3
– OutputsizecanbeapolynomialfunctionofTask_Lengtho Ouptut_1.Size =polynomial [10,2]Lengtho 10*Length^2
– Interleaving_Option canbe...
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
15
ABagofTaskApplicationExample
• Interleaving_Optioncanbe
read sleep write
interleave-nothing:
interleave-read-compute:
interleave-compute-write:
interleave-all:
time
0
1
2
3
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
16
AMulti-stageWorkflow
• Num_Stage =3• Stage_Name =Stage_1• …• Stage_Name =Stage_2• …• Write_Buffer =65536• Input_Files_Each_Task =2• Input_Task_Mapping =combination
Stage_1.Output_12• …
• Stage_Name =Stage_3• …• Input_Files_Each_Task =6• Input_Task_Mapping =combination
Stage_2.Output_16• …
https://github.com/applicationskeleton/Skeleton/blob/master/src/sample-input/multi-stage.input
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
17
Input_Task_Mapping
• SpecifyInput_Task_Mapping:– Input_Task_Mapping =combinationStage_1_output_12– Equivalentto“Nchoosek”mathematically– 4files,6tasks=>
o file0,file1:task0o file0,file2:task1o file0,file3:task2o file1,file2:task3o file1,file3:task4o file2,file3:task5
https://github.com/applicationskeleton/Skeleton/blob/master/src/sample-input/multi-stage.inputApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
18
Input_Task_Mapping
• IfInput_Task_Mapping isnotspecified– Input_Files_Each_Task =2– Input_1.Source =Stage_1.Output_1– Input_2.Source =Stage_1.Output_1
– Filesaremappedtotasksinanaturalorder– file0,file1:task0– file2,file3:task1– file4,file5:task2– file6,file7:task3
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
19
Input_Task_Mapping
• Externalmappingoption– Input_Task_Mapping =externalsample-input/mapping.sh– catsample-input/mapping.sh
o echoStage_1_Output_0_1 Stage_1_Output_0_2o echoStage_1_Output_0_1 Stage_1_Output_0_3o echoStage_1_Output_0_1 Stage_1_Output_0_4o echoStage_1_Output_0_2 Stage_1_Output_0_3o echoStage_1_Output_0_2 Stage_1_Output_0_4o echoStage_1_Output_0_3 Stage_1_Output_0_4
– Theith linemapstotheith task
https://github.com/applicationskeleton/Skeleton/blob/master/src/sample-input/external-mapper.inputApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
20
ASingleStageIterativeApplication
Num_Stage =1
Stage_Name =Stage_1Task_Type =serialNum_Tasks =4Task_Length =uniform10Num_Processes =1Read_Buffer =65536Write_Buffer =65536Input_Files_Each_Task =1Input_1.Source=filesystemInput_1.Size=uniform1048576
Output_Files_Each_Task =1Output_1.Size=uniform1048576
Interleave_Option =0Iteration_Num =3Iteration_Stages =Stage_1Iteration_Substitute =Stage_1.Input_1,Stage_1.Output_1
https://github.com/applicationskeleton/Skeleton/blob/master/src/sample-input/single-stage-iterative.input
Stage_1.Input_1andStage_1.Output_1 shouldhaveIDENTICALnumberoffiles
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
21
AMultiStageIterativeApplication
https://github.com/applicationskeleton/Skeleton/blob/master/src/sample-input/multiple-stage-iterative.input
Stage_Name =Stage_3Task_Type =serialNum_Tasks =6Task_Length =uniform32Num_Processes =1Read_Buffer =65536Write_Buffer =65536Input_Files_Each_Task =1Input_1.Source=Stage_2.Output_1
Output_Files_Each_Task =1Output_1.Size=uniform1048576
Interleave_Option =0Iteration_Num =3Iteration_Stages =Stage_3,Stage_4Iteration_Substitute =Stage_3.Input_1,
Stage_4.Output_1
Stage_Name =Stage_4Task_Type =serialNum_Tasks =6Task_Length =uniform32Num_Processes =1Read_Buffer =65536Write_Buffer =65536Input_Files_Each_Task =1Input_1.Source=Stage_3.Output_1
Output_Files_Each_Task =1Output_1.Size=uniform1048576
Interleave_Option =0
Stage_3.Input_1andStage_4.Output_1 shouldhaveIDENTICALnumberoffiles
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
22
DeterminingSkeletonParametersManually
• Steps– PlaceI/OfilesonRAMdisk,useUnixtimecommandtomeasureruntime
– UseUnixstrace commandtofindnumberofreadsandwrites,totaldatareadandwritten
– AlignI/Ocallsinsequenceorder1 todetermineI/Oconcurrency,I/Obuffersize,andinterleavingoption
1Z.Zhang,D.S.Katz,M.Wilde,J.Wozniak,I.Foster.MTCEnvelope:DefiningtheCapability ofLargeScaleComputers intheContextofParallelScriptingApplications, Proceedings of22ndInternational ACMSymposiumonHigh-PerformanceParallelandDistributedComputing(HPDC'13),2013
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
23
DeterminingSkeletonParametersAutomatically
• Wraptaskinaprofiler(usesLinuxperf,hardwareandkernelcounters,systemtools)– Measureresourceconsumptionovertimeasprofile:timedsequenceofCPUcycles,memoryanddiskI/Ooperations
– DependenciesbetweencomputeandI/Oimplicitlycapturedinprofileinorderofthesequence
• Profilesaresystemindependent– Assumingcomparableoptimizationatsoftwareandhardwarelevels
• Profilemetricsaresuitableasinputforemulation
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
24
SkeletonAppsvs.RealApps
• Applications:– Case1:a6x6degreeimagemosaicinMontage– Case2:thefirst256queriesofNRxNR testinBLAST– Case3:partialseismicstudyofCyberShakepostprocessingonsiteTest
• Platformconfiguration:– 64computenodesonIBMBlueGene/P– TasksarelaunchedwithAMFORA[1]– EachtaskstagesinputfilefromGPFS,executethetask,thenwritestheoutputfilestoGPFS
[1]ZhaoZhang,DanielS.Katz,TimothyG.Armstrong,JustinM.Wozniak,andIanT.Foster."Parallelizingtheexecutionofsequentialscripts.”ProceedingsoftheInternationalConferenceforHighPerformanceComputing,Networking,StorageandAnalysis(SC13).2013.
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
25
MontageStatistics
#Tasks
#Inputs
#Outputs
Input(MB)
Output(MB)
SkeletonTask Length(uniform)
InterleavingOption
Error inStage
mProject 1319 1319 2594 2800 10400 11.6 0 -0.2%
mImgtbl 1 1297 1 5200 0.8 30.1 0 -2.1%
mOverlaps 1 1 1 0.8 0.4 9.1 0 -0.2%
mDiffFit 3883 7766 7766 31000 487 1.8 0 -3.3%
mConcatFit 1 3883 1 1.1 4.3 2.1 0 -1.5%
mBgModel 1 2 1 4.5 0.07 288 0 0.03%
mBackground 1297 1297 1297 5200 5200 0.4 0 -1.6%
mAdd 1 1297 1 5200 7400 519 0 -0.9%
Total -1.3%
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
26
BLASTStatistics
#Tasks
#Inputs
#Outputs
Input(MB)
Output(MB)
SkeletonTask Length
InterleavingOption
Error inStage
split 1 1 64 3800 3800 0 3 -1.9%
formatdb 64 64 192 3800 4400 uniform42 3 -0.6%
blastp 1024 4096 1024 70402 966 normal[109.2,14.9]
3 1.6%
merge 16 1024 16 966 867 normal[4.4,4.1]
3 1.1%
Total 1.4%
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
27
CyberShake PostProcessing Statistics
#Tasks
#Inputs
#Outputs
Input(MB)
Output(MB)
SkeletonTask Length
InterleavingOption
Error inStage
Extract 128 130 256 5400 11000 uniform6.39
0 2.6%
Seis 4096 4352 4096 11000 96 normal[26.9,13.3]
0 2.4%
PeakGM 4096 4096 4096 96 1.4 uniform0.23
0 2.3%
Total 2.4%
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
28
UsingApplicationSkeletons
• DataCaching– Comparingsharedfilesystem(PVFS)andin-memoryfilesystem(AMFORA)performanceformProjectPP
– Using64n1-highmem-2instancesonGoogleComputeEngine(GCE)
PVFS AMFORA Improvement
mProjectPP-real 285.2seconds 100.9seconds 63.0%
mProjectPP-skeleton 273.7seconds 101.3seconds 64.6%
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
29
UsingApplicationSkeletons
• TaskScheduling– Data-awareschedulingvs.FIFO– Using16n1-highmem-2GCEinstances
• mProjectPP-realhas0.7%improvement• mProjectPP-skeletonhas1.6%improvement
• Usedskeletonstoshow:5xlargerinputfilesize=>16.4%time-to-solutionimprovementwithdata-awarescheduling
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
30
UsingApplicationSkeletons
• I/OTuning– Multiplemetadataservervs.Singlemetadataserver– Using16n1-highmem-2instances
• mProjectPP-realshows1.1%improvement• mProjectPP-skeletonshows1.2%improvement
• Useskeletonstoshow:10xshortertasklength=>31.2%improvementformultiplemetadataservers
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
31
RelatedWork
• Simplifyparallelapplicationcode– Extractkernels:NASParallelBenchmarks,BerkeleyDwarfs/Motifs,CORALbenchmarks,etc.
– Simplifynon-kernelpartofapp:Kerbyson (SC12),Worley(SC94),miniapps (e.g.,Mantevo,MADbench)
• Simplifyparallelapplicationsintime– Sodhi (Cluster2008)
• Usesystemtracesinplaceofapplications– Chen(VLDB12),Harter(FAST14),Ouserhout (NSDI15)
• Skeleton-likeapproaches– Skel (forI/O),Tigres (distributedapptemplates),WGL(similartoourwork,butsimpler)
• Otherworkthatusedthetermskeleton• SeeFGCSpaperforcomparisons
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
32
Conclusion• Skeleton toolcancompose skeleton applicationinatop-down
manner: application, stage,task• Skeleton taskabstractionallowsspecificationoftasktype,tasklength,
numberofprocesses, I/Obuffer, I/Oquantity, interleaving option,andfilenumber
• Cancreateeasy-to-access, easy-to-build, easy-to-change, andeasy-to-runbag-of-tasks, (iterative)map-reduce, and(iterative)multi-stageworkflow applications
• Skeletonapplications canbeeasily shared,makingmiddleware andtoolexperiments morereproducible
• Skeletonapplicationshaveperformance closetothatoftherealapplicationswithanoverall errorof-1.3%,1.5%,and2.4%forMontage,BLAST,andCyberShake PostProcessing
• Skeletons canshowtheeffectiveness ofsystemimprovements suchasdatacaching,taskscheduling, I/Otuning
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
33
FutureWork
• Useapplicationtracedatatoproduceskeletonapplications
• Determineawaytorepresentthecomputationalworkinataskthatwhencombinedwithaparticularplatformcangiveanaccurateruntimeforthattask
• Supportconcurrenttasksthatneedtorunatthesametimetoexchangeinformation
• Test ondistributedsystemswherelatencies,particularfileusage,andotherissuesmaybemoreimportantthanontheparallelsystemsandcloudenvironments
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
34
References• Software:
– TheSkeleton toolisopensourceat:https://github.com/applicationskeleton/Skeleton
– v1.2publishedas:D.S.Katz,A.Merzky,M.Turilli,M.Wilde,Z.Zhang2015.ApplicationSkeletonv1.2.,DOI:10.5281/zenodo.13750
– Tryit!Contribute toit!• Papers:
– Z.ZhangandD.S.Katz,"ApplicationSkeletons:EncapsulatingMTCApplicationTaskComputationandI/O,”Proceedings of6thWorkshoponMany-TaskComputingonGridsandSupercomputers (MTAGS), 2013.
– Z.Zhang,D.S.Katz,"UsingApplicationSkeletons toImproveeScienceInfrastructure," Proceedingsof10thIEEEInternational Conference oneScience,2014.DOI:10.1109/eScience.2014.9
– D.S.Katz,A.Merzky,Z.Zhang,S.Jha,"ApplicationSkeletons:ConstructionandUse ineScience," FutureGeneration ComputingSystems,2015.DOI:10.1016/j.future.2015.10.001
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
www.ci.anl.govwww.ci.uchicago.edu
35
Acknowledgements
• ThisworkwassupportedinpartbytheU.S.DepartmentofEnergyundertheASCRawardDE-SC0008617(theAIMESproject)
• IthasbenefitedfromdiscussionswithMatteoTurilli,JonWeissman,andLavanya Ramakrishnan
• ComputingresourceswereprovidedbytheArgonneLeadershipComputingFacility
• WorkbyKatzwassupportedbytheNationalScienceFoundationwhileworkingattheFoundation.Anyopinion,finding,andconclusionsorrecommendationsexpressedinthismaterialarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheNationalScienceFoundation
ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience
top related