Extending Petascale I/O with Data Services

Hasan AbbasiKarsten SchwanMatthew Wolf

Jay LofsteadScott Klasky (ORNL)

Motivation

• I/O bottleneck• Petascale data sizes• Data overload• Faster solution

Observations

• Fast Extraction• Flexibility in where we execute operations• Managed output to data consumer• Flexible resource utilization

Compute Area Using ADIOS for flexibility in

choosing output method Data is serialized using FFS COD provides a processing hook

within the compute application SmartTap generates the output

buffer through a user defined function

DataTap moves the data to the staging area

Staging Area

Additional resources for buffering before storage Simple operations like aggregation Complex analysis and compression operations Domain specific services

Combination of extraction, processing and storage Placement to optimize performance

Runtime Overhead

2k 16k 64k 112k0

10

20

30

40

50

60

70

80

90

POSIXDataTap

Number of Nodes

Runti

me

Ove

rhea

d (%

)