Center forSubsurface Sensing & Imaging Systems

Center forSubsurface Sensing & Imaging Systems

HP Visit @ UPRMFebruary 27, 2003HP Visit @ UPRMFebruary 27, 2003

Overview of Research Thrust R3

R3 Fundamental Research Topics

R3A Parallel Processing

• Middleware Tools

• FPGA Acceleration

R3B Solutionware Development

• Image and Sensor Data Databases

• Subsurface Toolboxes

R3 Fundamental Research Topics

R3A Parallel Processing

• Middleware Tools

• FPGA Acceleration

R3B Solutionware Development

• Image and Sensor Data Databases

• Subsurface Toolboxes

David Kaeli, Northeastern UniversityDavid Kaeli, Northeastern University

CenSSIS Barriers Addressed by R3 ProjectsCenSSIS Barriers Addressed by R3 Projects

Lack of Computationally Efficient, Realistic Models Lack of Computationally Efficient, Realistic Models

Barrier 6 Barrier 6 Lack of Rapid Processing and Management of Large Image DatabasesLack of Rapid Processing and Management of Large Image Databases

Barrier 7 Barrier 7 Lack of Validated, Integrated Processing andComputational ToolsLack of Validated, Integrated Processing andComputational Tools

Barrier 4 Barrier 4

Middleware ToolsMiddleware Tools

Parallelization using MPI – a software pathway to exploiting GRID-level resources

Presently exploiting NCSA resources (Mariner Center at BU) and Internet-2 Profile-guided program instrumentation/optimization Developing parallelization strategies for MATLAB Utilizing MPI-2 to address barriers in I/O performance

Impact on SSI applications: Reduced the runtime of a single-body Steepest Descent Fast Multipole

Method (SDFMM) application by 74% on a 32-node Beowulf cluster• Hot-path parallelization• Data restructuring

Reduced the runtime of a Monte Carloscattered light simulation by 98% on a 16-node SGI Origin 2000• Matlab-to-C compliation• Hot-path parallelization

Parallelization using MPI – a software pathway to exploiting GRID-level resources

Presently exploiting NCSA resources (Mariner Center at BU) and Internet-2 Profile-guided program instrumentation/optimization Developing parallelization strategies for MATLAB Utilizing MPI-2 to address barriers in I/O performance

Impact on SSI applications: Reduced the runtime of a single-body Steepest Descent Fast Multipole

Method (SDFMM) application by 74% on a 32-node Beowulf cluster• Hot-path parallelization• Data restructuring

Reduced the runtime of a Monte Carloscattered light simulation by 98% on a 16-node SGI Origin 2000• Matlab-to-C compliation• Hot-path parallelization

Soil

Air

Mine

Scattered Light Simulation Speedup

1

10

100

1000

10000

100000

Ru

nti

me

in s

eco

nd

sOriginal

Matlab-to-C

Hot pathparallelization

Techniques for Parallelizing MATLABTechniques for Parallelizing MATLAB

Manage completely independent MATLAB processes distributed over different processors

Message passing within MATLAB (e.g., MultiMATLAB)

MATLAB calls to parallel libraries (multi-threaded LAPACK, PLAPACK)

Backend compilers can convert MATLAB to C, and automatically inserting MPI calls (e.g., RTExpress)

Multiple MATLAB sessions

A Single MATLAB session

Our Approach for Parallelizing MATLABOur Approach for Parallelizing MATLAB

Convert MATLAB to C using

the MATLAB mcc compiler

Profile the C program to capture

both data flow and control flow

Parallelize the “hot” regions of the

the application using MPI

Convert array structs (generated by mcc)

to pointer-based structs where needed

Main0/2502

Tfqmr1/1604

3/35

Mult11/1602

Mult4/1599

Multfaflone4m523/6Multfaflone4

359/5

Multdifl706/0

0/0

0/0

0/0

18/0

16/0

0/0 0/0

11

1 1

273

273

273

273

273

273

273

54K2.4M

0.6M

FunctionSelf /Children Runtime

Number of Calls

Eliminating I/O Barriers in ParallelSSI ApplicationsEliminating I/O Barriers in ParallelSSI Applications

Many SSI applications tend to be file bound or memory bound (or both)

While we can use MPI to parallelize processing, and use MPI-I/O to perform collective I/O, we still are limited to access to a file on a single physical disk

Our present work looks at parallelizing I/O by partitioning files associated with MPI processes

We attempt to utilize, slower and commodity (IDE) local secondary storage

Many SSI applications tend to be file bound or memory bound (or both)

While we can use MPI to parallelize processing, and use MPI-I/O to perform collective I/O, we still are limited to access to a file on a single physical disk

Our present work looks at parallelizing I/O by partitioning files associated with MPI processes

We attempt to utilize, slower and commodity (IDE) local secondary storage

Eliminating I/O Barriers in ParallelSSI ApplicationsEliminating I/O Barriers in ParallelSSI Applications

0

500

1000

1500

2000

2500

3000

3500

NPB/BT SPEChpc MPI-Tile Perf MandelbrotT

ota

l E

xecu

tio

n T

ime

(sec

on

ds)

MPI-IO

Partitioned

Parallelize application using MPI

Parallelize I/O using MPI collective I/O

Profile I/O access patterns on a per process basis

Use profile to guide partitioning to reduce overall execution time by 27%-82%

Presently targeting both file-bound and memory-bound applications

Some Recent PublicationsSome Recent Publications

“Profile-based Characterization and Tuning Subsurface Sensing Applications”, M. Ashouei, D. Jiang, W. Meleis, D. Kaeli, M. El-Shenawee, E. Mizan, M. and C. Rappaport, Special issue of the SCS Journal, November 2002.

“Parallel Implementation of the Steepest Descent Fast Multipole Method (SDFMM) on a Beowulf Cluster for Subsurface Sensing Application”, D. Jiang, W. Meleis, M. El-Shenawee, E. Mizan, M. Ashouei, and C. Rappaport, IEEE Microwave and Wireless Components Letters, January 2002.

“Electromagnetics Computations Using MPI Parallel Implementation of the Steepest Descent Fast Multipole Method (SDFMM)”, M. El-Shenawee, C. Rappaport, D. Jiang, W. Meleis, and D. Kaeli, Applied Computational Electromagnetics Society Journal, August 2002.

“Profile-Guided I/O Partitioning”, Y. Wang and D. Kaeli, Submitted to ICS’03.

Other techniques being assessed in this work: Globus Toolkit – NSF Middleware Initiative MatlabMPI – MIT LL RTExpress - Parallelizing compiler for Matlab Sparse matrix representations

“Profile-based Characterization and Tuning Subsurface Sensing Applications”, M. Ashouei, D. Jiang, W. Meleis, D. Kaeli, M. El-Shenawee, E. Mizan, M. and C. Rappaport, Special issue of the SCS Journal, November 2002.

“Parallel Implementation of the Steepest Descent Fast Multipole Method (SDFMM) on a Beowulf Cluster for Subsurface Sensing Application”, D. Jiang, W. Meleis, M. El-Shenawee, E. Mizan, M. Ashouei, and C. Rappaport, IEEE Microwave and Wireless Components Letters, January 2002.

“Electromagnetics Computations Using MPI Parallel Implementation of the Steepest Descent Fast Multipole Method (SDFMM)”, M. El-Shenawee, C. Rappaport, D. Jiang, W. Meleis, and D. Kaeli, Applied Computational Electromagnetics Society Journal, August 2002.

“Profile-Guided I/O Partitioning”, Y. Wang and D. Kaeli, Submitted to ICS’03.

Other techniques being assessed in this work: Globus Toolkit – NSF Middleware Initiative MatlabMPI – MIT LL RTExpress - Parallelizing compiler for Matlab Sparse matrix representations

Embedded Processing for Accelerated Vessel Enhancement in Retinal ImagesEmbedded Processing for Accelerated Vessel Enhancement in Retinal Images

The Application Processing live video of the patient retina during laser retinal surgery Highlighting the vascular structure helps the surgeon avoid damage Retinal vascular tracing; detection of blood vessels in images of the retina Utilizes a k-means matched filter to find blood vessels and traces out

structure Embedded into a Xilinx Virtex-based Firebird Computing Engine Data are 1024x1024 images can be processed at 30 frames/sec

Embedded processing, combined with parallel processing, provides a powerful heterogeneous computing platform for discovery

Presently being applied to identify vascular structures present in sub-skin tumors @ MGH

NU/RPI Research Team• Miriam Leeser – NU-ECE• Badri Roysam – RPI-ECSE

CenSSIS SolutionwareCenSSIS Solutionware

Toolbox Development Support the development of CenSSIS Solutionware that demonstrates

our “Diverse Problems – Similar Solutions” model Delivered a software-engineered Multi-View Tomography Toolbox,

developed in OOMATLAB Developing three new CenSSIS Toolboxes

Registration – RPI/WHOI Hyperspectral Imaging – UPRM 3-D Modeling - NEU

Establish software development and testing standards for CenSSIS

Image and Sensor Data Database Develop an web-accessible image database for CenSSIS that enables

efficient searching and querying of images, metadata and image content

Develop image feature tagging capabilities

Toolbox Development Support the development of CenSSIS Solutionware that demonstrates

our “Diverse Problems – Similar Solutions” model Delivered a software-engineered Multi-View Tomography Toolbox,

developed in OOMATLAB Developing three new CenSSIS Toolboxes

Registration – RPI/WHOI Hyperspectral Imaging – UPRM 3-D Modeling - NEU

Establish software development and testing standards for CenSSIS

Image and Sensor Data Database Develop an web-accessible image database for CenSSIS that enables

efficient searching and querying of images, metadata and image content

Develop image feature tagging capabilities

Matlab 6

CenSSIS Toolbox Development

CenSSIS Applications written in MATLAB, C, C++, Java, MPI, VHDL and Verilog

MVT MSD LPM Modeling

Identify classes of CenSSIS algorithms that can be more generally specified to extend applications to multiple research domains

Utilize Software Engineering principles to produce reusable and extensible software toolboxes• Object-oriented design principles used• Exploratory Software Development Model• Library and revision control to support infrastructure

Identify classes of CenSSIS algorithms that can be more generally specified to extend applications to multiple research domains

Utilize Software Engineering principles to produce reusable and extensible software toolboxes• Object-oriented design principles used• Exploratory Software Development Model• Library and revision control to support infrastructure

Inverse Modeling

Framework

OO-MATLAB

The CenSSIS Multi-View Tomography (MVT) ToolboxThe CenSSIS Multi-View Tomography (MVT) Toolbox

Matlab 6

Electrical Impedance

Tomography

Electrical Resistance

Tomography

Diffuse Optical Tomography

MVT

Frequency-Domain

Diffusion Library

OO-C++ and IML++

www.censsis.neu.edu/MVT/mvt.html

Current Toolbox DevelopmentCurrent Toolbox Development

Development of multi-language toolboxes Performance tuning Presently working on three additional

toolbox effort Developing an MPI-2 version of the MVT

Toolbox

Development of multi-language toolboxes Performance tuning Presently working on three additional

toolbox effort Developing an MPI-2 version of the MVT

Toolbox

Matlab 6MSD LPM Modeling

Deliver an web-accessible database forCenSSIS that enables efficient searching and querying of images, sensor data, metadata and image content

Database Characteristics:

• Relational complex queries (Oracle8i)

• Data security, reliability and layered user privileges

• Efficient search and query of image content and metadata

• Content-based image tagging using XML adopting MPEG-7 standards

• Easy upload and registration of user images and metadata

• Indexing algorithms (2D, 3D and 4D) and partitioning of

the database for better performance

• Explore object relational technology to handle collections

CenSSIS Image Database SystemCenSSIS Image Database System

12

34

nucleus

<Image><!-- General Cell Infomation --> <CellInformation> <ID> 9 </ID> <ClinomicsID> 931175495 </ClinomicsID> <DOB> 2/7/30 </DOB> <SEX> F </SEX> <COLL_DATE> 11/2/1993 </COLL_DATE> <Primary_site> Breast </Primary_site> <INITIAL> II </INITIAL> <GRADE> POORLY DIFFERENTIATED </GRADE> <HISTOLOGY> UNKNOWN </HISTOLOGY> <PRIM_SITE2> NONE </PRIM_SITE2> <PRIM_DATE> 4/1/1992 </PRIM_DATE> <MET1_SITE> NONE </MET1_SITE> <MET1_DATE> NONE </MET1_DATE> <TUBE_TYPE> p </TUBE_TYPE> </CellInformation>

<StillRegion id="IMG0001"> <MediaProfile> <MediaFormat> <FileFormat>jpeg</FileFormat> <System>PAL</System> <Medium>CD</Medium> <Color>color</Color> <FileSize>332.228</FileSize> </MediaFormat> . . . . . . . .

</MediaCoding> <MediaInstance> <Identifier IdOrganization='Clinomics' IdName= 'BreastCancerCell'>BreastCancerCell// image0001 </Identifier> <Locator> <MediaURL>file://D:/Breast/cells/imag0001.jpg</MediaURL> </Locator> </MediaInstance> </MediaProfile>

<StructuredAnnotation> <Who>Patient239</Who> <whatObject>Human primary breast tumor cells</whatObject> <WhatAction> growing in a NASA Bioreactor </WhatAction> <where> St. Mary’s Hospital </where> <When> 09/25/2000 </When> <why> Investigate tumor cells behaviour on microcarrier beads </why> <TextAnnotation xml:lang='en-us'> Higher magnification of view illustrating breast cancer cells with intercellular boundaries on bead surface </TextAnnotation> </StructuredAnnotation>

</StillRegion></Image>

General ImageInfo

Image SourceInfo

Image FormatInfo

Image FileLocation Info

Image DonorInfo

Image FeatureInfo

CenSSIS R3 Research Thrust SummaryCenSSIS R3 Research Thrust Summary

Providing both SSI-related computing research expertise and supporting CenSSIS infrastructure needs

Addressing key research barriers in computational efficiency, embedded computing and image/sensor data management

Developing GRID-level tools and an services • I/O Partitioning Algorithms• Subsurface Toolboxes• Image/Sensor Data Databases

Providing both SSI-related computing research expertise and supporting CenSSIS infrastructure needs

Addressing key research barriers in computational efficiency, embedded computing and image/sensor data management

Developing GRID-level tools and an services • I/O Partitioning Algorithms• Subsurface Toolboxes• Image/Sensor Data Databases