parallel and distributed computing for neuroinformatics allen d. malony university of oregon...

Parallel and Distributed Computing for Neuroinformatics

Allen D. Malony

University of Oregon

ProfessorDepartment of Computerand Information Science

DirectorNeuroInformatics Center

Computational Science Institute

University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics

Outline Neuroinformatics

Dynamic brain analysis problem NeuroInformatics Center (NIC) at UO

Neuroinformatics research at the NIC Dense-array EEG analysis (APECS, HiPerSat, Mc) Brain image segmentation Computational head modeling Ontologies and tool integration (NEMO, GEMINI) Parallel and distributed computing emphasis

ICONIC Grid HPC system at UO Cerebral Data Systems Oregon E-science


Neuroscience and Neuroinformatics

Application of computer and information science to the understanding of brain organization and function

Integration of information across many levels Physical and functional Gene to behavior Microscopic to macroscopic scales

Challenges in brain observation and modeling Structure and organization (imaging) Operational and functional dynamics (temporal/spatial) Physical, functional, and cognitive operation (models)

How to create and maintain of integrated views of the brain for both scientific and clinical purposes?


Human Brain Dynamics Analysis Problem

Understand functional operation of the human cortex Dynamic cortex activation Link to sensory/motor and cognitive activities Multiple experimental paradigms and methods Multiple research, clinical, and medical domains

Need for coupled/integrated modeling and analysis Multi-modal observation (electromagnetic, MR, optical) Physical brain models and theoretical cognitive models

Need for robust tools Complex analysis of large multi-model data Reasoning and interpretation of brain behavior

Problem solving environment for brain analysis


Neuroimaging Technology

Alternative techniques for analyzing brain dynamics Blood flow neuroimaging (PET, fMRI)

Good spatial resolution allows functional brain mapping

Temporal limitations to tracking of dynamic activities Electromagnetic measures (EEG/ERP, MEG)

msec temporal resolution can distinguish fast neurological components

Spatial resolution sub-optimal requires a mapping to cortical sources

Need integrated neuroimaging technology (How?) Achieve both good spatial and temporal resolution


Integrated Dynamic Brain Analysis

IndividualBrain Analysis

Structural /FunctionalMRI/PET

DenseArray EEG /

MEG

ConstraintAnalysis

Head Analysis

Source Analysis

Signal Analysis

Response Analysis

Experimentsubject

temporaldynamics

neuralconstraints

CorticalActivity Model

ComponentResponse Model

spatial patternrecognition

temporal patternrecognition

Cortical ActivityKnowledge Base

Component ResponseKnowledge Base

good spatialpoor temporal

poor spatialgood temporal neuroimaging

integration


Experimental Methodology and Tool Integration

source localization constrained to cortical surface

processed EEG

BrainVoyager

BESA

CT / MRI

EEG segmentedtissues

16x256bits permillisec(30MB/m)

mesh generation

EMSEInterpolator 3D

NetStation


NeuroInformatics Center (NIC) at UO Application of computational science methods to

human neuroscience problems Tools to help understand dynamic brain function Tools to help diagnosis brain-related disorders HPC simulation, large-scale data analysis, visualization

Integration of neuroimaging methods and technology Need for coupled modeling (EEG/ERP, MR analysis) Apply advanced statistical signal analysis (PCA, ICA) Develop computational brain models (FDM, FEM) Build source localization models (dipole, linear inverse) Optimize temporal and spatial resolution

Internet-based capabilities for brain analysis services, data archiving, and data mining


NIC Organization

Allen D. Malony, Director Don M. Tucker, Associate Director Sergei Turovets, Computational Physicist Bob Frank, Mathematician Dan Keith, Software Engineer (distributed systems grid) Chris Hoge, Software Engineer (computational) Ryan Martin / Brad Davidson, Systems administrators Gwen Frishkoff, Research Associate, Univ. Pittsburgh Kai Li, Ph.D. student (brain segmentation) Adnan Salman, Ph.D. student (computational modeling) Performance Research Lab (my other hat)


Brainwave Research 101

Electroencephalogram (EEG)

Electrodes (sensors) measure uV EEG time series analysis

Event-related potentials (ERP) link brain activity to sensory–motor, cognitive functions statistical average to increase signal-to-noise ratio (SNR)

Signal cleaning and component decomposition Localize (map) to neural sources


Electrical Geodesics Inc. (EGI)

EGI Geodesics Sensor Net Dense-array sensor technology

64/128/256 channels 256-channel GSN

AgCl plastic electrodes

Net Station Advanced EEG/ERP data analysis

and visualization Stereotactic EEG sensor registration State of the art technology Research / clinical products EGI/CDS medical services


Epilepsy

Epilepsy affects more than 5 million people yearly U.S., Europe, and Japan

EEG in epilepsy diagnosis Childhood and juvenile absence Idiopathic (genetic) Distinguish different types

EEG in presurgical planning Localize seizure onset Fast, safe, inexpensive Dense array improves accuracy Requires good source modeling


EEG Time Series - Progression of Absence Seizure

First full spike–wave

Pre-spike“buzz’


Topographic Waveforms – First Full Spike-Wave

350ms interval


Topographic Mapping of Spike-Wave Progression

Palette scaled for wave-and-spike interval (~350ms)-130 uV (dark blue) 75 uV (dark red)

1 millisecond temporal resolution is required Spatial density (256) to capture shifts in topography


Dense-array EEG signal analysis and decomposition Artifact cleaning and component analysis

Automatic brain image segmentation Brain tissue identification Cortex extraction

Computational head modeling Tissue conductivity estimation Source localization

Statistical analysis to detect brain states Discriminant analysis Pattern recognition

Electromagnetic databases and ontologies

Neuroinformatic Challenges


Applying ICA for EEG Blink Removal

Component analysis is used to separate EEG signals Independent component analysis (ICA)

Blinks are a major source of noise in EEG data Blink signals are separable from cognitive responses

Raw EEGFormatting

EEGpreprocessing

Event infoTime markersBlink events

Bad channel removalBaseline correction…

ICA Analysis

ICA algorithms - Infomax - FastICA - …ICA components

Identify blinksand remove

Blink templatesReconstitute EEG w/out blink data

ERP Analysis


Independent Component Analysis

Mixedsinusoids- raw EEG

Originalsinusoids

ICA

EEG waveform


Tool for EEG Data Decomposition (APECS)

Automated Protocol for Electromagnetic Component Separation (APECS)

Motivation EEG data cleaning (increases signal-to-noise (SNR)) EEG component separation (addresses superposition) Data preprocessing prior to source localization

Distinctive Features Implements different decomposition methods Multiple metrics for component classification Quantitative and qualitative criteria for evaluation


APECS Evaluation: Qualitative Criteria

Bad GoodOriginal

Spatio-temporal blink profiles

Blink-freebaseline


APECS Evaluation: Quantitative Criteria

Covariance between “baseline” (blink-free) and ICA-filtered data: Infomax and FastICA.

Infomax gives consistently better results. FastICA results are more variable.

ICA decompositions most successful when only one spatial projector is strongly correlated with blink “template” (spatial filter).


Parallelization of Component Analysis Algorithms

Dense-array EEG increases analysis complexity Long time measurements require more processing ICA algorithms are computationally challenging

Processing time and memory requirements Increase performance through ICA parallelization High-Performance Signal Analysis Toolkit (HiPerSAT)

Matlab ICA algorithms implemented in C++ Infomax: Matlab (runica.m) parallel (OpenMP) FastICA: Matlab (fastica.m) parallel (MPI)

Validate results with Matlab standard algorithms Evaluate accuracy and compare speedup


HiPerSAT Parallel Infomax

Requires multi-threading Over 3 times

faster thanMatlab 3-fold increase

on 4 processors Speedup falls after

4 processors Limits on parallelization

of loop matrix operations May be able to improve

with larger blocking


HiPerSAT Parallel FastICA

Linear speedup Over 130 times faster

than Matlab 8-fold increase

on 32 processors Performance

may allow finerprocessing

Tradeoff improvedaccuracy versus morecomplex processing andexecution time


Matlab Tool Integration

Many neuroimaging tools are based on Matlab EEGLAB (UC San Diego) BrainStorm (USC / Los Alamos National Lab) SPM (University College of London)

Matlab is mostly a closed computational environment EEG/MEG analysis can overwhelm Matlab

Limited to workstation processing resources Memory requirements high due to Matlab workspace

Desire to use Matlab as a client in distributed systems Matlab is not multi-threaded Complicates building concurrent for external interfaces


Mc: Matlab Concurrent

Matlab is built on top of a JVM Use for GUI and graphics

We can leverage JVM for concurrency and interaction How do we create concurrent tasks in Matlab?

Matlab programming semantics issue Create task abstraction Provide Matlab package for constructing tasks

Concurrent tasks interface with runtime layer Client task manager runs tasks on servers and monitors Server task executor schedules tasks on resources

Voila! Mc


Mc System Architecture


Mc Task Manager

Single background thread running in the JVM Coordinate several components:

File transfer Ganymed SSH2 implementation for Java

Matlab interactionMatlabControl class

Scripting with Jython dynamic access to objects in the JVM Matlab m-scripting through MatlabControl class

waits for free cycles when execution Matlab code

Responds to user or timer events Manages execution of tasks and status reporting


Task Manager Workflow


HiPerSAT, APECS, and Mc

Demonstrated APECS running HiPerSAT tasks 20 simultaneous HiPerSAT servers on a cluster Compare with sequential EEGLAB processing

ICA processing on 20 1-GB EEG files Performance speedup

5x for C++ optimization and 2-way parallelism 10x speedup for 20 concurrent HiPerSAT tasks

Also demonstrate for multiple Matlab clients Recently applied to EEG processing for imagery

analysis 100+ simultaneous tasks across 8 different platforms


Papers K. Glass, G. Frishkoff, R. Frank, C. Davey, J. Dien, A. Malony,

A Framework for Evaluating ICA Methods of Artifact Removal from Multichannel EEG, ICA Conference, Grenada, Spain, 2004.

R. Frank and G. Frishkoff, APECS: A Framework for Implementation and Evaluation of Blink Extraction from Multichannel EEG, Journal of Clinical Neurophysiology, to appear, 2006.

D. Keith, C. Hoge, A. Malony, and R. Frank, “Parallel ICA Methods for EEG Neuroimaging,” International Parallel and Distributed Processing Symposium (IPDPS 2006), May 2006.

C. Hoge, D. Keith, and A. Malony, “Client-side Task Support in Matlab for Concurrent Distributed Execution,” Austrian-Hungarian Workshop on Distributed and Parallel Systems (DAPSYS), September 2006.


Spatial and temporal dynamicsare important to observetogether

Linked cortical networks Fronto-thalamic circuit (executive control) Limbic circuit (episodic memory)

Problem of Superposition How many sources? Where are they located?

Can only infer locations using“scalp space” information

Animated Topography of Spike–Wave Dynamics

QuickTime™ and aAnimation decompressor

are needed to see this picture.


Addressing Superposition: Brain Electrical Fields

Brain electrical fields are dipolar Volume conduction

Depth and location indeterminacy Highly resistive skull (CSF: skull est. from 1:40 to 1:80) Left-hemisphere scalp field may be generated by a

right-hemisphere source Multiple sources superposition

Radial source Tangential sources one and two sources varying

depths


Source Localization

Mapping of scalp potentials to cortical generators Signal decomposition (addressing superposition) Anatomical source modeling (localization)

Source modeling Anatomical Constraints

Accurate head model and physics Computational head model formulation

Mathematical Constraints Criteria (e.g., “smoothness”) to constrain solution

Current solutions limited by Simplistic geometry Assumptions of conductivities


Brain Sources of Epileptic Seizure

Single time point source solution Need to identify sources for each msec time sample Visualize dynamics in “source space”


Dipole Sources in the Cortex

Scalp EEG is generated in the cortex

Interested in dipole location, orientation, and magnitude Cortical sheet gives

possible dipole locations Orientation is normal to

cortical surface Need to capture convoluted

geometry in 3D mesh From segmented MRI/CT

Linear superposition


Advanced Image Segmentation

Native MR gives high gray-to-white matter contrast

Image analysis techniques Edge detection, edge

merger, region growing Level set methods and

hybrid methods Knowledge-based

After segmentation, color contrasts tissue type

Registered segmented MRI


Network Flow Based Skull Stripping

Graph construction with LoG Nodes, edges, edge weights,

source terminals, sink terminals Identify non-brain terminals

Scalp, eyeballs, orbits Fourth ventricle, pons

Identify brain terminals Knowledge-based terminal

detection Min-cut (Max-flow)

Separate source from sink Sum of weights is minimum


Cortical Surface Extraction Pipeline

Initial segmentation of white matter Scale-space based analysis of LoG Incorporation of spatial knowledge

Cortical surface topology correction Minimize cortical surface are increase Based on network flow theory Based on Cauchy-Crofton formula

Cortical surface geometry refinement White matter shape knowledge

radius function constraints sheet constraints

Medial surface (skeleton) manipulation


Topology Correction in Cortex Extraction

Before topology correction

After topology correction

Extracted Cortex


Building Computational Brain Models

MRI segmentation of brain tissues Conductivity model

Measure head tissue conductivity Electrical impedance tomography

small currents are injectedbetween electrode pair

resulting potential measuredat remaining electrodes

Finite element forward solution Source inverse modeling

Explicit and implicit methods Bayesian methodology


Conductivity Modeling

Governing Equations ICS/BCS

Discretization

System of Algebraic Equations

Equation (Matrix) Solver

Approximate Solution

Continuous Solutions

Finite-DifferenceFinite-Element

Boundary-ElementFinite-Volume

Spectral

Discrete Nodal Values

TridiagonalADISOR

Gauss-SeidelGaussian elimination

(x,y,z,t)J (x,y,z,t)B (x,y,z,t)


Conductivity Optimization and Parallelization

Design as a conductivity search problem Master launches new inverse problems with guesses Inverse solvers run iterative forward calculations Compare solutions with measured results

Parallelization approach Forward solver

Alterative Direction Implicit (ADI) method Optimization search

Simplex or simulated annealing (Monte Carlo) Hybrid


Alternating Direction Implicit (ADI) Method

Finite difference method C++ with OpenMP for parallelization LAPACK for matrix operations


Conductivity Search Architecture


Conductivity Optimization Dynamics

Attempting todetermine tissueconductivity values Scalp Skull Brain CSF

Simplex algorithm Values converge

to minimal error Increase number of features in future work


Multi-Cluster Search Dynamics


Papers K. Li, A. Malony, and D. Tucker, “Automatic Brain MR Image

Segmentation with Relative Thresholding and Morphological Image Analysis,” International Conference on Computer Vision Theory and Applications (VISAPP), Setúbal, Portugal, February 2006.

K. Ki, A. Malony, and D. Tucker, “A Multiscale Morphological Approach to Topology Correction of Cortical Surfaces,” International Workshop on Medical Imaging and Augmented Reality (MIAR 2006), Sanghai, China, August 2006.

A. Salman, S. Turovets, A. Malony, J. Eriksen, and D. Tucker, “Computational Modeling of Human Head Conductivity,” International Conference on Computational Science (ICCS), best paper, May 2005.

A. Salman, S. Turovets, A. Malony, V. Vasilov, “Multi-Cluster, Mixed-Mode Computational Modeling of Human Head Conductivity,” International Workshop on OpenMP (IWOMP), June 2005.

S. Turovets, A. Salman, A. Malony, P. Poolman, C. Davey, D. Tucker, “Anatomically Constrained Conductivity Estimation of the Human Head in Vivo: Computational Procedure and Preliminary Experiments,” Electrical Impedance Tomography (EIT), July 2006.


Neural ElectroMagnetic Ontology (NEMO) How can brain electromagnetic (EEG and MEG) data be

compared and integrated across experiments and laboratories? Need a system for representation, storage, mining, dissemination Need standardization of methods for measure generation

Identification and labeling of components Patterns of interest

General agreement on criteria for component identification Patterns can be hard to identify Variability in techniques for measure generation

NEMO will address issue by providing Spatial and temporal ontology database Use for large-scale data representation, mining, and meta-analysis Components in average EEG and MEG (ERPs)


Electromagnetic Data Spaces and Representation


NEMO Architecture Composed of three modules

Database mining Inference engine Query (user) interface

Methods for measure generation Spatial ontologies and temporal ontologies Cognitive functional mapping

User interactions Query formulation Mapping-rule definitions

Scalable integration system Online repository for storing metadata

Spatio-temporal ontologies, database schema, mappings


NEMO System


ICONIC Grid

SMPServerIBM p655

GraphicsSMP

SGI Prism

Shared Storage System

Gbit Campus Backbone

NIC CIS CIS

Internet 2

SharedMemory

IBM p690

DistributedMemory

IBM JS20

CNI

DistributedMemory

Dell Pentium Xeon

NIC4x8 16 16 2x8 2x16

graphics workstations interactive, immersive viz other campus clusters

40 TerabytesTape

Backup112 totalprocessors


Computational Integrated Neuroimaging System

… …

raw

storageresources

virtualservices

compute resources


CIS Computational and Visualization Laboratory Support interdisciplinary computer science

Informatics Computational science

Resource development Phase 1 (complete)

NSF MRI grant ($1M) ICONIC Grid

Phase II (underway) Visualization Lab ($100K)

rear projection» 3D stereo and 2x2 tiled

3x4 tiled 24” LCD display

Phase III …


GEMINI Project

Grid-enabled Methods for Integrated NeuroImaging Dynamic neuroimaging algorithms and visualization Grid-based integration (processing and data sharing) High-end tool integration and environments Neuroinformatics data ontologies GEMINI is

A distributable set of programs and configuration files A set of web services and WSRF-accessible resources An internet-accessible service executing on systems Based on open technologies


GEMINI Model

Zones GEMINI resource

servers Tasks

Job representation Agents

Execution context Build on endpoint

references Supports workflow


GEMINI Architecture


Leveraging Internet, HPC, and Grid Computing

Telemedicine imaging and neurology Distributed EEG and MRI measurement and analysis Neurological medical services Shared brain data repositories Remote and rural imaging capabilities

Build on emerging web services and grid technology Leverage HPC compute and data centers Create institutional and industry partnerships

Electrical Geodesics, Inc. Cerebral Data Systems (UO partnership with EGI) Other industrial partnerships


Cerebral Data Systems Partnership between EGI and University of Oregon Develop and market neuroinformatics services

Neurological medical data transfer, storage, and analysis High-performance, expert EEG and MR analysis Telemedicine and distributed collaboration Shared brain repositories

Target markets Research and clinical Epilepsy diagnosis and pre-surgical planning MR image segmentation

Technology integration Internet, computional grids, and HPC


CDS Computational Server and Imaging Clients


Oregon E-Science Grid

Region 4

Region 1Region 2

Region 3

Region 5

Internet 2 /National LambdaRailRegional networks

Institutional partnersRegional points of presence

PSI

parallel and distributed computing for neuroinformatics allen d. malony university of oregon...

Documents

brain observation

dynamic brain functiontools

opticalphysical brain

cognitive operation

uoneuroinformatics research

fmrigood spatial

methodsmultiple research

clinical purposes