Demonstration Study for Applying AVED to Demonstration Study for Applying AVED to Still Images from Station M Still Images from Station M

MBARI Internal Project 900905

Update, next steps, workflow overviewUpdate, next steps, workflow overview

Danelle ClineDanelle Cline

MBARI May 13rd, 2009MBARI May 13rd, 2009 22

AgendaAgenda UpdatesUpdates

Next steps discussionNext steps discussion

Data workflow overview Data workflow overview

Action itemsAction items

MBARI May 13rd, 2009MBARI May 13rd, 2009 33

UpdatesUpdates Modifications completed to core AVED software for still frame Modifications completed to core AVED software for still frame

processingprocessing• Required changes in segmentation and saliency modelRequired changes in segmentation and saliency model• Added customized cmd-line option for processing time-lapse imagesAdded customized cmd-line option for processing time-lapse images

--mbari-timelapse-stills--mbari-timelapse-stills

Examples StaM4211PsychroSeries/WhiteEchSeries:Examples StaM4211PsychroSeries/WhiteEchSeries:

Working on getting more compute resources to process a Working on getting more compute resources to process a demonstration data setdemonstration data set

QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture. QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture.

MBARI May 13rd, 2009MBARI May 13rd, 2009 44

Next StepsNext Steps Now that we know the types of possible Now that we know the types of possible

detections…detections…

Decide use case for demonstrationDecide use case for demonstration• This will drive what and how much data to This will drive what and how much data to

process, and what kind of training libraries to process, and what kind of training libraries to create for image classification.create for image classification.

Example use cases:Example use cases:• Process a collection looking for temporal Process a collection looking for temporal

changes in fauna and structures on the changes in fauna and structures on the seafloor, focusing on the sessile fauna seafloor, focusing on the sessile fauna polychaete(Paradiopatra) burrows and glass polychaete(Paradiopatra) burrows and glass sponge (Hyalinacea) stalks .sponge (Hyalinacea) stalks .

• Process a collection around major El Niño La Niña events between 1997, searching for a few select animals previously analyzed by hand to ground truth against AVED

• Other ideas?  Other ideas?  

Example AVED events from Sta4211 image set

MBARI May 13rd, 2009MBARI May 13rd, 2009 55

Data Workflow Data Workflow

MBARI May 13rd, 2009MBARI May 13rd, 2009 66

AVED ProcessAVED Process

Image Preprocessing

•Scale and reformat•Histogram equalize•Mask equipment,

•time code overlays, •black bars, etc.

Post-processing

Segmentation and

Tracking

Detectionevents.XML

Every frame

Every frame

MBARI May 13rd, 2009MBARI May 13rd, 2009 77

AVED EditorAVED Editor

• Optional, but can useful for removing “false Optional, but can useful for removing “false detections”, or combining events detections”, or combining events

MBARI May 13rd, 2009MBARI May 13rd, 2009 88

ClassificationClassification Matlab program developed by Perona student Marc’Aurelio Matlab program developed by Perona student Marc’Aurelio

Ranzato at Caltech and Universita’ degli studi di PadovaRanzato at Caltech and Universita’ degli studi di Padova• Developed to analyze biological particlesDeveloped to analyze biological particles• Based on extracting features using Based on extracting features using

local jets (Schmid et al. 1997) (local jets (Schmid et al. 1997) (convolution of convolution of the image with a derivative of Gaussian kernel)the image with a derivative of Gaussian kernel)

image and power spectrum principal image and power spectrum principal components (Torralba et al. 2003)components (Torralba et al. 2003)

• Model training data with mixture of Gaussians (Choudrey Model training data with mixture of Gaussians (Choudrey and Roberts 2003)and Roberts 2003)

Implemented in MatlabImplemented in Matlab• processes grayscale square subimages of the processes grayscale square subimages of the

segmented scene containing the object to be classifiedsegmented scene containing the object to be classified

MBARI May 13rd, 2009MBARI May 13rd, 2009 99

Classifier ExampleClassifier Examplesmall benthic image setsmall benthic image set

QuickTime™ and aNone decompressor

are needed to see this picture.

Example training imagesExample training images

LeukotheleLeukothele

otherother

RathbunasterRathbunaster

MBARI May 13rd, 2009MBARI May 13rd, 2009 1010

Action ItemsAction Items

1.1. Provide the data set (Ken, Jake, Mike)Provide the data set (Ken, Jake, Mike)

2.2. Engineer the workflow to process the Engineer the workflow to process the data set (Danelle) data set (Danelle)

When steps 1 and 2 complete, can start When steps 1 and 2 complete, can start on creating a training library (Linda)on creating a training library (Linda)

MBARI May 13rd, 2009MBARI May 13rd, 2009 1111

Q&AQ&A Project #900905Project #900905

Project wiki: Project wiki: https://oceana.mbari.org/confluence/display/AVEDSTILL/