carnegie mellon ambient intelligence...
TRANSCRIPT
Spatiotemporal Data Mining for Monitoring Ocean Objects
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Yang Cai, Ph.D.Ambient Intelligence Lab
Carnegie Mellon [email protected]
Collaborators
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Karl Fu, Carnegie MellonXavier Boutonnier, Carnegie MellonDaniel Chung, Carnegie MellonRichard Stumpf, NOAATimothy Wynne, NOAAMitchell Tomlison, NOAAJames Acker, GSFCCynthia Heil, FWRI Y. Hu, LaRC
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Spatiotemporal dynamics of surface objects is a part of our everyday-life, from “red tide”, river plume, vessels, flood, urban growth, to urban traffic congestion.
Scientific Questions
Tracking:
Given an object in an image sequence (t=1,..,n), how to find the object at t=n+1 and beyond?
Prediction:
Given databases of historical data and current physical and biochemical conditions, how to predict the occurrence of the interested object at a particular time and location?
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Why do we need data mining here?
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
1. Lots of historical data (8-year SeaWiFS and40-year cell count and wind databases)
2. Current models haven’t been successful.
3. Data mining appears to be inexpensive.
4. Most of domain experts still spend 80% time to do ‘manual’ data mining.
Challenges to Data Mining Technologies
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
• visual or hyper-spectrum content
• space and time
• deformation + transport
• missing data (80% clouds in images)
• multiple databases
• multi-physics
Objective
Key Milestones
• Motion tracking model 6 mo/1 yr• Case studies with plume/HAB 6 mo/1 yr• Spatiotemporal motion model 6 mo/2 yr• Spatial frequency pattern model 6 mo/2 yr
• To track the movement of ocean objects that have been identified
• To predict the movement of identified objects.
Sponsored by NASA ESTO-AIST-QRS-04-3031
Co-I/Partner
Co-I, Richard Stumpf, NOAA
Approach
spatiotemporal object motion tracking
TRLin = 4
Spatiotemporal Data Mining System for Tracking and Modeling Ocean Object Movement
• Computer vision and visualization• Statistical spatiotemporal data minining.• Case studies with SeaWiFS datasets.
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
V.M.S.V. Methodology
Vision:
• Spatial Density Filter
• Correlation Filter and Particle Filter
• Mutual Information
Mining:
• Spatiotemporal Neural Network
• Spatiotemporal Bayesian Model
• Periodicity Transform
Simulation:
• Cellular Automata
Visualization:
• Pseudo Color, Mapping, Animation...
Images above show a harmful algae bloom (HAB), highlighted as chlorophyll anomaly, drifting along the southwest Florida coast in December 2001.
Case Study:
Harmful Algal Blooms
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Correlation Study
Use chlorophyll as a surrogate for Karenia Brevis blooms (NOAA)
References:
Tomlinson, M.C., R.P. Stumpf, V. Ransibrahmanakul, E.W. Truby, G.J. Kirkpatrick, B.A. Pederson, G.A. Vargo, C. A. Heil., 2004. Evaluation of the use of SeaWiFS imagery for detecting Karenia brevis harmful algal blooms in the eastern Gulf of Mexico. Remote Sensing of Environment, v. 91, pp. 293-303.
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Chlorophyll channel Anomaly channel
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Image Data Preprocessing
• Remove noises in the satellite images
• Grouping objects
• Recover the missing data
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Spatial Density Clustering
1. Set all the neighboring dots within a distance (Dis) as one test set 2. If number of dots > Min, then the point is a core point 3. Remove the non-core points 4. Go to step 1
Dis
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Sample of SDC vs. Binary Morphology
noisy image SDC output Morphology output
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Missing Data Recovery
1. Which is which?
2. Concavity of objects.
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Interpolation of a convex object
We take all the points of the contours of the marginal objects and by linear interpolation calculate the position of the interpolated point.
The Hull Convex of the interpolated points gives the contour of the interpolated convex object.
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Work around concavity
1. First we extract the concavity.2. Then we interpolate the object and the
concavities.3. Then we remove the part
corresponding to the interpolated concavity from the interpolated object
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Results
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Shape Grouping Results
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Correctly Marked Surface Objects (HAB)
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Object Tracking with Correlation Filter
where,
a is the test image
b is the reference object in the previous image to be tracked.
FFT(x) represent Discrete Fast Fourier Transform
IFFT(x) is Inverse Discrete Fast Fourier Transform.
Shape Correlation = IFFT(FFT(a).* conj(FFT(b)))
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Tracking Results
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Tracking HAB with Correlation Filter within 4-day interval
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Tracking of a bloom which has split into 2 pieces sampled in an interval of 4 days using particle filter
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
River plume tracking with Mutual Information
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Neural network spatiotemporal prediction model
Radial Basis Function
Kohonen Network
raw image sequence
clustered points **
predicted image for t+1
** from Cartesian to polar coordinates
r
θ
point in frame t-1
point in frame t
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Neural Network Prediction Model Example
• The first 3 images are input shapes (zero wind speed)
• The last image is the predicted shape overlaid on top of the ground truth.
• The blue dots are the clustered data points that represent the shape.
** Reference: Mitchell, T. Machine learning, McGraw-Hill, 1997
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Prediction model results
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Prediction model results
Cluster Resolution vs. Time
0.00
100.00
200.00
300.00
400.00
500.00
0 500 1000 1500 2000 2500
Cluster Resolution (clusters)
Tim
e (C
PU ti
me)
time
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Periodicity Transform
26.4526.5
26.5526.6
26.65
-82.4
-82.3
-82.2
-82.17.2
7.25
7.3
7.35
x 105
latitude
Space time coordinates of the measurements
longitude
date
0 2000 4000 6000 8000 10000 12000 140000
1
2
3
4
5
6x 105
0 100 200 300 400 500 600 7000
0.05
0.1
0.15
0.2
0.25
Periodicity
Pow
er
Power of the components of periodicity transform of a medium set of data
2 12 2
20
1 ( )p
iEnergy x x i
p
−
== = ∑
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Experiments with cell count data
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Zoomed result
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Super-zoomed results
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Cellular Automata
CA is a two-dimensional simulation of surface physics, chemistry or biology. It’s simple however, it could be computationally expensive for large problems.
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Visualization of the prediction model
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
3-D Stereo Projection Lab
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
V.M.S.V. Methodology
Vision:
• Spatial Density Filter
• Correlation Filter and Particle Filter
• Mutual Information
Mining:
• Spatiotemporal Neural Network
• Spatiotemporal Bayesian Model
• Periodicity Transform
Simulation:
• Cellular Automata
Visualization:
• Pseudo Color, Mapping, Animation...
Conclusions
1. Data mining without domain expertise is like fishing-in-the-dark. Multiple experts are needed.
2. Vision-Mining-Simulation-Visualization (VMSV) method puts human experts in the loop, which increase the chance of success.
3. Vision algorithm enables automated image-based data mining.
4. Neural network model shows promising in compressing the shape information in orders of magnitude. However, it has its limitations in long-term prediction.
5. Bayesian prediction shows promising in long-term prediction and efficient computational speed.
6. How to couple the multi-physics models with data mining models is a big challenge.
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Publications
1. Y. Cai, R. Stumpf, etc. Spatiotemporal Data Mining for Prediction of Harmful Algal Blooms, International Harmful Algae Conference, Copenhagen, September 8-12, 2006
2. Y. Cai, Y. Hu, Onboard Inverse Physics from Sensor Web, Proceedings of Space Missions and Challenges, SMC-IT, JPL, 2006
3. Y. Cai and K. Fu, Spatiotemporal Data Mining with Cellular Automata, Proceedings of International Conference of Computational Science, ICCS 2006, May 30, UK
4. Y. Cai, D. Chung, K. Fu, R. Stumpf, T. Wynne, M. Tomlinson, Spatiotemporal Data Mining with Micro Visual Interaction, submitted to Journal of Knowledge and Information Systems
5. Y. Cai, K. Fu, R. Stumpf, T. Wynne, M. Tomlinson, Spatiotemporal Data Mining for Monitoring Ocean Objects, submitted to NASA Data Mining Workshop, JPL, 2006
6. Y. Cai, Y. Hu, Sensory Stream Data Mining on Chip, submitted to NASA Data Mining Workshop, JPL, 2006
7. Y. Cai, (editor), Special Issue of Visual data Mining, Journal of Information Visualization, to be published by Elsevier, 2006
8. Y. Cai and J. Abascal, (editors), Ambient Intelligence in Everyday Life, Lecture Notes in Artificial Intelligence, LNAI 3864, to be published by Springer, April, 2006
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab
Acknowledgement
The study is supported by NASA ESTO grant AIST-QRS-04-3031. We are indebted to our collaborators in NOAA, FWRI, GSFC. The authors appreciate the comments and suggestions from Karen Meo, Kai-Dee Chu, Steven Smith, Gene Carl Feldman and James Acker from NASA. Also, many thanks to Christos Faloutous and Mel Siegel from CMU and Andrew Moore from Google Research Center in Pittsburgh.
Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab