1/14/2017
1
How Does Artificial Intelligence Work with Remote Sensing Technologies for
Multi-scale Environmental Change Detection?
Prof. Ni-Bin ChangDirector, Stormwater Management Academy
University of Central FloridaOrlando, FL, USA
WHAT IS REMOTE SENSING?• Remote sensing is defined as the
• acquisition,• processing,• and interpretingof images that are remotely obtained from sensors
recording the interaction between electromagnetic energy and the Earth’s surface
• Remote Sensing Platforms:• Ground-based• Airplane-based• Satellite-based
2
1/14/2017
2
The amount of radiation that is emitted and reflected from an object is a function of wavelength
Materials can be identified based upon their unique spectral signatures
Source: Siegmund, Menz 2005
SPECTRAL SIGNATURE OF SOIL, VEGETATION AND WATER, AND SPECTRAL BANDS OF LANDSAT 7
THE RESEARCH NICHES FOR ENVIRONMENTAL REMOTE SENSING
• Who kinds of limitation of remote sensing technology dowe face at present?
2013-04-12 2013-05-14 2013-06-15 2013-07-17
2013-08-18 2013-09-19 2013-10-21 2013-11-22
1/14/2017
3
WHAT IS LAKE EUTROPHICATION?
• Eutrophication is the excess loading of nutrients to a water body and the resulting effects of ecosystem health linked to the surplus quantities of nutrients.
(a) Incipient stage (b) Fostering stage (b) Fostering stage (d) Bloom spreading stage
Algal bloom event observed in Sept. 18-Sept. 22, 2008 (Chang et al., 2008).
Empirical Methods
Analytical Method
Semi-empirical
Empirical Method Satellite Parameters Location Reference
Artificial Neural Network
Landsat TM
Chlorophyll-a and suspended sediment
Delaware Bay, U.S.
Keiner and Yan, (1998)
Genetic Programming (GP)
SPOT Chlorophyll
Yeong-Her-Shan
reservoir,
China
Chen (2003)
Radial Basis Function Neural
(RBFN) Network Models
Landsat TM
Chlorophyll-a and Suspended Matter
Beaver Lake, U.S.
Panda et al., (2004)
FEATURE EXTRACTION IN IMAGE CLASSIFICATION
1/14/2017
4
0
10
20
30
40
50
60
70
80
Regression ANN GP FUSION
1990s
2000s
2010s
FEATURE EXTRACTION METHODS FROM SATELLITE DATA
Chang, N. B., Imen, S., and Vannah, B. (2015): Remote sensing for monitoring surface water quality status andecosystem state in relation to the nutrient cycle: a 40-year perspective. Critical Reviews of EnvironmentalScience and Technology, 45(2), 101-166.
Source: Chang et al., 2015
Suspended Sediment Concentration: Landsat
Chlorophyll-a: Hyperspectral remotesensing is more reliable for monitoring Chl-aconcentration than multispectral remotesensing because it can consider the reflectanceof the extremely narrow wavebands.
Nutrients: Landsat and MODIS
Total Organic Carbon: Landsat and MODIS
Microsystin: MODIS and Landsat
RECENT BREAKTHROUGHS IN RESEARCH
1/14/2017
5
IEEE Systems Journal
Chang, N. B., Xuan, Z., and Yang, J. (2013): Exploring spatiotemporal patterns of nutrient concentrations in a coastal bay with MODIS images and machine learning models. Remote Sensing of Environment, 134, 100-110.
Source
1/14/2017
6
Chang et al., 2013
FEATURE EXTRACTION USING GENETIC PROGRAMMING
MYD09GA
MODIS Reprojection Tool
ArcGIS
Output Map
Ground Truth Data Inverse Model (GP)
MYD09A1
GP model building; inverse modeling
Genetic Programming (GP) Supervised, unsupervised, and semi-supervised learning algorithm
Support Vector Machine (SVM) Supervised learning algorithm toward unsupervised learning
algorithm
Artificial Neural Network (ANN) Supervised or unsupervised learning algorithm depending upon
the training techniques employed
CATEGORIZATION OF MACHINE LEARNING ALGORITHMS
1/14/2017
7
• Tampa Bay Water Atlas provides historical In situ data via Online Databases through USF.
• The data collections can be tracked back more than 40 years.
• 45 monitoring stations
• 581 data points for model calibration
• 159 data points for model validation
IN-SITU OBSERVATIONS AS GROUND-TRUTH
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2007 6,13 24 26 21 4
2008 30
2009 9 25 29 6,13
• From the SeaDAS main menu, we selected several products related to our study
• These include • the MODIS data with bandwidth between 405 and 683nm an• two other products of information concerning colored dissolved organic
material (CDOM) and chlorophyll-a (chlor-a).
Chang et al., 2012
RETRIEVAL OF SYNCHRONOUS SATELLITE DATA IN CLEAR DAYS TO AVOID CLOUD IMPACT
1/14/2017
8
GENETIC PROGRAMMING MODEL CALIBRATION AND VALIDATION
Overcome the current challenge of feature extraction
Chang et al., 2013
• Genetic algorithms used in evolutionary computing were developed by John Holland in 1975.
• GP is a specialization of genetic algorithms that employ machine learning and data mining techniques to identify system behaviors based upon empirical data.
• GP mimics natural evolutionary pathways to optimize computer programs in order to converge toward a target solution.
BACKGROUND OF GP MODEL
1/14/2017
9
Crossover
• Choose a random point on the two parents
• Split parents at this crossover point
• Create children by exchanging tails
• Pc typically in range (0.6, 0.9)
Mutation:
• Alter each gene independently with a probability pm
• pm is called the mutation rate• Typically between 1/pop_size and 1/
chromosome_length
GENERIC ALGORITHMS
a) General structure of a GP
b) Crossover between programs
c) Program mutation
TREE-BASED GP MODEL
1/14/2017
11
TP (mg L-1) = (((X2+ V4 + V0) / X1 + V2)2 + V0) / X12 +
0.03184f / X1 + 0.03184f
X1 = 1- X2- X3- X4
X2 = (X3 + 0.19189f - V6) / (1 - X3 - X4) + V0
X3 = 2*(2X4 + 0.03924f) + V1 + V0 + 2V2 - 2V3
X4 = 2V12*(-0.62967f)-V3
f: floating point register
THE GP MODEL ANALYSIS FOR TP ESTIMATION
1/14/2017
13
Chang N.B., Vannah B., Yang Y.J., 2014, Comparative sensor fusion between hyperspectral andmultispectral satellite sensors for monitoring Microcystin distribution in Lake Erie”, IEEE Journal ofSelected Topics in Applied Earth Observations and Remote Sensing , 7(6), pp. 2426-2442.
Reference
26
URBANIZATION AND WATERSHED MANAGEMENT
AP Photo: Haraz N. Ghanbari
The toxins cause Toledo's water supply out of service on Aug. 3 2014.
1/14/2017
14
Because Microcystis is a bacterium that usesphotosynthesis for energy production, highconcentrations of Microcystis can be linkedwith elevated chlorophyll-a levels.
Phycocyanin is a pigment in all cyanobacteriathat shares a positive correlation withMicrocystin levels.
Surface reflectance of phycocyanin, chlorophyll-a,and Microcystis are suitable indicators for theprediction of Microcystin levels in a lake.
The surface reflectance curves for chlorophyll-aand phycocyanin in surface waters peak at 552,650, 680, and 720 nm.
RETRIEVAL OF MICROCYSTINFROM SATELLITE DATA
FUSION OF TWO MULTISPECTRAL IMAGES FOR ENVIRONMENTAL APPLICATIONS
STAR-FM = Spatial and Temporal Adaptive Reflectance Fusion Model
• Landsat
• High spatial resolution
• Low revisit rate
• MODIS
• High temporal resolution
• Low spatial resolution
1/14/2017
15
INTEGRATED DATA FUSION AND MINING (IDFM)
Objectives: generate daily highspatial resolution of syntheticimage for intensive monitoring.
1MERIS and MODIS Data Acquisition
MERIS Reflectance Bands (300 m)
MODIS Reflectance Bands (1000 m)
Image Processing Steps:-Reproject to UTM 17N-Crop out Land
Image Processing Steps:-Reproject to UTM 17N-Resample to 300 m-Crop out Land-Offset and scale data
MERIS Reflectance Bands (300 m)
MODIS Reflectance Bands (300 m)
2
Data Fusion3
Fused Surface Reflectance Bands (300 m)
Ground-Truthing Data
Microcystin Concentration Map
(300 m)
Data Mining
Microcystin Prediction Model
4
5
• Procedural Flow:1. Data Acquisition2. GIS Image Processing3. Image Fusion (STARFM)4. Train and Validate GP
Model5. Generate Microcystin
Concentration Maps• Created by inputting
fused band data into the GP model
30
• Goodness of Fit:• R = 0.9098• R2 = 0.8278
MODIS #1
Landsat #1
MODIS #2 MODIS #3
Landsat #3
• Input Data Streams:
• 3 MODIS
• 2 Landsat
• STARFM Output:
• 1 Fused ImageFused Image
(F)
Data Fusion Through STAR-FM
FUSION OF HYPERSPECTRAL AND MULTISPECTRAL IMAGES FOR ENVIRONMENTAL APPLICATIONS
1/14/2017
16
Satellite SensorProduct
Selection
Spatial
Resolution
Temporal
ResolutionBands Used
Terra MODIS
Surface
Reflectance
(Level 2)
1000 m Daily 8-16
ENVISAT MERIS
Surface
Reflectance
(Level 2P)
300 m 1-3 Days 1-13
MERIS (MEdium Resolution Imaging Spectrometer) measures the reflectance of the Earth (surface and atmosphere) in the solar spectral range (390 to 1040 nm) and transmits 15 spectral bands back to the ground segment.
Integrating Hyperspectral and Multispectral Remote Sensing
Band centers with the highest performance for the traditional two-band models
The band centers most used by thetraditional two-band models fall in therange of 560-570 nm and 660-681 nm.
This range corresponds to the spectralfeatures observed in this figure.
FUSION RESULTS OF BIO-OPTICAL MODELS
1/14/2017
17
STATISTICAL ANALYSIS
• The GP model using MERIS inputs had the best overallperformance.
• All GP models exemplified higher explanatory powerthan the two band model.
• The fused model had the longest computational time,yet this model has a number of inherent advantages:• 300 m spatial resolution for enhanced algal bloom delineation• Daily revisit time, which is necessary for an early warning system
Two Band MODIS GP MERIS GP Fused GPObserved Microcystin Mean (µg·L⁻¹) 1.425 0.4430 1.425 0.6606Predicted Microcystin Mean (µg·L⁻¹) 0.8082 0.4342 1.388 0.5737Root Mean Square Error (µg·L⁻¹) 1.846 0.1351 0.4951 0.4798Ratio of St. Dev. 0.4235 0.6926 0.7127 0.6363Mean Percent Error (%) 13.22 23.80 5.809 24.18Square of the Pearson Product Moment Correlation Coefficient (R2) 0.5256 0.9204 0.9469 0.8824
Computational Time (seconds) < 1 452 192 1300
MERIS GP Model
Fused multispectral sensor
Fused hyperspectral sensor
26 Data points
41 Data points
41 Data pointsMore capable Predicting low
concentration
RESULTS OF PREDICTIVE CAPABILITY WITH 3 GENETIC PROGRAMMING MODELS
1/14/2017
18
(a) Hyperspectral sensor GP model (B) Multispectral sensor GP model
The 30m spatial resolution of the multispectral image provides more detailed outline, while thecoarser (300 m) hyperspectral resolution predicts microcystin concentration in locations that moreclosely align with harmful algal bloom (HAB) presence.
Dark red dots denotes areas of high microcystin concentration that pose a health threat, whileyellow spots indicate low to medium microcystin concentrations.
RESULTS OF MICROCYSTIN CONCENTRATION MAPS
SMIR: Smart Memory for Image Reconstruction
SIASS: Spectral Information Adaptation and Synthesis Scheme
RECENT BREAKTHROUGHS IN MY RESEARCH
Chang, N. B., Bai, K. X., and Chen, C. F. (2015): Smart information reconstruction via time-space-spectrumcontinuum for cloud removal in satellite images, IEEE Journal of Selected Topics in Applied EarthObservations, 99, 1-19.
Bai, K. X., Chang, N. B., and Chen, C. F. (2015): Spectral information adaptation and synthesis scheme formerging cross-mission consistent ocean color reflectance observations from MODIS and VIIRS. IEEETransactions on Geoscience and Remote Sensing, 54(1), 311-329.
Bai, K. X., Chang, N. B., and Chen, C. F. (2015): Spectral information adaptation and synthesis scheme for merging cross-mission consistent ocean color reflectance observations from MODIS and VIIRS. IEEE Transactions on Geoscience and Remote Sensing, 54(1), 311-329.
Chang, N. B., Bai, K. X., and Chen, C. F. (2015): Smart information reconstruction via time-space-spectrum continuum for cloud removal in satellite images, IEEE Journal of Selected Topics in Applied Earth Observations, 99, 1-19.
Reference
1/14/2017
19
NEW APPROACHES
Spectral Information Adaptation and SynthesisScheme (SIASS)oprinciples: cross-mission sensors provide multi-
observations at different crossing timeomethod: merging cross-mission ocean color products
could improve spatial and temporal coverage
SMart Information Reconstruction (SMIR)oprinciples: memory effect of interrelationships between
cloudy pixels and cloud free pixelsomethod: reconstruction from cloud free pixels
Cross-mission Data Merging with Image Reconstruction and Mining (CDMIM) oprinciples: SIASS + SMIR + Feature Extractionomethod: large scale complex systems
SIASS: SPECTRAL INFORMATION ADAPTATION AND SYNTHESIS SCHEME
• Multiple Satellites
• Visible Infrared Imaging Radiometer Suite (VIIRS)
• Moderate-Resolution Imaging Spectroradiometer (MODIS)-Aqua
• Moderate-Resolution Imaging Spectroradiometer (MODIS)-Terra
• Features
Courtesy of NASA
1/14/2017
20
chlor-a concentration retrieved from three ocean color bands
After merging, the spatial coverage has been greatly improved, but there are still some value missing pixels.
Statistics of Terra-MODIS Ocean Color images with full coverage over Lake Nicaragua
through 2003-2013.
Annual C5 C11 Year 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2009-2013 2003-2013
Num. of Images 293 291 296 303 291 303 300 292 290 290 283 366 366
POC (%)
Min. 0.08 0.04 0.35 0.73 0.18 0.15 0.33 0.04 0.04 0.26 1.5 0.01 0
Max. 100 100 100 100 100 100 100 100 100 100 100 100 97.44
Avg. 71.87 71.23 75.5 75.55 75.31 73.51 72.53 74.36 72.65 70.3 70.26 37.59 17.53
* C5: Composited climatologic images with the latest 5 years (2009-2013) images, C11: same as C5 but with the past 11 years (2003-2013) images.
Figure Time series of the cloud coverage over Lake Nicaragua with 5 (a) and 11 (b) years composition.
1/14/2017
21
ARTIFICIAL NEURAL NETWORKS
• Schematic comparison between a biological neuron and an artificial neuron.
• Biological Neurons
• Artificial Neurons
after Winston, 1991; Rich and Knight, 1991
Feedforward Backpropagation Neural Networks
Analogy between biological neurons and artificial neurons
Courtesy of David Leverington
THE ARCHITECTURE OF EXTREME LEARNING MACHINE
Shen et al., 2014
1/14/2017
22
Data Merging
Data Merging and Reconstruction
A: 18.6% A + V: 24.64% A + V +T: 24.92%
Figure 1: Comparisons of (left) original MODIS-Aqua (A) ocean color reflectance image before and after fusing at 678 nm with (b) VIIRS-NPP (V) and (c) MODIS-Terra (T) on June 14, 2014. The percent coverage was given as the ratio between the number ofpixels having valid data value and the total number of pixels covered in the lake, and a value of 100% means the fully coverage.
Data Reconstruction
Data Merging and Reconstruction
A + V +T: 24.92% Reconstructed: 98.12%
SMIR
Figure 2: Comparisons of (left) the merged ocean color reflectance image and (right) the reconstructed one on June 14, 2014.
1/14/2017
23
Water Quality Monitoring with the CDMIM Algorithm
Total Nitrogen
DOY: 2014168 DOY: 2015075
Figure 3: Predicted total nitrogen (TN) concentrations in Lake Nicaragua at June 17, 2014 (DOY: 2014168) and March 16, 2015 (DOY: 2015075) using data mining approaches.
Water Quality Monitoring with the CDMIM Algorithm
DOY: 2014168 DOY: 2015075
Figure 4: Predicted total phosphorous (TP) concentrations in Lake Nicaragua at June 17, 2014 (DOY: 2014168) and March 16, 2015 (DOY: 2015075) using data mining approaches.
Total Phosphorous
1/14/2017
24
SUMMARY
2012: IDFM provides the biases for advanced earth observations.
2014: SIASS removes the biases among cross-mission sensors prominently.
2014: SMIR can recall more invariant spectral features reserved in previous images.
2016: CDMIM can be applicable for uncovering many environmental change detection in almost all-weather conditions.