crops workshop sensor fusion
DESCRIPTION
SensorTRANSCRIPT
![Page 1: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/1.jpg)
Sensor Fusion
Efi Vitzrabin, Yael Edan
Ben-Gurion University of the Negev
![Page 2: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/2.jpg)
Background
Object
Object
Object
Why do we need multiple sensors?
![Page 3: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/3.jpg)
Background
Object Object
![Page 4: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/4.jpg)
Applications
• Mapping for mobile robots
• Target detection
• Localization (GPS+INS)
![Page 5: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/5.jpg)
Aims
Acquisition, processing, and combination of
information generated by multiple knowledge sources
![Page 6: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/6.jpg)
Aims
Provide improved information for detection, estimation, and decision-making.
1+1 >> 2
![Page 7: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/7.jpg)
Agenda • Interaction modes
• Representation levels
• Virtual sensors
• Taxonomy
• JDL Model
• Performance measures
• Application examples
• Dirty secrets in sensor fusion
![Page 8: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/8.jpg)
Interaction models
Complementary
Object
![Page 9: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/9.jpg)
Interaction models
Competitive
Object
![Page 10: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/10.jpg)
Interaction models
Cooperative
Object
![Page 11: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/11.jpg)
Representation levels
• Signal level (pixel level)
• Feature level
• Decision level (symbol/object level)
![Page 12: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/12.jpg)
Signal level representation
Sensor 1
Sensor 2
Sensor N
.
.
.
Fused Signal Output
![Page 13: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/13.jpg)
Feature level representation
Sensor 1
Sensor 2
Sensor N
.
.
.
Classifier
Output decision
Feature extraction
Feature extraction
Feature extraction
Feature extraction
![Page 14: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/14.jpg)
Decision level representation
Sensor 1
Sensor 2
Sensor N
.
.
.
Classifier of classifiers
Output decision
Feature extraction
Classifier
Feature extraction
Classifier
Feature extraction
Classifier
Feature extraction
![Page 15: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/15.jpg)
Virtual sensors
Instead of using only real sensors, virtual sensors (i.e. combination of algorithms) can be used as sensors.
![Page 16: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/16.jpg)
IASTED International Conference, Robotics and Automation 2000
16
Virtual Sensor (VS)
IR
VISION
GPS US
![Page 17: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/17.jpg)
Taxonomy
• Architecture
• Update method
• Dynamic/Static
• Feedback and memory
• Algorithms
• Optimization
![Page 18: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/18.jpg)
Architectures
• Distributed
• Centralized
![Page 19: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/19.jpg)
Update method
• Synchronous
• Asynchronous
![Page 20: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/20.jpg)
Dynamic/Static update
• Static
• Dynamic
![Page 21: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/21.jpg)
Feedback and memory
• Feedback –incorporated in the fusion system from the last sensory output.
• Memory – each sensor can also store in memory the last action it decided.
![Page 22: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/22.jpg)
Feedback and memory
Sensor 1
Sensor 2
Sensor N
.
.
.
Classifier of classifiers
Output decision
Feature extraction
Classifier
Feature extraction
Classifier
Feature extraction
Classifier
Feature extraction
Sensor fusion system with feedback without memory
![Page 23: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/23.jpg)
Feedback and memory
Sensor 1
Sensor 2
Sensor N
.
.
.
Classifier of classifiers
Output decision
Feature extraction
Classifier
Feature extraction
Classifier
Feature extraction
Classifier
Feature extraction
Sensor fusion system with feedback and memory
![Page 24: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/24.jpg)
IASTED International Conference AIA 2001 24
Sensor fusion algorithms
![Page 25: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/25.jpg)
Pixel/Signal level algorithms
• Kalman Filtering
• Estimation methods
![Page 26: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/26.jpg)
Feature level SF algorithms
• Neural networks
• Pattern recognition
• Fuzzy logic
• Decision trees
![Page 27: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/27.jpg)
Decision level SF algorithms
• Bayesian inference
• Dempster–Shafer
• Weighted decision methods
• Minimax
![Page 28: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/28.jpg)
Taxonomy - Optimization
Optimization criterion Description
Least squares Minimize the sum of residuals
Weighted least squares Minimize the sum of the weighted squares of the residuals
Mean square error Minimize the accepted value of the squared error
Bayesian weighted least squares
Minimize the sum of the weighted squares of the residuals constrained by a-priori knowledge of the value
Maximum likelihood estimate
Maximize the multivariate probability distribution function
![Page 29: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/29.jpg)
JDL Model
(Handbook of Multisensor Data Fusion: Theory and Practice 2009)
![Page 30: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/30.jpg)
Overview
(Handbook of Multisensor Data Fusion: Theory and Practice 2009)
![Page 31: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/31.jpg)
Performance measures
• Object detection
• Misclassified objects
• Error in distance of classified objects
• Rate of detections
• False positive detections
• …
![Page 32: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/32.jpg)
Example 1
GPS, INS (Inertial Navigation System) integration – Signal level Both the GPS and INS provide the same data (location). Each sensor has its own characteristics: GPS – long time stability, large noise; INS – short time stability The fusion system is: distributed, asynchronous , dynamic, feedback, no memory. This type of GPS, INS fusion is called loose coupled
LOC
LOC
LOC
![Page 33: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/33.jpg)
Examples 2
Obstacle Ultrasonic
Lasers
Autonomous vehicle must drive and detect obstacles. Each sensor (3 laser scanners and ultrasonic sensor) creates an independent obstacle map. Sensor fusion task is to fuse the various obstacle maps.
![Page 34: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/34.jpg)
Examples 2
The fusion level is at the decision level The fusion system is: centralized, asynchronous - each sensor can work in different frequency, dynamic – the vehicle moves, memory – the last map remains in the memory, new obstacles are added to it. Each obstacle map can have a different resolution and is updated as soon as sensory information is available.
![Page 35: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/35.jpg)
Example 2 – other alternatives
If the sensors send their raw data, then: The fusion level is at the feature level. The fusion system is: centralized, asynchronous - each sensor can work in different frequency, dynamic – the vehicle moves, memory – the last map remains in the memory, new obstacles are added to it.
Disadvantages Advantages
Depended on all sensors Can have more precise results
Hard to predict behavior In complex environment
More sophisticated algorithm is needed
![Page 36: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/36.jpg)
cRops Sensor Fusion
![Page 37: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/37.jpg)
crops sensor fusion
• Decision level
• Centralized
• Asynchronous
• Dynamic
• Feedback
• Adaptive weighted decision methods
![Page 38: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/38.jpg)
cRops sensor fusion The accuracy of each virtual sensor:
a – accuracy of each virtual sensor
c – classifier result
m – number of training data
The final classification result is the weighted sum:
When:
m
t
ii mtytca1
/))()((
n
i
ii jcwjC1
)()(
n
k
i
ii
a
aw
1
![Page 39: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/39.jpg)
cRops sensor fusion
Weights adapt to changes in sensory
performances by changing the accuracy
of each sensor depending on feedback:
1
1
/))()(()(t
ktj
ii kjCjcta
![Page 40: Crops Workshop Sensor Fusion](https://reader034.vdocument.in/reader034/viewer/2022042703/563dbabd550346aa9aa7a6b9/html5/thumbnails/40.jpg)
Dirty secrets in sensor fusion (Handbook of Multisensor Data Fusion: Theory and Practice 2009)
• There is no substitute for a good sensor.
• There is no substitute for good data.
• All sensors have limitations.
• Downstream processing cannot absolve the sins of upstream processing.
• The fused answer may be worse than the best sensor.
• There are no magic algorithms.
• There will never be enough training data.