machine vision group multimodal sensing-based camera applications miguel bordallo 1, jari hannuksela...
TRANSCRIPT
MACHINE VISION GROUP
Multimodal sensing-based camera applications
Miguel Bordallo1, Jari Hannuksela1, Olli Silvén1 and Markku Vehviläinen2
1 University of Oulu, Finland2 Nokia Research Center, Tampere, Finland
Jari Hannuksela, Olli SilvénMachine Vision Group, Infotech Oulu
Department of Electrical and Information EngineeeringUniversity of Oulu, Finland
MACHINE VISION GROUP
Outline
Introduction• Modern movile device with multiple
sensorsVision-based User InterfacesSensor data fusion systemApplication case implementations
• Motion-based image browser• Motion sensor assisted panorama
imagingConclusions/Summary
MACHINE VISION GROUP
Introduction
• More and more applications and features are being crammed into handhelds
• Causes usability complications given the constraints of current mobile UIs
• Increased computing power not harnessed for UIs
• Keypad and pointer based UIs and/or touchscreens in current devices– User’s hand obstructs the view– Require two handed operation
MACHINE VISION GROUP
Modern mobile device with multiple sensors• The phone includes touch screen, GPS,
accelerometers, light sensor, proximity sensor • Two cameras: low resolution for video calls and high
resolution for photography and video capture• Newer phones will include magnetometers,
gyroscopes
MACHINE VISION GROUP
Motivation for vision based user interfaces
Allow recognition of the context- Detect user’s actions- Recognize environment
Allow 3D informationProvide interactivity
- Real-time feedback- Single hand use
MACHINE VISION GROUP
Limitations of vision based UIs
Fast movements
Low light
Difficult conditions
+
MACHINE VISION GROUP
The solution: sensor data fusion
Fusing the data obtained by several sensors
• Ambience light sensor determines illumination conditions
• Video analysis detects ego-movements and analyzes the context
• Accelerometers provide complementary motion data
MACHINE VISION GROUP
Video analysis
- Every frame divided into regions- Selection of feature blocks- Estimation of block displacements- Analysis of uncertainty
- Results: 4-paramenter model- X, Y, Z, r
MACHINE VISION GROUP
Sensor data fusion
Model the device movement with the folowing
Define a state vector: position, speed, acceleration
Define a measurement model
Apply Kalman filtering adding accelerometer values: State prediction + state correction
MACHINE VISION GROUP
Application cases
• Sensor data fusion method applied in two applications– Implemented on a Nokia N900 mobile phone
• Motion based image browser– Allows browsing large images and maps with one hand operation– Works under different light conditions
• Sensor assisted panorama imaging– Stitches panorama images in real time from video frames– Increased robustness against fast movements and no-texture
frames
MACHINE VISION GROUP
Motion based image browser
Uses fusion model from accelerometers + video analysis to generate commands
• Scroll up/down/left/right• Zoom in/out
Light sensor decides:• if camera should be turned on • weighting factors and uncertainties• 3 modes defined:
• Good image quality (video analysis + accelerometer correction)• Bad image (accelerometers have increased contribution)• No image (only accelerometers are used)
MACHINE VISION GROUP
Sensor assisted panorama Imaging
•Based on video analysis
•Guides the user with instructions
•>360 degrees panoramas •Real-time registration•Real-time frame evaluation and selection•Real-time frame correction
•Increased robustness via sensor-data integration
MACHINE VISION GROUP
Panorama imaging: Sensor uses
•Uses sensor fusion model to compute camera motion•Increased robustness against fast movements and frames with low/smooth texture
Registration
MACHINE VISION GROUP
Panorama: Sensor uses II
•Uses accelerometer data to detect blur•Detects unwanted shake/tilt•Integrated in scoring system
Selection
MACHINE VISION GROUP
Summary
• Vision based interfaces offer high interactivity with one hand operation
• They present several limitations• Sensor fusion improves motion estimation
adding robusness against fast movements and dark conditions
• The framework can be included in several applications (e.g. as a part of Motion Estimation API)
MACHINE VISION GROUP
Conclusions
• We have presented a sensor fusion framework that fuses vide analysis with motion sensors (acelerometers+magnetometers+gyroscopes)
• We have presented two applications cases that make use of sensor data fusion and integration
• The applications presented are by no means the only ways to apply vision or multiple sensors, and one may find new interesting possibilities in further research