chris harrison (1 & 2) hrvoje benko (1) omnitouch: …...hrvoje benko (1) andrew d. wilson (1)...
TRANSCRIPT
OmniTouch: Wearable Multitouch Interaction Everywhere
Chris Harrison (1 & 2)
Hrvoje Benko (1)
Andrew D. Wilson (1)
Affliations: 1. Microsoft Research
2. Carnegie Mellon University
Objectives
• Developed a novel wearable depth-sensing and projection system that enables interactive multitouch applications on everyday surfaces. The paper shows that anything one can do on today’s mobile devices, they could do in the palm of their hand.
• The developed system allows the wearer to use their hands, arms and legs as graphical, interactive surfaces.
• It also allows the users interact on appropriate surfaces from the environment to expand the interactive area.
• OmniTouch provides a X-Y plane (2D interface) as in a touchscreen mobile phone display with clicking or hovering of fingers are detected for interactions.
Project Demo
Hardware Setup
The lone hardware module consisted of a Microsoft Kinect (PrimeSense Depth Camera), Microvision ShowWX+ laser pico-projector, and a computer tethering the Kinect and pico-projector.
Hardware Capabilities
• Resolution: 320 x 240• Speed: 30 FPS• Error on depth sensing: 5mm for an object that is
20cm far on Z-axis
PrimeSense depth camera
• Wide angle• Focus free projection of graphical elements regardless
depth
Microvision ShowWX+ laser pico-projector
Multitouch Finger Tracking
• Finger SegmentationA. Depth derivative in the X-Y
plane using slidingB. Derivative of the depth mapC. Finger slices overlaid in blueD. Path finding and tip
estimation
Close up example of a candidate finger slice
Multitouch Finger Tracking (cont’d)
• Finger Click Detection• Midpoint of the active fingers
are computed• Flood fill towards the fingertip• This operation is performed on
the depth map
Hovering Clicked
On demand projected interfaces
Due to the dynamic nature of the objects of real world, the displays must be projected in such a way as to account for their host surface’s position and orientation in 3D space.
Surface segmentation and tracking• Distinct surfaces are segmented by performing
a 3D connected components operation on the depth map. Surfaces smaller than hand size are discarded.
• Object orientation is computed on a 3D co-ordinate system.
• Lock points are calculated taking the centroid of the object on which the display is being projected. This method is chosen for stability.
Characteristics of the projected interfaces
• OmniTouch can use a surface’s lock point and orientation to provide an interface that tracks with a surface.
• When a projecting on a large table, the interface will still be hand-sized.
One Size Fits All
• The authors performed surface classification among a set of five common surfaces (hand, arm, pad, wall, and table) by considering a variety of features derived from each surface’s depth image.
• Unfortunately, this classification-driven approach suffers from scalability issues, since it is simply not possible to build a classifier for every conceivable surface.
Classification-Driven Placement
• So that users can drag and replace the entire projection.
User-Specified Placement
Context-awareness of projection surface
• The surface tracking procedure generates a variety of metadata, such as whether a surface is public or private, organic or flat, big or small, and vertically or horizontally oriented – all of which are exposed to applications wishing to be more context sensitive.
User study
The four surfaces we tested and user click distributions. 95% confidence ellipses shown in green. Axis units in mm.
System accuracy
Finger click detection• Click receiving: 96.5%• Click segmentation:
98.9%
1Finger click spatial accuracy• Finger click spatial
precision: 95% (Button diameter, next slide)
2
Button diameter
Button diameter necessary to encompass 95% of touches. Error bars denote standard deviation across
all trials.Button diameter needed to encompass 95% of
touches. Error bars show standard deviation across trials.
My opinions
• Novel idea. Well written manuscript. Strong background work. Stable implementation and excellent demonstration.
• Achieved accuracies advocate for feasibility on consumer level applications.
Positives:
• Application receives a lot of ‘WOW’s. But this particular technology didn’t go far in the consumer market of gadgets.
• Ranges of applications are limited to typical smartphone applications, which is not that bad. However, the authors should have presented one application regarding multimedia presentation. For example, this class presentation could have been better using OmniTouch.
Negatives:
Follow up works
Questions?