![Page 1: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/1.jpg)
07
Ing. Matteo Valoriani
HCI – Lesson 3
Gesture based interaction
![Page 2: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/2.jpg)
Interface Evolution
Command Line Interface
Graphical User Interface
CLI GUI
NUI
![Page 3: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/3.jpg)
Natural User Interface
3
Computer Vision
Facial Recognition
Spatial Recognition
Augmented Reality
Gesture Sensing
Audio Recognition
Voice Command
Natural Speech
Ambient Noise
Touch
Single Touch
Multi-Touch
Pen Input
Sensors
Geospatial Sensing
Accelerometers
Biometrics
Ambient Light
Brain Waves
Mind control
Mood Recognition
![Page 4: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/4.jpg)
Multi-Touch interface
HCI 2012/2013
Matteo Valoriani
![Page 5: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/5.jpg)
make a withdrawal…
![Page 6: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/6.jpg)
megawatts86
![Page 7: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/7.jpg)
Airplane check-in…
![Page 8: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/8.jpg)
mrkathika
![Page 9: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/9.jpg)
Shopping…
![Page 10: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/10.jpg)
pin add
![Page 11: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/11.jpg)
What’s happened?
![Page 12: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/12.jpg)
Jefferson Han Han was named to Time Magazine's 2008 listing
of the 100 Most Influential People in The World.
Natural experience
“When you see an object you want touch it, it’s the most immediate and spontaneous reaction that a human can have. Interact with a computer without keyboard or other, but only with hands, I believe that is the best way to break the technological barrier.”
![Page 13: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/13.jpg)
AFTER
BEFORE
• More devices
• Individual experience
• 1 click
What is changing?
• Single device
• Collaborative experiences
• More points
![Page 14: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/14.jpg)
Sensible to touch surface (touch screen)
• RESISTIVE
• CAPACITIVE
Multi-Touch Hardware
![Page 15: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/15.jpg)
Camera-based technology:
• Laser Line Plan
• Frustracted Total Internal Reflection
• Diffused illumination
• Pixel Sense
Multi-Touch Hardware (2)
http://sethsandler.com/multitouch/
![Page 16: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/16.jpg)
SmartPhone
iphone – iPod Touch
Multitouch Devices
![Page 17: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/17.jpg)
Tangible Table/ Wall
Microsoft Surface 2
Multitouch Devices(2)
Perceptive Pixel Multi-touch wall
![Page 18: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/18.jpg)
Monitor and tablet
Dell Multi-Touch Monitor Apple iPad
Multitouch Devices (3)
![Page 19: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/19.jpg)
Apple TrackPad
Multitouch Devices (4)
![Page 20: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/20.jpg)
All new versions of the principal OS support multitouch, but in different ways:
• Pen input
• Multi-Touch screen
• Multi-Touch trackpad
Multi-Touch Software
![Page 21: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/21.jpg)
• Microsoft "Surface SDK" and "Windows Presentation Foundation" include API, documentation and tool to develop multitouch App on Windows 7 and Surface
• "Cocoa Touch" is a library to develop software for iPhone, iPod Touch, e iPad. "Cocoa Touch" is included in iPhone SDK.
• Android SDK include tool, emulator, debugger and library to develop App for Android OS
• Gestureworks (by Ideum) is an interesting Flash multitouch. The Gestureworks software allows to develop multiuser and multitouch-enabled applications with Adobe Flash.
Multi-Touch Software(2)
![Page 22: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/22.jpg)
http://www.lukew.com/touch/TouchGestureGuide.pdf
Common Gestures
![Page 23: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/23.jpg)
• Nails
• Gloves
• Dirty fingers
• Gestures (are the so easy?)
• Accuracy
Problems: Input
![Page 24: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/24.jpg)
Gorilla arm problem
Problems: Accessibility
![Page 25: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/25.jpg)
Touch-based applications introduce new important constraints in design of interface and interaction:
• Target’s dimensions must be fit to fingers’ dimension (min 10mm)
Problems: Usability
![Page 26: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/26.jpg)
• Labels’ position: hands and fingers can hide information
Problems: Usability (2)
![Page 27: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/27.jpg)
Problems: Usability (3)
• Gesture’s design: some actions can hide part of information too.
![Page 28: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/28.jpg)
Hell
Tricks & Tips
• Iceberg Tips: create a wider invisible area
• Adaptive Targets: device tries to guess next button pressed by user and zoom it
![Page 29: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/29.jpg)
• Don’t assume that people will know that they can touch a screen.
• Create an “attract state” that demonstrates interactivity while nobody is using the device
• Make touchable things look touchable
• Design for fingers
• Make sure hands don’t cover up information necessary for interaction
• Don’t rely on traditional mouse-based interactions, such as hover & double click
• Use consistent and familiar gestures
Tricks & Tips (2)
![Page 30: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/30.jpg)
• TED : Jeff Han (2006) Ideas Worth Sharing http://www.youtube.com/watch?v=5JcSu7h-I40
• Multi Touch (new touchscreen technology) http://www.youtube.com/watch?v=1ftJhDBZqss
• Google Earth on PQWindow Multi-Touch Screen http://www.youtube.com/watch?v=tLgbPZuWeFU
• DIY Multi-touch table http://nuigroup.com/go and http://www.tuio.org/ and http://sparkon.net/
Some Links
![Page 31: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/31.jpg)
Surface 2.0
![Page 32: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/32.jpg)
• Project began in 2001
• Introduced in 2008
• A surface computing platform from Microsoft.
“Microsoft Surface represents a fundamental change in the way we interact with digital content. With Surface, we can actually grab data with our hands, and move information between objects with natural gestures and touch. Surface features a 30-inch tabletop display whose unique abilities allow for several people to work independently or simultaneously. All without using a mouse or a keyboard.”
Microsoft.com
What is Microsoft Surface?
![Page 33: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/33.jpg)
A form of computing that offers “a natural way of interacting with information,” rather than the “traditional user interface.”
• Direct Interaction: The ability to "grab" digital information with hands - interacting with touch/gesture, not with a mouse or keyboard.
• Multi–Touch: The ability to recognize multiple points of contact at the same time, not just one (Ex. One finger, like with most touch screens), but dozens.
What is surface computing?
![Page 34: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/34.jpg)
• Multi–User: The Surface’s screen is horizontal, allowing many people to come together around it and experience a “collaborative, face–to–face computing experience”.
• Object Recognition: Physical objects can be placed on the Surface’s screen to “trigger different types of digital responses” (Ex. cell phones, cameras, & glasses of wine).
What is surface computing?
![Page 35: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/35.jpg)
• Wireless transfer pictures from camera to Surface and cell phone. “Drag and drop virtual content to physical objects.”
• Digital interactive painting
• At a phone store? Place cell phone on the Surface and get information, compare different phones, select service plan, accessories, and pay at table!
Some common uses
![Page 36: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/36.jpg)
• At a restaurant? View menu, order drinks and meal at your table! It’s a durable surface you can eat off of (withstands spills, etc.). Need separate checks? Split bill at and pay at table.
• Play games and use the Internet.
• Watch television
• Jukebox! Browse music, make play lists.
• Billboard for advertising
• Maps
Some common uses (2)
![Page 37: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/37.jpg)
2008 - Surface 1.0
• 15.000 $ • ≈ 70 kg • 30" • 1024 x 768 • Vista OS
![Page 38: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/38.jpg)
2011 - Surface 2.0
• 7,600 $ • ≈ 30 kg • 1920x1080, Full HD • 50 simultaneous
touches • 40" (102 cm) LCD
panel with Gorilla Glass
• Windows 7 OS • Pixelsense
![Page 39: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/39.jpg)
Mounting
Vertical
Horizontal
![Page 40: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/40.jpg)
• DiamondTouch
• reacTable
• Gesture Table
• Lemur Input Device
• Philips Entertaible
Similar DEVICES
![Page 41: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/41.jpg)
• Promo Microsoft http://www.microsoft.com/surface/en/us/whatissurface.aspx?movielinkid=experiencemovie http://www.youtube.com/watch?v=6VfpVYYQzHs&feature=related
• Case Studies http://www.microsoft.com/surface/en/us/casestudiesmanchester.aspx
• Microsoft Surface wine-tasting demo http://www.youtube.com/watch?v=Y3KzprGxpZU&feature=related
• Garibaldi Project on Microsoft Surface http://www.youtube.com/watch?v=c1SKr8dQKwo
• Patient Consultation Interface Surface Application http://www.youtube.com/watch?v=Qf0WxOo3O4g&feature=related
• Banking App http://www.youtube.com/watch?v=TU6Ghp0L8bM&feature=related.
• Playing with Microsoft Surface http://www.youtube.com/watch?v=SUfRSZppUYs&feature=related
Video
![Page 42: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/42.jpg)
• Touch2Much - Microsoft Surface Museum/Gallery Application http://www.youtube.com/watch?v=DDrCq9632YY
• AR.Drone Quadrotor Flight via Microsoft Surface http://www.youtube.com/watch?v=x1bbT8M6uRs
Video (2)
![Page 43: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/43.jpg)
• Microsoft® Surface® 2 Development Whitepaper http://www.microsoft.com/download/en/details.aspx?id=26715
• Microsoft® Surface® 2 Design and Interaction Guide http://www.microsoft.com/download/en/details.aspx?id=26713
• Microsoft® Surface® 2 SDK http://www.microsoft.com/surface/en/us/default.aspx
• Multitouch Book (free PDF) http://nuicode.com/attachments/download/115/Multi-Touch_Technologies_v1.01.pdf
Resources
![Page 44: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/44.jpg)
Gesture based interaction
HCI 2012/2013
Matteo Valoriani
![Page 45: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/45.jpg)
Some examples
![Page 46: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/46.jpg)
• Camera
• Monitor
• Microphone
The actions of child active specific behaviors of avatar
QuiQui’s Giant Bouce
Game for children (4 - 9 years) Game’s paradigm: story telling whit characters animated
![Page 47: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/47.jpg)
QuiQui’s Giant Bouce (2)
When the child cries the avatar breathes fire
When the child runs the avatar runs
![Page 48: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/48.jpg)
• The EyeToy is a color digital camera device, similar to a webcam, for the PlayStation 2.
• The technology uses computer vision and Gesture recognition to process images taken by the camera.
• This allows players to interact with games using motion, color detection and also sound, through its built-in microphone.
• Limited success due to the low precision
Sony Eye Toy
![Page 49: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/49.jpg)
• The console was released on November 19, 2006. About eight days after, 600,000 Wii’s were reported to be sold.
• It has revolutionized game play and has impacted society: anyone can play
Wii
![Page 50: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/50.jpg)
• The Wii remote, or “Wiimote”, interacts with a sensor bar by using accelerometers, infrared LED’s, and triangulation.
• In general, a player’s Wiimote movements would determine their character’s actions. A gamer would have to move in order to play.
Wii Technologies
![Page 51: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/51.jpg)
• Wii and wiimote comunicate by bluetooth
• TED 2008: Johnny Lee show how is possible connect wiimote with a normal pc and use them in innovative application:
– interactive whiteboard
– 3D head tracking
– finger tracking
• Many others researcher start to use wiimote in academia projects: http://hackaday.com
Wii Hacks
![Page 52: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/52.jpg)
Kinect
![Page 53: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/53.jpg)
• It is a motion sensing input device by Microsoft for the Xbox 360 video game console.
• Based on a webcam-style add-on peripheral for the Xbox 360 console, it enables users to control and interact with the Xbox without the need to touch a game controller, through a natural user interface using gestures and spoken commands.
• Kinect was launched in on November 2010
• 8 million units in its first 60 days
Kinect
![Page 54: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/54.jpg)
• In November 2010, is released the first Linux driver that allows the use of both the RGB camera and depth sensitivity functions of the device.
• In December 2010, PrimeSense, that produces the camera, released their open source drivers with motion tracking middleware called NITE
• Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year.
Kinect programming
![Page 55: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/55.jpg)
What does it do?
![Page 56: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/56.jpg)
Provided Data
![Page 57: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/57.jpg)
=
Immersive user experience
Kinect’s magic
![Page 58: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/58.jpg)
Cursors (hands tracking): Target an object
Avatars (body tracking): Interaction with virtual space
• Depend by the tasks
• Important aspect in design of UI
Interaction metaphors
![Page 59: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/59.jpg)
The shadow/mirror effect
Shadow Effect: • I see the back of my avatar • Problems with Z
movements
Mirror Effect: • I see the front of my avatar • Problem with mapping
left/right movements
![Page 60: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/60.jpg)
How to design a gesture?
![Page 61: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/61.jpg)
Game mindset ≠ UI mindset
User Interaction
Challenging = fun Challenging = easy and effective
![Page 62: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/62.jpg)
Gesture semantically fits user task
Abstract Meaningful
![Page 63: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/63.jpg)
System’s UI feedback relates to the user’s physical movement
User action fits UI reaction
1 2 3 4 5 6 7 8 9 10
![Page 64: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/64.jpg)
User action fits UI reaction
1 2 3 4 5 6 7 8 9 10 5
System’s UI feedback relates to the user’s physical movement
![Page 65: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/65.jpg)
Each gesture feels related and cohesive with entire gesture set
Gestures family-up
1 2 3 4 5 6 7 8 9 10
![Page 66: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/66.jpg)
Will users want/need to perform the proposed gesture repeatedly?
Repeting Gesture?
![Page 67: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/67.jpg)
Repeting Gesture?
Will users want/need to perform the proposed gesture repeatedly?
![Page 68: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/68.jpg)
Different gesture dependig on hand: only left hand can do gesture A
Handed gestures
1 2 3 4 5 6 7 8 9 10
![Page 69: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/69.jpg)
One-handed gestures are preferred
Number of Hands
1 2 3 4 5 6 7 8 9 10 6 7 8 9 10
![Page 70: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/70.jpg)
Two hand gesture should be symmetrical
Symmetrical two-handed gesture
![Page 71: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/71.jpg)
Interactions requiring more work and effort should have a higher payoff
Gesture payoff
1 2 3 4 5 6 7 8 9 10 6 7 8 9 10
![Page 72: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/72.jpg)
Fatigue is the start of downward that kills gesture
Fatigue increase messiness poor performance frustration bad UX
Fatigue kills gesture
![Page 73: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/73.jpg)
Gorilla arm problem: try to put the hand up for 10 minutes…
Fatigue kills gesture (2)
![Page 74: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/74.jpg)
User posture may affect design of a gesture
User Posture
![Page 75: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/75.jpg)
• Physical variable
The challenges
![Page 76: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/76.jpg)
• Physical variable
• Environment
The challenges
![Page 77: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/77.jpg)
• Physical variable
• Environment
• Input variability
• Recognizing intent
The challenges
![Page 78: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/78.jpg)
• Physical variable
• Environment
• Input variability
• Recognizing intent
The challenges
![Page 79: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/79.jpg)
• Physical variable
• Environment
• Input variability
• Recognizing intent
The challenges
![Page 80: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/80.jpg)
• Physical variable
• Environment
• Input variability
• Recognizing intent
The challenges
![Page 81: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/81.jpg)
Takeaways
![Page 82: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/82.jpg)
• Gestures should have a clear cognitive association with the semantic functions they perform and the effects they achieve. Intuitiveness can be enforced by appropriate interface and feedbacks.
• The semantics of gestural patterns that belong to everyday life or common task should be as consistent as possible to their “conventional” meaning, but also take into account that intuitiveness is strongly associated with users’ cultural background, general knowledge, and experience.
Semantic intuitiveness
![Page 83: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/83.jpg)
• Gestural communication involves more muscles than keyboard interaction or speech. Gestural commands must therefore be concise and quick, and minimize user’s effort and physical stress.
• Two types of muscular stress are known: static, the effort required maintaining a posture for a fixed amount of time; dynamic, related to the effort required to move a portion of the body through a trajectory.
Minimalize Fatigue
![Page 84: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/84.jpg)
• It must be easy for the user to learn how to perform and remember gestures, minimizing the mental load of recalling movement trajectories and associated actions.
• The learning rate depends on tasks, user experience, skills, as well as the size of the gesture language (more gestures decrease the learnability rate).
Favor ease of learning (Learnability) 1/2
![Page 85: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/85.jpg)
• The gestures that are most natural, easy to learn and are immediately assimilated by the user are those that belong to everyday life, or involve the least physical effort. These gestures should be associated to the most frequent interactions.
• Complex gestures can be more expressive and give more control, but have a higher learnability burden.
• Hence there is clearly a tension between design requirements, among which a compromise must be made: naturalness of gestures, minimum size of the gesture language, expressiveness and completeness of the gesture language.
Favor ease of learning (Learnability) 2/2
![Page 86: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/86.jpg)
• Users can perform unintended gestures, i.e., movements that are not meant to communicate with the system they are interacting with.
• The “immersion syndrome” occurs if every movement is interpreted by the system, whether or not it was intended, and may determine interaction effects against the user’s will.
Intentionality (Immersion Syndrome) 1/2
![Page 87: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/87.jpg)
• The designer must identify well-defined means to detect the intention of the gestures, as distinguishing useful movements from unintentional ones is not easy.
• Body tension and non-relaxed posture of users can be used to make explicit the user intention to start interaction, issue a command, or confirm a choice.
• The tense period should be short to not generate fatigue.
Intentionality (Immersion Syndrome) 2/2
![Page 88: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/88.jpg)
• Appropriate feedback indicating the effects and correctness of the gesture performed is necessary for successful interaction, and to improve the user's confidence in the system.
Not-self-revealing
![Page 89: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/89.jpg)
• http://www.xbox.com/en-US/Kinect/Kinect-Effect (rehabilitation)
• http://www.youtube.com/watch?v=id7OZAbFaVI&feature=related (Medical suse)
• http://www.kinecthacks.com/kinect-interactive-hopscotch/ (game for children)
• http://www.youtube.com/watch?v=9xMSGmjOZIg&feature=related (Holographic projection)
• http://www.youtube.com/watch?v=1dnMsmajogA&feature=related (Kinect Glasses-less 3D)
• http://www.youtube.com/watch?v=s0Fn6PyfJ0I&feature=related (Kinect Virtual Fashion, the Future of Shopping at Home)
• http://www.youtube.com/watch?v=4V11V9Peqpc&feature=related (Kinect Projection mapping with box2D physics)
Videos
![Page 90: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/90.jpg)
• http://www.youtube.com/watch?v=oALIuVb0NJ4 (Multidevice computing)
• http://www.youtube.com/watch?v=-yxRTn3fj1g&feature=related (Kinect Touch wall )
• http://www.youtube.com/watch?v=KBHgRcMPaYI&feature=related (Kinect window Bank of Moscow)
• http://kinecthacks.net/motion-control-banking-is-so-easy-even-your-pet-can-do-it/ (Kinect Banking App Video)
• http://www.youtube.com/watch?v=FMCIO0KNjrs (Art)
• http://www.youtube.com/watch?v=g6N9Qid8Tqs&feature=related (Interactive LED Floor)
• http://www.youtube.com/watch?v=c6jZjpvIio4 (Kinect in Education)
• http://www.youtube.com/watch?v=_qvMHAvu-yc&feature=related (explore universe)
Videos (2)
![Page 91: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/91.jpg)
• http://channel9.msdn.com/Search?term=kinect&type=All (Others projects)
• http://kinecthacks.net/ (Others projects)
• http://www.modmykinect.com (Others projects)
• http://kinectforwindows.org/resources/ (Microsoft SDK)
• http://www.kinecteducation.com/blog/2011/11/13/9-excellent-programming-resources-for-kinect/ (resources)
• http://kinectdtw.codeplex.com/ (gesture recognition library)
• http://kinectrecognizer.codeplex.com/ (gesture recognition library)
• http://projects.ict.usc.edu/mxr/faast/ (gesture recognition library)
• http://leenissen.dk/fann/wp/ (gesture recognition library)
Resources and tools
![Page 92: HCI Lesson 3 - BeeP...•Microsoft released a non-commercial Kinect SDK for Windows on June 16, 2011, with a commercial version following at next year. Kinect programming What does](https://reader034.vdocument.in/reader034/viewer/2022050202/5f5583eb694669691c2d5f8a/html5/thumbnails/92.jpg)
See you next time